Re: [prometheus-users] Rollup and save historical data

[sayf eddine Hammemi]

Hello
What you need is downsampling, which is not supported directly by
prometheus (as a matter of fact, storing historical data is not either
recommended on bare prometheus),
It is time to check external storage solutions for prometheus and choose
the one that suits your needs (mimir, cortex, thanos, m3db, etc ...)

On Thu, Jun 15, 2023 at 9:01 AM Boris Behrens  wrote:

> Hi Ben,
> I've read the link you gave me and I will try to answer the best I can.
>
> > What problem are you solving for?
> I want to keep historical data for longer periods with less disk space
> used.
> I currently keep around 4 month of data for around 1.3T of disk space.
> and I would love to have 1-2 years of data without having to buy 20TB
> disks :)
>
> > Why do you think you need this?
> For debugging and monitoring I only need the last couple of days with
> a raw resolution. And seeing trends can be downsampled (is this the
> correct term?) because it is just historical trend data.
> I think it's rather a use case than an actual problem I am working on.
> I would like to see in the past how things changed and overlay it with
> timestamps from changes. Things begin to looks way different when you
> check the past.
>
> > You say "a lot of endpoints", but what is a lot? 1,000? 10,000? 100,000?
> 1,000,000? How many series?
> A couple hundred endpoints with around 1.5m of series. But I don't
> think that this information is helpful in the current context.
>
> Thank you for your time and effort to help me with my stupid little
> problem.
> Cheers
>  Boris
>
> Am Do., 15. Juni 2023 um 06:24 Uhr schrieb Ben Kochie :
> >
> > My first question is "why?" (https://xyproblem.info/)
> >
> > What problem are you solving for? Why do you think you need this?
> >
> > You say "a lot of endpoints", but what is a lot? 1,000? 10,000? 100,000?
> 1,000,000? How many series?
> >
> > On Thu, Jun 15, 2023 at 6:21 AM Boris  wrote:
> >>
> >> Dear prometheus community,
> >>
> >> my prometheus scrapes a lot of endpoints and I only need to 10s for a
> month. After that I would like to rollup the data to 5m averages and after
> 6 months I would like to have only hourly averages.
> >>
> >> So I can have quite a long time of data and can see changes over a long
> time with the same dashboard.
> >>
> >> I am sorry to ask this question but it seems like I don't have the
> correct search term. Downsampling seems to keep the raw data and all
> searches lead me to timescaledb.
> >>
> >> Cheers
> >>  Boris
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups "Prometheus Users" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> an email to prometheus-users+unsubscr...@googlegroups.com.
> >> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/8D01579C-5696-40BB-9841-A1E3E0094DD1%40kervyn.de
> .
>
>
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
> im groüen Saal.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/CAHjiyFTAWUHwrpfT9NJZxwudSrUveSvsKpG7bcpb3nRm_-CrRA%40mail.gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8Wg8zOXbx4ifhyFr-dfSw%2BqbXYAKzJBMXa%3DDCi3%2BiJVH5Q%40mail.gmail.com.

Re: [prometheus-users] Can't access specific metrics

[sayf eddine Hammemi]

Prometheus does not implement any multi-tenancy to prevent you from
accessing such metrics, you are in one of the following case
- Prometheus is not scraping itself (meta monitoring), in that case you
need to add prometheus as target
- These metrics are intentionally dropped in the prometheus configuration
- You have another system on top of Prometheus that is applying access
policies

The first one is the most probable

On Fri, Feb 17, 2023 at 2:02 PM 'Morgan WAHBI' via Prometheus Users <
prometheus-users@googlegroups.com> wrote:

> Hi guys,
>
> on the prometheus server's of my company i can't access to the metrics to
> estimate the disk used for a month
>
> so i can't access all prometheus_tsdb_* like
>
> prometheus_tsdb_head_samples_appended_total
>
> i was trying this command
> 2592000* (rate(prometheus_tsdb_head_samples_appended_total[1d]) *
> (rate(prometheus_tsdb_compaction_chunk_size_bytes_sum[1d]) /
> rate(prometheus_tsdb_compaction_chunk_samples_sum[1d])))
>
> I precise that im not admin of the platform maybe this metrics are
> available just when you are admin?
>
> If you have any idea, im interested to dive in
>
> Ty
> morgan
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/663266ea-fe64-4871-a63e-0538eacd001en%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8Wiy4eRswk3z%2Bh4veeg8dj7oJv8UGev0L0AVOcJUwFy1Jg%40mail.gmail.com.

Re: [prometheus-users] Change response of Alert

[sayf eddine Hammemi]

I think he meant to parametrize the url in the webhook from things in the
alert. if that is not possible the solution would be to have a local
adapter that receives the alerts and route them depending on their values,
sort of of Alert gateway

On Mon, Jul 11, 2022 at 12:49 PM Stuart Clark 
wrote:

> On 2022-07-11 11:23, Test Kumar wrote:
> > Hi Team,
> >
> > Is there a way so that I can change the response of Webhook alert .
> > Because I need to work according to the response.
> >
>
> How do you mean?
>
> Webhooks are connected to Alertmanager as a destination for alerts
> (depending on the routing rules). It is a one-way process - the webhook
> is triggered when an alert fires (and optionally clears). I'm not really
> sure what you are hoping for?
>
> --
> Stuart Clark
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/6cf523898e331fbb2ffc39ff10b176f0%40Jahingo.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WjD7_Q0UH8pLHgeKizTT5v7X4Df8VKMLDLXqb77JzxF_g%40mail.gmail.com.

Re: [prometheus-users] Prometheus alert trigger AWS Lambda for disk-resize

[sayf eddine Hammemi]

Hello,

For my knowledge, the best way is to set up an ApiGateway in front of the
lambda and use alertmanager webhooks to trigger it.

Btw I don't feel comfortable acting automatically on an alertmanager
payload. I ld add more logic in the lambda function to recheck the disk
(size and content (in case a faulty process is filling the disk with
useless data)) before sending the resize command, to prevent any faulty
data or flapping alert, also to not keep resize to the infinity

On Tue, Mar 8, 2022 at 10:51 AM David Friedman <
dovid.fried...@glassboxdigital.com> wrote:

> Hi all
>
> I have Prometheus set up with AlertManager and I want to set up that when
> I have a disk that gets to 90 % it will trigger an AWS Lambda that will
> enlarge my ec2 disk
>
> I see that I can set up a AWS SNS with AlertManager and that can trigger
> the Lambda
> What will be the best way to do this ?
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/4fdfee6d-8714-4b81-85e6-765425b38444n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WgAHqMe5%2BeW_JLT9VcGcXJATECphK9yNgdtE%3DaRbenoYg%40mail.gmail.com.

Re: [prometheus-users] AlertManager silence disappeared after container restart

[sayf eddine Hammemi]

I think you need to set up persistent storage, also make sure the process
is given time to gracefully shutdown to prevent corrupted data

On Tue, Feb 8, 2022, 08:28 Dovid Friedman <
dovid.fried...@glassboxdigital.com> wrote:

> Hi all
> I have AlertManager running in a container but every time I restart the
> container all my silence disappear
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/50ae07b7-b8c4-43c4-9bbb-152b83d6f0d5n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WjcBWq0-LUC_Ergb%3Du7j1qN3erz5hE1yV_6rZjJaSdCNw%40mail.gmail.com.

Re: [prometheus-users] How to mock query functions in AlertManager unit tests

[sayf eddine Hammemi]

Imo it is easier to generate your test cases dynamically, you can in that
case choose the values based on the current time

On Fri, Jan 7, 2022, 16:59 Emmanuel Alan Ferreyra Montes <
fermman...@gmail.com> wrote:

> I'm trying to run some tests over an alert I've added but I haven't been
> able to "mock" of defining the value of the function "time". Do you know
> how to do it?
>
> I'm currently doing this:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *- interval: 5m input_series: - series:
> 'probe_ssl_earliest_cert_expiry{instance="instance1", job="job-1"}' values:
> '1700 3500 500 1000' - series: 'time' values: '1
> 2000 2500 3000' alert_rule_test: - alertname: CertificateExpiry
> eval_time: 0m exp_alerts: - exp_labels: severity: warning instance:
> instance1 exp_annotations: summary: "Certificate in instance1 is due to
> expiry in less than 30 days" description: "Certificate in instance1 is due
> to expiry in less than 30 days."*
>
> and the alert is defined as follow:
>
>
>
>
>
>
>
> *- alert: CertificateExpiry expr: probe_ssl_earliest_cert_expiry - time()
> < 86400 * 30 labels: severity: warning annotations: summary: "Certificate
> in {{ $labels.instance }} is due to expiry in less than 30 days"
> description: "Certificate in {{ $labels.instance }} is due to expiry in
> less than 30 days."*
>
> Do you know how can I define the values of time in the same way I do it
> with the series?
>
> Thanks in advance,
>
> Emmanuel
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/3d9e32d3-fa3a-4415-8ad7-4c5db5d1e7b8n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WjOJkBrZV-ftWcpmd0cUta8Hi5WBbpi_FjUW7MNW97kjA%40mail.gmail.com.

Re: [prometheus-users] Moderation for new subscribers?

[sayf eddine Hammemi]

+1
Meanwhile I just set up a gmail filter to automatically delete them.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WhJcJ6uWLwxSiDm6fjLe6z399%3D%3Dg3QM6bMbrMvFeWLd4A%40mail.gmail.com.

Re: [prometheus-users] Grafana Alerts: different conditions for different labels?

[sayf eddine Hammemi]

Hello, You can store your thresholds in a metric that has the same values
of that label, that way you can compare on that label in only one query,
look at this
https://www.robustperception.io/using-time-series-as-alert-thresholds

On Wed, Jul 7, 2021 at 10:59 AM Or Fins  wrote:

> In the meantime I posted to community.grafana.com . What are the Grafana
> lists?
> On Wednesday, July 7, 2021 at 11:56:57 AM UTC+3 Stuart Clark wrote:
>
>> On 2021-07-07 09:40, Or Fins wrote:
>> > Thank you but it's not what I meant (I think). To make it clear, what
>> > I meant is instead of this:
>> >
>> > I'd like to do something like this:
>> >
>> > In the first one I separate the conditions by query, and in the second
>> > one I want to separate by label value. Is this possible?
>> >
>>
>> That sounds like a Grafana specific question, so while there are
>> probably some people on this mailing list who might know I'd also
>> suggest posting to one of the Grafana lists...
>>
>> --
>> Stuart Clark
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/2a73d53d-4cdd-48ba-aeef-d00ad7ff0f73n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WgJAom7taXGaB1b-e972XA%3DxHjqiZ2SXV8DV2a1adUBFw%40mail.gmail.com.

Re: [prometheus-users] Multiple Prometheus monitor the same targets, and each node's Prometheus will call the remote write, resulting in data duplication

[sayf eddine Hammemi]

Hello, I discovered this a while ago, https://github.com/beorn-/highlander,
it is a reverse proxy for remote_write that deduplicates from multiple
prometheis. this ofc in the case of the remote destination does not support
dedup out of the box or is expensive resource-wise

On Mon, Jun 28, 2021 at 9:41 AM Stuart Clark 
wrote:

> Deduplication is handled by the remote write destination, such as Thanos.
> Usually there is the requirement to add an external label identifying the
> cluster and the replica to allow this to happen.
>
> On 28 June 2021 08:07:44 BST, ronghao li 
> wrote:
>>
>> Multiple Prometheus monitor the same targets, and each node's Prometheus
>> will call the remote write, resulting in data duplication.How to solve this
>> problem？
>>
>>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/7A738BD7-D4EC-44A0-9343-BF4520A069C9%40Jahingo.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8Wh3aFKH78RsuchbbGgBbfAiXmb8TuLY%2BFbA7mRbGm0_OQ%40mail.gmail.com.

Re: [prometheus-users] alertmanager HA

[sayf eddine Hammemi]

The statefulset keeps track of the pods (hence the word stateful), they
will get assigned the same name and they will keep the same hostname every
time they die and come back,
ie: a statefulset named foobar with two pods will always have two pods
named foobar-0 and foobar-1
combined with a headless service, the pods are reachable at
foobar-{0/1}...svc.cluster.local

On Wed, Jun 23, 2021 at 9:08 AM nina guo  wrote:

> May I ask what's the difference once I change to Statefulset?
>
> On Wednesday, June 23, 2021 at 2:42:56 PM UTC+8 sayf.eddi...@gmail.com
> wrote:
>
>> Hello, you can make use of Kubernetes Statefulset instead of deployment
>> in this case
>>
>> On Wed, Jun 23, 2021 at 8:20 AM nina guo  wrote:
>>
>>> Hi guys,
>>>
>>> I'm implementing alertmanager HA.
>>>
>>> alertmanager0:
>>> -
>>> containers:
>>>   - name: alertmanager
>>> image: quay.io/prometheus/alertmanager:latest
>>> args:
>>>   - "--config.file=/etc/alertmanager/config.yml"
>>>   - "--storage.path=/alertmanager"
>>>   - "--cluster.listen-address=0.0.0.0:9094"
>>> ---
>>>
>>> alertmanager1:
>>> 
>>> containers:
>>>   - name: alertmanager
>>> image: quay.io/prometheus/alertmanager:latest
>>> args:
>>>   - "--config.file=/etc/alertmanager/config.yml"
>>>   - "--storage.path=/alertmanager"
>>>   - "--cluster.listen-address=0.0.0.0:9095"
>>>   - "--cluster.peer=internal_ip_of_alertmanager0:9094"
>>>
>>> alertmanager service:
>>> -
>>> apiVersion: v1
>>> kind: Service
>>> metadata:
>>>   name: alertmanager
>>>   namespace: monitoring
>>> spec:
>>>   selector:
>>> app: alertmanager
>>>   type: NodePort
>>>   ports:
>>> - port: 9093
>>>   targetPort: 9093
>>>   nodePort: 31000
>>>
>>> I have a question that if alertmanager0 restarts, then the internal ip
>>> will be changed. So I may have to update the deployment file and re-apply
>>> again.
>>> May I ask if there is a way to assign a DNS something like this to the
>>> internal IP address?
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-use...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/1cabcbca-7905-4337-807e-33b7359685c7n%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/fb12fdf7-0bb1-4670-b9d1-312d6ebd481an%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WgoDRaG9sSUyjaMn_P9L-eSMSh32Vu1MQpBsBdxcK12aQ%40mail.gmail.com.

Re: [prometheus-users] alertmanager HA

[sayf eddine Hammemi]

Hello, you can make use of Kubernetes Statefulset instead of deployment in
this case

On Wed, Jun 23, 2021 at 8:20 AM nina guo  wrote:

> Hi guys,
>
> I'm implementing alertmanager HA.
>
> alertmanager0:
> -
> containers:
>   - name: alertmanager
> image: quay.io/prometheus/alertmanager:latest
> args:
>   - "--config.file=/etc/alertmanager/config.yml"
>   - "--storage.path=/alertmanager"
>   - "--cluster.listen-address=0.0.0.0:9094"
> ---
>
> alertmanager1:
> 
> containers:
>   - name: alertmanager
> image: quay.io/prometheus/alertmanager:latest
> args:
>   - "--config.file=/etc/alertmanager/config.yml"
>   - "--storage.path=/alertmanager"
>   - "--cluster.listen-address=0.0.0.0:9095"
>   - "--cluster.peer=internal_ip_of_alertmanager0:9094"
>
> alertmanager service:
> -
> apiVersion: v1
> kind: Service
> metadata:
>   name: alertmanager
>   namespace: monitoring
> spec:
>   selector:
> app: alertmanager
>   type: NodePort
>   ports:
> - port: 9093
>   targetPort: 9093
>   nodePort: 31000
>
> I have a question that if alertmanager0 restarts, then the internal ip
> will be changed. So I may have to update the deployment file and re-apply
> again.
> May I ask if there is a way to assign a DNS something like this to the
> internal IP address?
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/1cabcbca-7905-4337-807e-33b7359685c7n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WhXyWWpTTB3wu9P2%2Bjd7-Md8Ez42FiAb4zpd2W%2BiQRrPA%40mail.gmail.com.

Re: [prometheus-users] How Query two job

[sayf eddine Hammemi]

I meant {job=~"job(1|2)"}

On Thu, May 6, 2021 at 10:23 AM sayf eddine Hammemi <
sayf.eddine.hamm...@gmail.com> wrote:

> You cant target a list but you can craft a regex
> {job=~"job(2|2)"}
>
> On Thu, May 6, 2021 at 9:33 AM Dudu  wrote:
>
>> In order to query all metrics under a single job i'm running this :
>> {job="job1"}
>>
>> What if i want to query all metrics of 2 jobs ?
>>
>> {job="job1","job2"} - This didnt do the trick
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/240b5e5c-2125-4d29-bd70-9d319dcd5786n%40googlegroups.com
>> <https://groups.google.com/d/msgid/prometheus-users/240b5e5c-2125-4d29-bd70-9d319dcd5786n%40googlegroups.com?utm_medium=email_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8Wjr4kh%3DzmkcT5sGdX5a7VD1QPhXqpFww%2BN4SUB8RM5sSg%40mail.gmail.com.

Re: [prometheus-users] How Query two job

[sayf eddine Hammemi]

You cant target a list but you can craft a regex
{job=~"job(2|2)"}

On Thu, May 6, 2021 at 9:33 AM Dudu  wrote:

> In order to query all metrics under a single job i'm running this :
> {job="job1"}
>
> What if i want to query all metrics of 2 jobs ?
>
> {job="job1","job2"} - This didnt do the trick
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/240b5e5c-2125-4d29-bd70-9d319dcd5786n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8Wge1b6meYtjU2UW2kAfT9XEMQVDfecqDski6dxPE4Zg-w%40mail.gmail.com.

Re: [prometheus-users] systemctl start prometheus failed

[sayf eddine Hammemi]

Can you try running the command directly to see what is the issue ?
/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml
--web.listen-address="0.0.0.0:9696"
--storage.tsdb.path=/opt/prometheus/data
--web.console.templates=/opt/prometheus/consoles
--web.console.libraries=/opt/prometheus/console_libraries

On Fri, Jan 29, 2021 at 10:16 AM guo galen  wrote:

> hi, I can't start prometheus with systemctl ,  thanks for any help.
>
> 1. vim /etc/systemd/system/prometheus.service
> [Unit]
> Description=prometheus
> Wants=network-online.target
> After=network-online.target
>
> [Service]
> User=prometheus
> Group=prometheus
> Type=simple
> Restart=on-failure
> ExecStart=/opt/prometheus/prometheus \
>   --config.file=/opt/prometheus/prometheus.yml \
>   --web.listen-address="0.0.0.0:9696" \
>   --storage.tsdb.path=/opt/prometheus/data \
>   --web.console.templates=/opt/prometheus/consoles \
>   --web.console.libraries=/opt/prometheus/console_libraries
>
> [Install]
> WantedBy=multi-user.target
>
> systemctl daemon-reload
> systemctl start prometheus
>
> 2. systemctl status prometheu
> prometheus.service - prometheus
>Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor
> preset: disabled)
>Active: failed (Result: start-limit) since Fri 2021-01-29 17:10:28 CST;
> 2min 59s ago
>   Process: 25576 ExecStart=/opt/prometheus/prometheus
> --config.file=/opt/prometheus/prometheus.yml --web.listen-address="
> 0.0.0.0:9696" --storage.tsdb.path=/opt/prometheus/data
> --web.console.templates=/opt/prometheus/consoles
> --web.console.libraries=/opt/prometheus/console_libraries (code=exited,
> status=2)
>  Main PID: 25576 (code=exited, status=2)
>
> Jan 29 17:10:28 zluat55 systemd[1]: Unit prometheus.service entered failed
> state.
> Jan 29 17:10:28 zluat55 systemd[1]: prometheus.service failed.
> Jan 29 17:10:28 zluat55 systemd[1]: prometheus.service holdoff time over,
> scheduling restart.
> Jan 29 17:10:28 zluat55 systemd[1]: Stopped prometheus.
> Jan 29 17:10:28 zluat55 systemd[1]: start request repeated too quickly for
> prometheus.service
> Jan 29 17:10:28 zluat55 systemd[1]: Failed to start prometheus.
> Jan 29 17:10:28 zluat55 systemd[1]: Unit prometheus.service entered failed
> state.
> Jan 29 17:10:28 zluat55 systemd[1]: prometheus.service failed.
>
> 3. uname -a
>  3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64
> x86_64 x86_64 GNU/Linux
>
> 4. prometheus version is prometheus-2.24.1.linux-amd64.tar.gz
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/c4883c67-eca5-4676-bf6e-398120550ffdn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WjbmC4xooRt8P9R-GN-4EcchptbuFfjaxKLNjQ-qFwPbw%40mail.gmail.com.

Re: [prometheus-users] Password Protection for my Prometheus Exporter

[sayf eddine Hammemi]

Hello, You are using the prometheus python client? because the feature is
supported out of the box
https://github.com/prometheus/client_python#handlers-for-authentication

On Wed, Jan 13, 2021 at 8:58 AM Dhanush Eswar 
wrote:

> Hello Everyone,
> I am new to Prometheus, I am trying to add the Basic HTTP Authentication
> for my Prometheus Exporter which is written in Python, Please help me out
> with some reference.
>
> Thanks,
> Dhanush Eswar G
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/a5eb5a8f-af6e-48b9-9763-85ec26a2b08en%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WhgbtYXYeA_EDXws-zBtHRacnAuLTX1J%3D2uNd5M_5HFKw%40mail.gmail.com.

Re: [prometheus-users] Https and basic authentication

[sayf eddine Hammemi]

I tried it too and made the mistake of placing the auth configuration in
the main config file which didn't work, these configs should be placed in a
separate `--web.config.file`. I hope this helps

On Thu, Jan 7, 2021 at 10:08 AM Mohan Nagandlla 
wrote:

> Thank you I will explore it and get back to you
>
> On Thu, 7 Jan, 2021, 2:17 PM Julien Pivotto, 
> wrote:
>
>> On 07 Jan 12:55, Mohan Nagandlla wrote:
>> > Hi team I found in version 2.24.0 the both https and authentication
>> > features are enabled. I am getting exactly how to config those by the
>> > documentation in official web site can any one guide me to config TLS
>> for
>> > prometheus and enabling the Authentication for the instance.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "Prometheus Users" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to prometheus-users+unsubscr...@googlegroups.com.
>> > To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/CALSpdGkDhbHFGEG%3D6n_rO19Vz55DRAb6J3Zdev1Gy87voG4XrQ%40mail.gmail.com
>> .
>>
>> Hello,
>>
>> The reference doc is here:
>> https://prometheus.io/docs/prometheus/latest/configuration/https/
>>
>> While we are crafting guides on the docs, I have already written a guide
>> here: https://inuits.eu/blog/prometheus-server-tls/
>>
>> regards,
>>
>> --
>> Julien Pivotto
>> @roidelapluie
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/CALSpdGnERzg6ZkSg01h2-m9gGT4RnEpzy17-XK7kuRWOmdO3yw%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WhpHaWf2s4dXYn%2Bq0hoT0KMfFAXm8X-Ne5wkYDGTiT0mA%40mail.gmail.com.

Re: [prometheus-users] How to (correctly) upsize a HA prometheus setup?

[sayf eddine Hammemi]

Hello,
Doesn't it mean that your alert expression is not reliable enough? maybe
you are trying to solve an XY problem here. did you try to make an
expression that does not trigger without enough history?

On Mon, Jul 20, 2020 at 10:26 AM 'Holger Freyther' via Prometheus Users <
prometheus-users@googlegroups.com> wrote:

> tl;dr: How to turn-up new Prometheus instances and not send (or silence)
> false alerts?
>
> My HA setup consists out of running several independent instances that
> scrape, evaluate the recording rules and alert. That should be pretty
> standard and works great until I need to increase the retention size (or
> turn-up in a new cluster, maintenance, ..). To upsize I get compute and new
> storage and spawn a new instance of Prometheus.
>
> My troubles start with alerts using larger windows on increase/rate (e.g.
> check progress of a job over a longer period of time). The new instance has
> an empty TSDB and just started to scrape it is likely to trigger this alert.
>
> I wondered if I can silence but AFAIK we make the label sets match across
> all instances of the HA set-up to allow Alertmanager to dedup.
>
> Is there a best practice for this?
>
> holger
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/c9d8c05d-ec07-4adc-b4bf-10cd88b0b5cen%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WhBPbmX2TzzDy6i_dX%3DbS0Z0Vmz%2BkqzNWojgh%3DZ28rUKA%40mail.gmail.com.

Re: [prometheus-users] Best way to scrape metrics from a JSON log files

[sayf eddine Hammemi]

As said Prometheus is just a fancy wrapper around a TSDB (time-series
database). Loki, being developed by Grafana has a sleek integration. You
can jump from metric to log file and vice versa inside grafana. Advice you
to check it out

On Sun, Jul 12, 2020 at 6:39 PM Stuart Clark 
wrote:

> On 12/07/2020 15:00, Yashar Nesabian wrote:
>
> Hi,
> some of our applications are generating JSON style logs, for now, we are
> using elastic search to store the logs and show them in the Grafana but
> this is not a good choice to use elastic search.
> I tried to use mtail but as their developers mentioned, "mtail is really 
> designed
> for getting metrics out of unstructured logs"
> So are there any other ways to send JSON style logs to the Prometheus?
>
>
> Prometheus is a metrics system. You would normally use it in parallel with
> an events system such as Elasticsearch or Loki.
>
> As you say tools like mtail can be used to create metrics from log files
> which can be very useful.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/03f1d087-3905-9123-7037-9b452e235d59%40Jahingo.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WhEY_VkrRqJSJOGULmb%2BprUTOBeiiK5AQQQV5D1D2-_iQ%40mail.gmail.com.

Re: [prometheus-users] How to check exporter down time in 1 week

[sayf eddine Hammemi]

rate() was an example, dont use it in your case

On Tue, Jul 7, 2020 at 6:21 AM Jack Chew  wrote:

> I try rate(up{idc="azure"}[5m] offset 1w) and then all value is 0. I need
> check the node within 1 week down time(up==0).
>
> 在 2020年7月6日星期一 UTC+8下午3:23:56，sayf eddine Hammemi写道：
>>
>> Hello, you can use offset to go back in time given you have enabled data
>> retention for at least one week
>> Exp: rate(http_requests_total[5m] offset 1w) (change it accordingly to
>> your metrics)
>>
>> On Mon, Jul 6, 2020 at 9:18 AM Jack Chew  wrote:
>>
>>> Hey guys,
>>>
>>>
>>>How i check up == 0 in 1week accourd  by instacne.
>>>
>>>
>>> Thanks
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to promethe...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/6127ffce-02ad-48ed-b70e-b5d27daf1647o%40googlegroups.com
>>> <https://groups.google.com/d/msgid/prometheus-users/6127ffce-02ad-48ed-b70e-b5d27daf1647o%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/42464747-6a2d-47bb-a174-3b81840c7278o%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/42464747-6a2d-47bb-a174-3b81840c7278o%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WiDNCUW%3Da34aVB55vQdNvvOSyNY9wOwicDUDORoUDaE7w%40mail.gmail.com.

Re: [prometheus-users] Prometheus query using variables from grafana templates having spaces doesn't work

[sayf eddine Hammemi]

Hello, did you try strict equality instead of regex?

On Mon, Jul 6, 2020 at 5:01 PM Debashish Ghosh 
wrote:

> Hi,
>I have created a grafana template varaible to extract names of
> organizations from Prometheus db using query
>
> label_values(custom_message_volume_endpoint_organization_total,Organization).
>
> This yields organization names that have spaces in some cases .. For
> example
> OrgA
> OrgB product1
> OrgC product2
>
> I use the following query to extract the timeseries corresponding to an
> organization..
>
> custom_latency_endpoint_organization_total{job="MyJob",Organization=~"$Organization"}
>
> It only works when I select the variable value OrgA but doesn't work for
> the other 2 . Apparently any organization name having space is discarded.
> Is there a workaround to get around this ?
>
> Thanks
> Debashish
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/c0fe6136-8ce4-4309-8516-0826cc3f12fbo%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WhUo_RZAxqBqK2jg6X5%2BNkfrFYQtKnv-NdXakkA4KrjGQ%40mail.gmail.com.

Re: [prometheus-users] Apache exporter

[sayf eddine Hammemi]

Given the impossibility of multiple processes listening on the same port on
the same server, you need to add a proxy service that will collect the
metrics from all the servers and expose it in one endpoint. You need also
to worry about is the unicity of metrics because all apache will be
exposing the same metric name you need a way to inject a unique label per
apache server before pushing them into the tsdb.  tools like
https://github.com/rebuy-de/exporter-merger will help but I don't see how
you can add a unique label with them.
The real XY problem here is why would want to run many apache servers in
one machine that is counter-intuitive.

On Mon, Jul 6, 2020 at 4:35 PM Yasmine Mbarek  wrote:

> I have more than one apache server on the same machine working on
> different ports , is there any way to monitor all of them with only one
> service ?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/69a40259-2f21-41d5-8fe5-80f12dbe5db2o%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WiJJv0TuBhH4_mpp6gqcPOaosUhg7BeL91x89_pOxG5iA%40mail.gmail.com.

Re: [prometheus-users] How to check exporter down time in 1 week

[sayf eddine Hammemi]

Hello, you can use offset to go back in time given you have enabled data
retention for at least one week
Exp: rate(http_requests_total[5m] offset 1w) (change it accordingly to your
metrics)

On Mon, Jul 6, 2020 at 9:18 AM Jack Chew  wrote:

> Hey guys,
>
>
>How i check up == 0 in 1week accourd  by instacne.
>
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/6127ffce-02ad-48ed-b70e-b5d27daf1647o%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WjgscxSMFPZ2bJcpJhrg6nEZm7eB4mOpoiQbXK06cWF4w%40mail.gmail.com.

Re: [prometheus-users] Custom Threshold for a particular instance.

[sayf eddine Hammemi]

The other proper way is to dynamically generate alerts where you hardcode
the thresholds based on labels.
Like using a combo of yaml/jinja to store the thresholds in a
maintainable format and have one command to regenerate everything.
Every time you want to change a value you just regenerate the alerts.

On Thu, Jul 2, 2020 at 8:38 PM Yagyansh S. Kumar 
wrote:

> Also, currently, I have only tried a single way to give custom threshold
> i.e based on the component name. For example, for all the targets under
> Comp-A have a threshold of 99.9 and all the targets under Comp-B have a
> threshold of 95.
> But now, I have to give a common custom threshold let say 98 to 5
> different targets, all of which belong to 5 different components and all
> the 5 components have more than 1 target but I want the custom threshold to
> be applied for only a single target from each component.
>
> On Fri, Jul 3, 2020 at 12:02 AM Yagyansh S. Kumar <
> yagyanshsku...@gmail.com> wrote:
>
>> Hi Christian,
>>
>> Actually, I want to another if there is any better way to define the
>> threshold for my 5 new servers that belong to 5 different components. Is
>> writing 5 different recording rules with the same name, and different
>> instance and component labels only way to proceed here? Won't that be a
>> little too dirty to maintain? What if it was 20 servers all belonging to a
>> different component?
>>
>> On Tue, Jun 30, 2020 at 11:43 AM Christian Hoffmann <
>> m...@hoffmann-christian.info> wrote:
>>
>>> Hi,
>>>
>>> On 6/24/20 8:09 PM, yagyans...@gmail.com wrote:
>>> > Hi. Currently I am using a custom threshold in case of my Memory
>>> alerts.
>>> > I have 2 main labels for my every node exporter target - cluster and
>>> > component.
>>> > My custom threshold till now has been based on the component as I had
>>> to
>>> > define that particular custom threshold for all the servers of the
>>> > component. But now, I have 5 instances, all from different components
>>> > and I have to set the threshold as 97. How do approach this?
>>> >
>>> > My typical node exporter job.
>>> >   - job_name: 'node_exporter_JOB-A'
>>> > static_configs:
>>> > - targets: [ 'x.x.x.x:9100' , 'x.x.x.x:9100']
>>> >   labels:
>>> > cluster: 'Cluster-A'
>>> > env: 'PROD'
>>> > component: 'Comp-A'
>>> > scrape_interval: 10s
>>> >
>>> > Recording rule for custom thresholds.
>>> >   - record: abcd_critical
>>> > expr: 99.9
>>> > labels:
>>> >   component: 'Comp-A'
>>> >
>>> >   - record: xyz_critical
>>> > expr: 95
>>> > labels:
>>> >   node: 'Comp-B'
>>> >
>>> > The expression for Memory Alert.
>>> > ((node_memory_MemTotal_bytes - node_memory_MemFree_bytes -
>>> > node_memory_Cached_bytes) / node_memory_MemTotal_bytes * 100) *
>>> > on(instance) group_left(nodename) node_uname_info > on(component)
>>> > group_left() (*abcd_critical* or *xyz_critical* or on(node) count by
>>> > (component)((node_memory_MemTotal_bytes - node_memory_MemFree_bytes -
>>> > node_memory_Cached_bytes) / node_memory_MemTotal_bytes * 100) * 0 + 90)
>>> >
>>> > Now, I have 5 servers with different components. How to include that in
>>> > the most optimized manner?
>>>
>>> This looks almost like the pattern described here:
>>> https://www.robustperception.io/using-time-series-as-alert-thresholds
>>>
>>> It looks like you already tried to integrate the two different ways to
>>> specific thresholds, right? Is there any specific problem with it?
>>>
>>> Sadly, this pattern quickly becomes complex, especially if nested (like
>>> you would need to do) and if combined with an already longer query (like
>>> in your case).
>>>
>>> I can only suggest to try to move some of the complexity out of the
>>> query (e.g. by moving the memory calculation to a recording rule
>>> instead).
>>>
>>> You can also split the rule into multiple rules (with the same name).
>>> You will just have to ensure that they only ever fire for a subset of
>>> your instances (e.g. the first variant would only fire for
>>> compartment-based thresholds, the second only for instance-based
>>> thresholds).
>>>
>>> Hope this helps.
>>>
>>> Kind regards,
>>> Christian
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/CAFGi5vB8S0_Gi03HSS%2BUFnQ%3DmWrWVwoBSAxJDhS3ed9r4QcTEA%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 

Re: [prometheus-users] Sending mails from Prometheus without passing by SMTP

[sayf eddine Hammemi]

Use webhook and a simple app that receives the hook and send the mail

On Thu, Jun 25, 2020, 10:54 Ilhem Hamdi  wrote:

> Hello,
>
> I need to send the alerting of  prometheus  to my email , the problem is I
> can not pass through SMTP , the Flux is blocked by Firewall .
>
> Any alternative idea ?
> Thanks in advance
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/CAHuiLccXCAYG%3Dq1r-yoztBT9Lo6kb-p%3D3wL45Nhi0EAAz3cruw%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WhP5ACVHcXqaMZRxq8exad5-97Fzwr2kvDQ-0OH%2BieqXw%40mail.gmail.com.

Re: [prometheus-users] Custom Threshold for Particular Instance.

[sayf eddine Hammemi]

Correct, that is my idea

On Wed, Jun 24, 2020 at 8:28 PM yagyans...@gmail.com <
yagyanshsku...@gmail.com> wrote:

> Hi. Thanks for such a quick response.
>
> Do you mean to say define 5 different recording rules with the same name
> for all those 5 servers?
> Also, in the recording rule now, I would have to give 2 labels correct?
> One is the instance itself and another is the component because my whole
> Memory expression is based on the component given in the recording rule.
>
> On Wednesday, June 24, 2020 at 11:52:46 PM UTC+5:30 sayf.eddi...@gmail.com
> wrote:
>
>> You can redefine another recoring rule with the same metric name but a
>> different value and different label, that way one record rule can have
>> multiple values depending on labels
>>
>> On Wed, Jun 24, 2020, 20:11 yagyans...@gmail.com 
>> wrote:
>>
>>> Hi. Currently I am using a custom threshold in case of my Memory alerts.
>>> I have 2 main labels for my every node exporter target - cluster and
>>> component.
>>> My custom threshold till now has been based on the component as I had to
>>> define that particular custom threshold for all the servers of the
>>> component. But now, I have 5 instances, all from different components and I
>>> have to set the threshold as 97. How do approach this?
>>>
>>> My typical node exporter job.
>>>   - job_name: 'node_exporter_JOB-A'
>>> static_configs:
>>> - targets: [ 'x.x.x.x:9100' , 'x.x.x.x:9100']
>>>   labels:
>>> cluster: 'Cluster-A'
>>> env: 'PROD'
>>> component: 'Comp-A'
>>> scrape_interval: 10s
>>>
>>> Recording rule for custom thresholds.
>>>   - record: abcd_critical
>>> expr: 99.9
>>> labels:
>>>   component: 'Comp-A'
>>>
>>>   - record: xyz_critical
>>> expr: 95
>>> labels:
>>>   node: 'Comp-B'
>>>
>>> The expression for Memory Alert.
>>> ((node_memory_MemTotal_bytes - node_memory_MemFree_bytes -
>>> node_memory_Cached_bytes) / node_memory_MemTotal_bytes * 100) *
>>> on(instance) group_left(nodename) node_uname_info > on(component)
>>> group_left() (*abcd_critical* or *xyz_critical* or on(component) count
>>> by (component)((node_memory_MemTotal_bytes - node_memory_MemFree_bytes -
>>> node_memory_Cached_bytes) / node_memory_MemTotal_bytes * 100) * 0 + 90)
>>>
>>> Now, I have 5 servers with different components. How to include that in
>>> the most optimized manner?
>>>
>>> Thanks in advance.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-use...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/58c7d88e-6538-4039-a5ae-4bd092cd8087n%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/93d1ca48-0275-4d37-b6f0-bc4eb15ffd8fn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8Wi6SRvbmyWK_J8y3zC-3-XyFckYbVn%3DmcWk9xv%3DOcNLYg%40mail.gmail.com.

Re: [prometheus-users] Custom Threshold for Particular Instance.

[sayf eddine Hammemi]

You can redefine another recoring rule with the same metric name but a
different value and different label, that way one record rule can have
multiple values depending on labels

On Wed, Jun 24, 2020, 20:11 yagyans...@gmail.com 
wrote:

> Hi. Currently I am using a custom threshold in case of my Memory alerts. I
> have 2 main labels for my every node exporter target - cluster and
> component.
> My custom threshold till now has been based on the component as I had to
> define that particular custom threshold for all the servers of the
> component. But now, I have 5 instances, all from different components and I
> have to set the threshold as 97. How do approach this?
>
> My typical node exporter job.
>   - job_name: 'node_exporter_JOB-A'
> static_configs:
> - targets: [ 'x.x.x.x:9100' , 'x.x.x.x:9100']
>   labels:
> cluster: 'Cluster-A'
> env: 'PROD'
> component: 'Comp-A'
> scrape_interval: 10s
>
> Recording rule for custom thresholds.
>   - record: abcd_critical
> expr: 99.9
> labels:
>   component: 'Comp-A'
>
>   - record: xyz_critical
> expr: 95
> labels:
>   node: 'Comp-B'
>
> The expression for Memory Alert.
> ((node_memory_MemTotal_bytes - node_memory_MemFree_bytes -
> node_memory_Cached_bytes) / node_memory_MemTotal_bytes * 100) *
> on(instance) group_left(nodename) node_uname_info > on(component)
> group_left() (*abcd_critical* or *xyz_critical* or on(component) count by
> (component)((node_memory_MemTotal_bytes - node_memory_MemFree_bytes -
> node_memory_Cached_bytes) / node_memory_MemTotal_bytes * 100) * 0 + 90)
>
> Now, I have 5 servers with different components. How to include that in
> the most optimized manner?
>
> Thanks in advance.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/58c7d88e-6538-4039-a5ae-4bd092cd8087n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WguXKohKsMmt%3DXKun5eqiSCqT%2BMyLCy9knh%3D_wUVss%2BCw%40mail.gmail.com.

Re: [prometheus-users] Prometheus-pod is not running

[sayf eddine Hammemi]

Prometheus is receiving a SIGTERM and it exists. check `kubectl describe
pod  -n ` if there are any events else
check kubelet logs.

On Wed, Jun 24, 2020 at 1:08 PM Devops Engineers 
wrote:

> Hello Everyone ,
>
> I am trying deploy prometheus operator  while deploy i am getting below
> error , cay any one help me on this to fix the issue
>
> Containers:
>   prometheus:
> Container ID:
> docker://68000ba65483853f2763712f2db3ca545b8dc943c938477b631f774ce
> Image: quay.io/prometheus/prometheus:v2.15.2
> Image ID:  docker-pullable://
> quay.io/prometheus/prometheus@sha256:91452sd38cf76a15a6aaeac069fcb445ce8fb125113d1bc5b15854bc1e8b6353
> Port:  9090/TCP
> Host Port: 0/TCP
> Args:
>   --web.console.templates=/etc/prometheus/consoles
>   --web.console.libraries=/etc/prometheus/console_libraries
>   --config.file=/etc/prometheus/config_out/prometheus.env.yaml
>   --storage.tsdb.path=/prometheus
>   --storage.tsdb.retention.time=30d
>   --web.enable-lifecycle
>   --storage.tsdb.no-lockfile
>   --web.external-url=
>   --web.route-prefix=/
> State:  Running
>   Started:  Wed, 24 Jun 2020 11:01:02 +
> *Ready:  False*
>
> level=info ts=2020-06-23T17:09:11.182Z caller=head.go:632 component=tsdb
> msg="WAL segment loaded" segment=33778 maxSegment=34083
> level=warn ts=2020-06-23T17:09:11.095Z caller=main.go:494 msg="Received
> SIGTERM, exiting gracefully..."
> level=info ts=2020-06-23T17:09:11.265Z caller=main.go:517 msg="Stopping
> scrape discovery manager..."
> level=info ts=2020-06-23T17:09:11.388Z caller=main.go:531 msg="Stopping
> notify discovery manager..."
> level=info ts=2020-06-23T17:09:11.388Z caller=main.go:553 msg="Stopping
> scrape manager..."
> level=info ts=2020-06-23T17:09:11.536Z caller=main.go:527 msg="Notify
> discovery manager stopped"
> level=info ts=2020-06-23T17:09:11.388Z caller=main.go:513 msg="Scrape
> discovery manager stopped"
> level=info ts=2020-06-23T17:09:11.871Z caller=main.go:547 msg="Scrape
> manager stopped"
>
> level=info ts=2020-06-23T17:09:28.209Z caller=manager.go:814
> component="rule manager" msg="Stopping rule manager..."
> level=info ts=2020-06-23T17:09:28.212Z caller=manager.go:820
> component="rule manager" msg="Rule manager stopped"
> ts=2020-06-23T17:09:28.214Z caller=dedupe.go:111 component=remote
> level=info remote_name=c16b16 url=http://metrics-adapter/v1/api/write
> msg="Stopping remote storage..."
> ts=2020-06-23T17:09:28.293Z caller=dedupe.go:111 component=remote
> level=error remote_name=c16b16 url=http://metrics-adapter/v1/api/write
> msg="error tailing WAL" err="readCheckpoint: readCheckpoint wasn't able to
> read all data from the checkpoint
> /prometheus/wal/checkpoint.032992/, size: 134217728, totalRead:
> 15204220"
> ts=2020-06-23T17:09:28.294Z caller=dedupe.go:111 component=remote
> level=info remote_name=c16b16 url=http://metrics-adapter/v1/api/write
> msg="WAL watcher stopped" queue=c16b16
> ts=2020-06-23T17:09:28.389Z caller=dedupe.go:111 component=remote
> level=info remote_name=c16b16 url=http://metrics-adapter/v1/api/write
> msg="Remote storage stopped."
> level=info ts=2020-06-23T17:09:28.389Z caller=notifier.go:598
> component=notifier msg="Stopping notification manager..."
> level=info ts=2020-06-23T17:09:28.556Z caller=main.go:718 msg="Notifier
> manager stopped"
> level=info ts=2020-06-23T17:09:53.472Z caller=main.go:663
> fs_type=EXT4_SUPER_MAGIC
> level=info ts=2020-06-23T17:09:53.472Z caller=main.go:664 msg="TSDB
> started"
> level=info ts=2020-06-23T17:09:53.472Z caller=main.go:730 msg="See you
> next time!"
> rpc error: code = Unknown desc = Error: No such container:
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/2157f4dd-c6c8-4ef3-9bc4-f277b7b01a1eo%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WiDy9yPwGtW-1Dz%3DFb8pSH8kDj5Tg9meaZGkYbM3MNNZQ%40mail.gmail.com.

Re: [prometheus-users] Metrics timeseries lag on master prometheus of federated metrics

[sayf eddine Hammemi]

Do you `honor_timestamps` when reading the federated metrics? are the
servers synchronized (ntp/chrony)?

On Wed, Jun 10, 2020 at 8:23 AM Raghavendra k  wrote:

>
> We have a Azure kubernetes cluster with the applications running as PODs.
> And on the same cluster but on a different namespace we have prometheus
> running to monitor the health of  the applications.
>
> We have a centralized monitoring service (Prometheus again) which is
> hosted on AWS as a kubernetes PODs.
>
> With two different hosted solutions Azure (for applications) & AWS (for
> centralized monitoring), we are trying to federate the Prometheus running
> on the Azure cluster from the Prometheus running on the AWS cluster.
> Federation is working absolutely fine, but there is a small lag in
> timeseries values of the federated metrics on the master prometheus
> (somewhere between 2-10s).
>
> I'm not able to figure out the actual cause for this time lag.
>
> Need help here to understand the time series lag and if that can be
> solved  by anyway
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/ffb0ad5e-5fc3-445b-b93b-b1512fb5b0cbo%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8Wh08pXpXMpuuJYdhnw%3DDO65uRfvo32JEydmxLawea%3D%3Dew%40mail.gmail.com.

[prometheus-users] Alerts as a code ?(Dynamically generating alerting rules)

[sayf eddine Hammemi]

Hi, I was wondering if there is any project that goes in the same direction 
as generating Prometheus alerts from code, python code maybe?
Python+Jinja maybe a path also but I am interested in fully python-based 
alerting generation.
Thnx

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/296d3fe2-9e89-4979-a71d-a6fd067d0e66o%40googlegroups.com.

Re: [prometheus-users] How to add http_proxy environment variable to prometheus alertmanager.yml ?

[sayf eddine Hammemi]

For me it is easier to setup alert manager as a systemd service then add a
drop-in directory to force proxy only on it with the file :
/usr/lib/systemd/system/alertmanager.service.d/proxy.conf that contains
[Service] Environment="HTTP_PROXY=http://proxy.example.com:80/;
"NO_PROXY=localhost,127.0.0.1,"

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8Wgk-Ee6b5tus9t26GzhNhjUzmc0%3DEWX3f4NWfsEO5DWxw%40mail.gmail.com.

Re: [prometheus-users] Prometheus snapshot restore doesn't show the old metrics

[sayf eddine Hammemi]

Hello, from the article you linked, older blocks are only snapshotted as
hardlinks while open blocks are dumped. Maybe you didn't copy the hardlinks
correctly? check the size of your folder

On Fri, Apr 24, 2020 at 2:29 PM KARIM MANAOUIL  wrote:

> Hi all,
>
> I am doing analysis of a Kubernetes cluster and because I don't have
> permanent access to it, I thought about this idea of importing Prometheus
> TSDB and analyze the data using a local Prometheus instance. I succeeded to
> do that using snapshots as told here
> . I
> dumped the snapshot into my local machine, spawned up a Prometheus
> container setting tsdb.path to the snapshot's path and it worked great, I
> can see the old data and perform queries. One night after, I woke up,
> uncompressed the snapshot again and launched a fresh new Prometheus
> instance but unfortunately now I can't see the old data. I have no idea why.
>
> The documentation 
> says something about heap-size and memory-chunks but that's doesn't work
> and arguments are invalid when I tried them.
>
> I noticed after the server is launched, the head is garbage collected and
> maybe this is the reason but honestly I have no idea.
>
> So please is there anyway to correct this issue and be able to use my old
> restored snapshot ?
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/de43a1a9-d2d9-4c4a-ae1f-e16850a4da54%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WjfVXQVSWRjKNVW7YSLsh3pzBDT%2BsFz%3DdxumPGKk5bc9A%40mail.gmail.com.

Re: [prometheus-users] JVM CPU % mismatch in weblogic when compared with CPU % in Jconsole

[sayf eddine Hammemi]

Hi, I can see that the jconsole graph has way more points than the
prometheus graph so it makes sense that there is precision inconsistency.
Jconsole looks like having a datapoint every 6 seconds. You may want to
fine tune your prom polling interval and your rate window. But I dont see
any additional gain from having that much precision.

On Wed, Apr 22, 2020 at 8:12 AM Adithyan Ganesan 
wrote:

> Sorry for the delayed response. I understand that polling frequency could
> be one of the factors and I am checking this. Do you have any idea what is
> the default polling frequency set for JMX exporter for Weblogic? I don't
> remember setting any specific frequency in my configuration file for JMX
> exporter. So I am assuming the default polling frequency is going to be
> used?
>
> Also, about the difference in utilization that I see, converted value
> displayed by *rate(process_cpu_seconds_total{job="$job"}[30s])  *is 5%
> -6% less than what I see through Jconsole or Jvisual VM. I have attached a
> screenshot in this post now.
>
> I have kept  my server running Prometheus in the same timezone as the
> machine running Weblogic.
>
>
>
>
> On Monday, 20 April 2020 14:57:27 UTC+5:30, Ben Kochie wrote:
>>
>> I don't know anything about weblogic/jconsole metrics. But I can say that
>> the process_cpu_seconds_total metric in Prometheus comes directly from the
>> kernel. So there's not a lot that can go wrong here.
>>
>> Perhaps there's a difference in polling frequency in weblogic? What is
>> your scrape interval in Prometheus?
>>
>> It may also be a time alignment issue. Prometheus may not be collecting
>> data at the same second that weblogic is.
>>
>> When you say "off", you don't say if it's higher or lower than the other
>> system.
>>
>> On Mon, Apr 20, 2020 at 11:07 AM Adithyan Ganesan 
>> wrote:
>>
>>> Hello,
>>>
>>> I have used the latest version of JMX exporter for plotting weblogic
>>> metrics. While plotting the CPU % , I noticed that there is a difference
>>> between the CPU I plotted in grafana using 
>>> *rate(process_cpu_seconds_total{job="$job"}[30s])
>>> *and the metrics that I see in Jconsole for the same weblogic instance.
>>>
>>> The percentage value arrived at by 
>>> *rate(process_cpu_seconds_total{job="$job"}[30s])
>>> *seems to be off by atleast 5-6 % and is missing out certain important
>>> high ranged values that I see as spikes in CPU from Jconsole.
>>>
>>> Could this be a bug ? I could be missing something important. If anyone
>>> has a suggestion, please let me know.
>>>
>>> Weblogic version being used is 10.3.6. I am not running this in
>>> Kubernetes.
>>>
>>> Thanks,
>>> Adi
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to promethe...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/a6ba1ab9-a260-46a3-9e5c-f39ea16b4cb7%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/460c9026-2fe1-4948-9eca-a015a1084fc9%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WiYPPLqi45ah%2BJZhCA1M3eUNuwHonxvu1M_%3D06nBYzdjw%40mail.gmail.com.

Re: [prometheus-users] help need on prometheus and grafana

[sayf eddine Hammemi]

Hi, If you want to visualize data you push from kafka, Then maybe
Prometheus is not suitable for your case. Prometheus has a PULL strategy,
you cant just push data into its TSDB.

On Tue, Apr 21, 2020 at 9:17 AM MANI KANDAN  wrote:

> hi all,
> please anyone help me to collect the actual data which i'm sending from
> kafka to prometheus to visualize the data in grafana
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/acf93971-aa7f-4930-bfae-4f73d3086e96%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8Wik-7iWKHLTcW4H6CH8-yF6jmt76bZj5abK5O2WVFgHWQ%40mail.gmail.com.

Re: [prometheus-users] Prometheus pods are going to crashloopbackoff

[sayf eddine Hammemi]

Hi, Are they writing to the same volume/path ? prometheus does not support
any kind of clustering

On Mon, Apr 20, 2020 at 7:06 AM Nishant Ketu 
wrote:

> Hi,
>
> I have deployed Prometheus through the helm, I am using statefulsets for
> multiple pods, It starts up very well, but after a few hours, the pods are
> not able to restart and going to crashloopbackoff. When I checked the logs
> it says "err="opening storage failed: invalid block sequence: block time
> ranges overlap"  \.
>
> Can someone help, what could be the issue and what is the solution?
>
> The information in this message may be proprietary and/or confidential,
> and protected from disclosure. If the reader of this message is not the
> intended recipient, you are hereby notified that any dissemination,
> distribution or copying of this communication is strictly prohibited. If
> you have received this communication in error, please notify ATMECS and
> delete it from your computer.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/dfb14839-68bc-4a21-aadb-61f019703d97%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8Whb7B6Drdz3Vt4kNo%2BS4pQttqpf9utgR0UgCnAnM9P74g%40mail.gmail.com.

Re: [prometheus-users] Pattern(s) for having a counter in short-lived applications (batch jobs)

[sayf eddine Hammemi]

You can use a gauge and create a recording rule that pushes the sum of the
gauge over time I guess

On Wed, Apr 15, 2020, 02:48 Murali Krishna Kanagala <
kanagalamur...@gmail.com> wrote:

> Hi Sean,
> Prometheus does not keep track of the counters. Metric source or the
> exporter does it.
>
> As the metric goes from 0 to N on every run of the cronjob you can create
> a synthetic metric in Prometheus that tracks the increase. If the labels
> are persistent you can track the increase and probably can aggregate them.
>
> On Tue, Apr 14, 2020, 6:31 PM Sean LeBlanc  wrote:
>
>> I'm trying to figure out if there is a way to have a counter index even
>> between application runs. My use case would be jobs scheduled via
>> Kubernetes CronJob, and having counters used for when certain methods are
>> called. The app would typically run in a few seconds and be done. So I set
>> up pushgateway, and try running the app - I see the counter show up in the
>> pushgateway, but it starts over again on the next run of the application.
>>
>> Is there some common practice that is done here that I should be
>> following if I want a counter's value to persist between application runs?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/927f938d-aa3a-4670-9a10-17371e911448%40googlegroups.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/CAKimyZv_OwQqBK%3DPErSeD%2B25Y-mfSPZ_nGiQtKM_63D%3D1NdZ8g%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WiKg_vzz1p0vObDiZ159%2BRtK4Y%3DdvFargN5JBQuWmYFNg%40mail.gmail.com.

[prometheus-users] HA scrape target

[sayf eddine Hammemi]

Hello
I have two scrape configs that provide the same metrics +- timestamps ofc 
for HA reasons. is there any simple way to configure Prometheus to ignore 
one and take another ? so if one of them is down we switch to the second. I 
am trying to avoid using HA DNS apparatus
Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/69e73fe5-f8f4-4960-87b4-29aa2c452fdf%40googlegroups.com.

Re: [prometheus-users] Problem with building a qury

[sayf eddine Hammemi]

Hello, I guess you can use regex to select both servers in one query
servername=~" SERVER1|SERVER2"

On Thu, Apr 2, 2020 at 3:55 PM SQL DBA  wrote:

> HI all,
> I'm extremely new to prometheus and have some issues with defining a
> query. The documentation didn't address the issue I'm having; probably
> something really simple but I jus dont see it. Here it is: I have about 50
> SQL Servers that I'm monitoring, this is fine. If I
> execute CPU_SQL_process it returns the info on all of them which is ok.
> I can filter down to 1; example -
>
> CPU_SQL_process{DB="SQL Server",Env="dev",cluster="Test
> node",host="",instance="x.xxx.com:9273",job="SQLSERVER_Tst",servername="SERVER1",type="CPU
> usage"}
>
>
> This returns fine but is I try to run with 2:
>
> CPU_SQL_process{DB="SQL Server",Env="dev",cluster="Test
> node",host="",instance="x.xxx.com:9273",job="SQLSERVER_Tst",servername="SERVER1",type="CPU
> usage"}
> CPU_SQL_process{DB="SQL Server",Env="dev",cluster="Test
> node",host="",instance="x.xxx.com:9273",job="SQLSERVER_Tst",servername="SERVER2",type="CPU
> usage"}
>
>
> I end up with errors - Error executing query: invalid parameter 'query':
> parse error at char 195: could not parse remaining input
> "CPU_SQL_pro"...
>
> From the documentation this should work but I must be missing something or
> have something in here I don't need. If anyone has any suggestion thank you
> in advance.
>
> Mike
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/dda3d8db-55d4-434f-a70c-8937a7172af3%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8Wg%2B3%3DHQo%3D7NcDeG5HiuiZEBnTbCG6v5LHKdWS4eMegRDg%40mail.gmail.com.

Re: [prometheus-users] SNMP_GENERATOR

[sayf eddine Hammemi]

Oh ! did you build the project? make sure to cd to the folder containing
the binary or add it to you $PATH

On Tue, Mar 31, 2020 at 7:45 PM yesmine kessentini <
yesminekessentin...@gmail.com> wrote:

> but when writing this command : which generator
> the response is :
> /usr/bin/which: no generator in
> (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/go/bin)
>
>
> Le mar. 31 mars 2020 à 18:42, yesmine kessentini <
> yesminekessentin...@gmail.com> a écrit :
>
>> thanks ,
>>
>> but i have already this package ,
>> /usr/lib/libnetsnmp.so.35
>> /usr/lib/libnetsnmp.so.30
>> /usr/lib/libnetsnmp.so.30.0.3
>> /usr/lib64/libnetsnmp.so.31
>> /usr/lib64/libnetsnmp.so.31.0.2
>> /usr/lib64/libnetsnmp.so
>> /usr/local/lib/libnetsnmp.so.35.0.0
>> /usr/local/lib/libnetsnmp.so.35
>> /usr/local/lib/libnetsnmp.so
>>
>>
>>
>>
>> Le mar. 31 mars 2020 à 17:06, sayf eddine Hammemi <
>> sayf.eddine.hamm...@gmail.com> a écrit :
>>
>>> This is a stackoverflow question
>>>
>>> ╭─sayf@degla ~
>>> ╰─$ sudo apt-file search libnetsnmp.so
>>> libsnmp-dev: /usr/lib/x86_64-linux-gnu/libnetsnmp.so
>>> libsnmp30: /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30
>>> libsnmp30: /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3
>>>
>>>
>>> You need this package
>>>
>>> On Tue, Mar 31, 2020 at 4:17 PM yesmine kessentini <
>>> yesminekessentin...@gmail.com> wrote:
>>>
>>>> HI ,
>>>>
>>>> i have this error while writing ./generator generate
>>>>
>>>> ./generator: error while loading shared libraries: libnetsnmp.so.35:
>>>> cannot open shared object file: No such file or directory
>>>>
>>>>  Can you help me in finding where is the issue?
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Prometheus Users" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to prometheus-users+unsubscr...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/prometheus-users/13049112-2117-4c04-840d-a6dd5f3edf49%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/prometheus-users/13049112-2117-4c04-840d-a6dd5f3edf49%40googlegroups.com?utm_medium=email_source=footer>
>>>> .
>>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/CAM6migbX%2BSmadX1aWLyJ%2BmoAKS7E6XuW1ZOhpSiAj%2B1T%3Dz5kpw%40mail.gmail.com
> <https://groups.google.com/d/msgid/prometheus-users/CAM6migbX%2BSmadX1aWLyJ%2BmoAKS7E6XuW1ZOhpSiAj%2B1T%3Dz5kpw%40mail.gmail.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WiussVSVKiZkupbxVUwTXZmyR8D-6Q0zTK9tZdcgVBwFA%40mail.gmail.com.

Re: [prometheus-users] SNMP_GENERATOR

[sayf eddine Hammemi]

This is a stackoverflow question

╭─sayf@degla ~
╰─$ sudo apt-file search libnetsnmp.so
libsnmp-dev: /usr/lib/x86_64-linux-gnu/libnetsnmp.so
libsnmp30: /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30
libsnmp30: /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3


You need this package

On Tue, Mar 31, 2020 at 4:17 PM yesmine kessentini <
yesminekessentin...@gmail.com> wrote:

> HI ,
>
> i have this error while writing ./generator generate
>
> ./generator: error while loading shared libraries: libnetsnmp.so.35:
> cannot open shared object file: No such file or directory
>
>  Can you help me in finding where is the issue?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/13049112-2117-4c04-840d-a6dd5f3edf49%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WjZsqZ6uHgxYH13ZnMi0h6h0oEeyiCvzrGfWBAWwOeFmw%40mail.gmail.com.

Re: [prometheus-users] Remote Write to another Prometheus

[sayf eddine Hammemi]

Prometheus does not receive it reads/polls, you have two solutions, either
to use federation as Stuart said, or you point remote read of the
"receiver" instance into the `sender:9090/api/v1/read`.
The second solution will query the data on demand (it won't move the data
to the second Prometheus)

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8Wgyi-D%2BkPuQAEH8vRmRSR1emOCVpQHh2pQjH7wiy1hPBA%40mail.gmail.com.

Re: [prometheus-users] scrape metrics for applications sitting behind a load balancer

[sayf eddine Hammemi]

For me, the load balancer itself should be polled, and it as the
sole accessor of the applications needs to expose metrics about them with
proper labels

On Wed, Mar 18, 2020 at 11:29 AM Brian Candler  wrote:

> On Wednesday, 18 March 2020 09:35:45 UTC, Jakub Jakubik wrote:
>>
>> then ignore or rewrite the instance label as it would be the same for
>> both.
>>
>>
> In that case you can actually return an "instance" label as part of the
> scrape results, and set "honor_labels: true" in the scrape job to trust and
> retain the instance label returned by the remote host.
>
> However, this seems to be completely the wrong thing to do here.  You
> should not be scraping two backends through a load balancer - you should be
> scraping the two backends separately and directly.  If there is no direct
> network connectivity between the prometheus server and the two backends,
> then enable it somehow, e.g. via port forwarding - or at worst use
> PushProx  which can act as
> a meet-in-the-middle.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/ce86f522-e4f0-4e81-9356-b3186ace1dac%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WjyFSBTDBRcwNFcKOz3RpLvmxCUT%2Bpse2%3DMAYYSF88A%3Dg%40mail.gmail.com.

Re: [prometheus-users] Problem with using custom timestamp

[sayf eddine Hammemi]

hmmm this maybe linked to the WAL size ?

On Fri, Mar 6, 2020 at 3:31 PM hai huang  wrote:

> I am getting the error message "Error on ingesting samples that are too
> old or are too far into the future" when trying to put my own custom
> timestamp into the scrape data. According to this issue (
> https://github.com/prometheus/prometheus/issues/4911), this problem
> problem has been resolved. However, in that issue, the reporter only tried
> to put a custom timestamp that is 30 minutes back. In my test, custom
> timestamp works fine unless it is older than 1 hour. Is there a
> configuration that one could tweak? Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/d92ae30c-dabb-4082-b7ee-8a6e8059451f%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WhFpFL%2BBMJA6Au4EWxMpnyt_x7hpGxmixqdX7dq%3DJjAiQ%40mail.gmail.com.

Re: [prometheus-users] Checking if NFS is hanged or not using node_exporter.

[sayf eddine Hammemi]

If the node-exporter will log errors if the nfs share hangs then u can use
mtail for example to scrape node exporter log files and export nfs errors,
that would be better than using a hand made script.

On Tue, Mar 3, 2020, 18:12 Ben Kochie  wrote:

> We added some mitigation for filesystem hangs. The node_exporter will
> notice a stuck filesystem and stop attempting to gather metrics from it
> until it gets un-stuck. Although, I don't think we have any metrics for
> when that happens, only log errors.
>
> On Tue, Mar 3, 2020 at 6:03 PM Serkan Çoban  wrote:
>
>> if I remember correctly node exporter will hang too when an nfs share
>> hangs. maybe you can test it...
>>
>> On Tue, Mar 3, 2020 at 6:26 PM Yagyansh S. Kumar
>>  wrote:
>> >
>> > I also thought about doing the same, but I am keeping that as a last
>> resort because that would require me to push the script to all my 2500+
>> servers.
>> >
>> > On Tuesday, March 3, 2020 at 8:46:27 PM UTC+5:30, Murali Krishna
>> Kanagala wrote:
>> >>
>> >> I would write a small shell script that tries to write to the nfs
>> mount  path and writes the status to a file which can be read by the text
>> file collector. And schedule that shell script cron. I think this is the
>> easiest solution.
>> >>
>> >> On Tue, Mar 3, 2020, 9:12 AM Yagyansh S. Kumar 
>> wrote:
>> >>>
>> >>> Already enabled the nfs and nfsd collectors. Till now I haven't found
>> anything that can accurately give me the information about NFS hang.
>> >>> Correct me if I am wrong, but I don't think it is a good indicator of
>> NFS hang as there may be times where no activity is happening on the NFS,
>> but that does not mean that NFS is hanged. (eg. I have 25 NFS mounts on one
>> of my servers, some of them are used rarely, so we won't find any
>> substantial IO on those mounts, but I need to know whether they are
>> accessible or not). Still, thanks for the suggestion, will try it out once.
>> >>>
>> >>>
>> >>> On Tuesday, March 3, 2020 at 8:35:03 PM UTC+5:30, Murali Krishna
>> Kanagala wrote:
>> 
>>  Try enabling the nfs options in the node exporter config. It will
>> spit out some metrics about the nfs status.
>> 
>>  Also look at the disk IO metrics from node exporter and if you see
>> no activity which indicates the nfs is not doing anything.
>> 
>>  On Tue, Mar 3, 2020, 7:10 AM Yagyansh S. Kumar 
>> wrote:
>> >
>> > I want to check if the NFS is hanged(i.e whether it is accessible
>> from the server or not, and if yes then what is the response time it is
>> getting). I know using the mountstats and nfs collector we have a lot of
>> metrics for NFS, but haven't found any that can tell me every time the NFS
>> hangs correctly.
>> > Thanks in advance.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "Prometheus Users" group.
>> > To unsubscribe from this group and stop receiving emails from it,
>> send an email to promethe...@googlegroups.com.
>> > To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/06929518-d3b5-4c2f-9490-b08cc664d26b%40googlegroups.com
>> .
>> >>>
>> >>> --
>> >>> You received this message because you are subscribed to the Google
>> Groups "Prometheus Users" group.
>> >>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to promethe...@googlegroups.com.
>> >>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/1dda60cc-0b20-47da-87ff-4f1c76ce076f%40googlegroups.com
>> .
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "Prometheus Users" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to prometheus-users+unsubscr...@googlegroups.com.
>> > To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/832f2823-eab1-4f40-8f91-ddbc00190551%40googlegroups.com
>> .
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/CAP9WWed%2BtxJVRSJc0mkCOkg6_neGAJRNEMq_hku87LPbYXAhjA%40mail.gmail.com
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/CABbyFmqMKQXYNOfdr7BeFA%3Dx%3D5fY%2Bk4EQ8oprL0Wh-8SNqmvoA%40mail.gmail.com
> 

Re: [prometheus-users] Prometheus Infra Cost

[sayf eddine Hammemi]

1 year data retention for 400+ servers needs a bit of design before
assessing tye cost. There is the metrics size, the number of prometheus
instances, federated prometheis and of course your remote storage (assuming
1 year retention is the company policy thus you need to rethink storage
security and replication etc )

On Mon, Mar 2, 2020, 12:54 Shivali Chauhan 
wrote:

> Hi Team,
>
> Could you please help me on Prometheus cost.
> How much it will cost for 400 servers for system monitoring for 1 year
> data retention.
>
>
> Regards,
> Shivali Chauhan
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/CAJzG2E8j-0-HNk53XcAfn%3D9tqJnjzbFgqhtdqiiRdd9CEZEzRg%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8WiphAyNkh-yW8aN4bmMaE8HLme-kKRjo-Hwa6A3RP7WcA%40mail.gmail.com.

[prometheus-users] Re: Prometheus federation vs remote_read

[sayf eddine Hammemi]

hmm correct the main prometheus can access the data but there no offline 
store.
Now I am wondering does federation preserve timestamp in recent version or 
I ll end up with double timestamped data
thnx

On Tuesday, February 18, 2020 at 2:16:12 PM UTC+1, Brian Candler wrote:
>
> OK - that's an unusual requirement, since you can build your scrape config 
> from a script.
>
> I don't think your solution 3 will work.  Remote_read sends queries to the 
> remote_engine when handling promQL queries, but I don't think it ingests 
> into the local TSDB.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/b6b5e47f-b8c6-46eb-a715-37bee4061069%40googlegroups.com.

[prometheus-users] Re: Prometheus federation vs remote_read

[sayf eddine Hammemi]

The use case is about using as minimal configuration in the main prometheus 
as possible which means only one configured endpoint, so exporter_exporter 
is a no go for me.

On Tuesday, February 18, 2020 at 1:10:30 PM UTC+1, Brian Candler wrote:
>
> It depends what you mean by "single endpoint".
>
> You can install exporter_exporter: you'll get a single endpoint listening 
> on a single port, which proxies to the various backend exporters.  This 
> means you only need to open one firewall port.
>
> However you'll still have to scrape it multiple times, with a different 
> param module=XXX for each backend.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/03ccfb42-d0bb-47f2-8993-9314be4b61af%40googlegroups.com.

[prometheus-users] Prometheus federation vs remote_read

[sayf eddine Hammemi]

Hello,
I have a special use case and I want to ask the community opinion on the 
implementation

*Usecase*: I want to expose all metrics from a given cluster (not k8s 
cluster) to a prometheus server through only one endpoint.
multiple components exposing metrics ---> single-endpoint --> main 
prometheus

I have thought of 3 solutions but with some caveats
*Solution1*: use a tool that merge metrics from multiple endpoints and 
expose them at once while injecting missing labels like instance and job
*Solution2*: Setup a local prometheus server that scrape all metrics and 
use its /federate endpoint as a single endpoint for the main prometheus 
server to scrape
*Solution3*: Use remote_read config option and point it to the 
local-prometheus/api/v1/read to ingest the local prometheus metrics

*Caveats*
*Solution1* is more like a dirty hack because metrics with different scrape 
intervals will be complicated to handle through only one endpoint
*Solution2*: the main prometheus server will timestamp the metrics again 
from the federation (not sure if federate  endpoint exposes metrics 
timestamp in that case i may honor timestamps)
*Solution3*: This looks the best solution so far as I read that remote_read 
will be more efficient that scraping ? (I may be mistaken)

Which solution you think it is the best, Any other suggestion is welcome
Best regards

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/3e32f579-2772-4d30-9412-b033dcb149e9%40googlegroups.com.