[prometheus-users] Missing data when applying apromql calculation

[xkilian]

On Serveur2 with Prometheus 2.23 or 2.27.1

I run a query against a metric with a calculation (any will do that apply 
to a vector, like rate, avg_over_time, etc.) In he following example I use 
rate.

rate(ifHCInMulticastPkts{host="HOSTNAME",ifName="ha2"}[2m])
Rate should return a constant 0.5 every minute, but returns the following:

162487056, 0.5 
162487062, 0.5 
162487068, 0.5 
162487074, null
162487080, null
162487086, null
162487092, 1.5037673189544352 
162487098, 0.5

The raw data is present. If I collect the same data using another 
prometheus serveur running 2.27.1 on a different machine, the data is 
collected and displayed correctly both raw data and with a vector 
calculation.

This behaviour applies to all the metrics collected from the same device 
and other devices, but not all other devices. 

   - No errors in the logs
   - Does not seem to be related to reloads or restarts
   - Changing the step, time range doesnt change anything.
   
I am not sure what I should be looking at to troubleshoot.

Thank you for your help.

xkilian

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8d7e19cc-8e19-4bac-81ee-ae227d0d72abn%40googlegroups.com.

[prometheus-users] using blackbox exporter along with ec2_sd_config

[ASIF ABDUL LATHEEF]

I have 15 ec2 instances and i want to check the service status of multiple 
ports ( like 22, 443, 80, 8000 etc ) of all these instances, which i think 
is done by using blackbox exporter. So, can i create a single job for this 
by combining ec2_sd_config with blackbox exporter.? how can i give multiple 
ports in such a case.?

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/a3dd1aa1-ec79-4993-bd64-d07967e91127n%40googlegroups.com.

Re: [prometheus-users] prometheus statefulset confimap

[Ian Billett]

Hello Nina,

I'm afraid I don't really understand your question - please could you try
asking it in a different way?

Best,

Ian

On Fri, Jun 25, 2021 at 8:18 AM nina guo  wrote:

> Hi guy,
>
> If multiple statefulset replicas can share only 1 configmap?
>
> ---
> volumeMounts:   #
> configutaion volumes on the container
> - name: prometheus-config-volume
>   mountPath: /etc/prometheus
> - name: prometheus-storage-volume
>   mountPath: /prometheus
>   volumes:  #
> /etc/prometheus container volume used by ConfigMap to update the
> configurations
>   - name: prometheus-config-volume
> configMap:
>   defaultMode: 420
>   name: prometheus-server-conf
>   imagePullSecrets:
>   - name: regcred-rg
>   volumeClaimTemplates:
>   - metadata:
>   name: prometheus-storage-volume
> spec:
>   accessModes: [ "ReadWriteOnce" ]
>   resources:
> requests:
>   storage: 10Gi
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/7221d866-a2d8-419c-924e-bfbae16a36afn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CADoMBNUhG0E-G8mJVaeCqXrwa5d79craU9ihwFBL-_5F_wOsAg%40mail.gmail.com.

Re: [prometheus-users] process-exporter setup

[Ian Billett]

Hi Danny,

I would recommend raise that question directly with the upstream repo - you
are much more likely to get an answer than in this much wider group :)

Best,

Ian

On Fri, Jun 25, 2021 at 11:02 AM Danny de Waard  wrote:

> Maybe not the correct group but i want to count certain processes with
> process-exporter. But i can't seem to figure out how to seperate them.
>
> I do a simple search now with this config:
> process_names:
> # comm is the second field of /proc//stat minus parens.
> # It is the base executable name, truncated at 15 chars.
> # It cannot be modified by the program, unlike exe.
>   - comm:
> - java
> - node_exporter
> - process-exporte
> - java8
> - Agent.Listner
> - splunkd
> - tee
>
> But what i want is a seperation of java processes on a argument.
>
> So i have some processes like:
> /usr/bin/sh -c /usr/lib/jvm/jre-11-openjdk/bin/java 
> /usr/bin/sh -c /usr/lib/jvm/jre-11-openjdk/bin/java 
> /usr/lib/jvm/jre-11-openjdk/bin/java 
> /usr/lib/jvm/jre-11-openjdk/bin/java 
>
> How do i count/monitor them seperatly?
>
> Should i do something with:
>   - name: "{{.Comm}}"
> cmdline:
> - '.+'
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/3b69bd63-3151-482c-ba7c-8dd6c55a8b29n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CADoMBNVXX-Z4cJXF%2BCgS2V%2BgB9pwhQ%2BBSx-u4BuVR_ELAMz9TA%40mail.gmail.com.

Re: [prometheus-users] Duplicated timeseries in CollectorRegistry

[Ian Billett]

Hey Barry,

I'm no expert, but I would think you are somehow implicitly using a global
CollectorRegistry object in that code?

Anyway, I would recommend raising this issue with prometheus/client_python
 repo who are the experts.

Best,

Ian



On Fri, Jun 25, 2021 at 3:05 PM Barry Al-Jawari 
wrote:

> Im currently facing an issue where I do get an error that is saying:
>
>` ValueError: Duplicated timeseries in CollectorRegistry:
> {'scraper_request_count_created', 'scraper_request_count_total',
> 'scraper_request_count'}`
>
> I have two scripts which we can call file1.py and file2.py
>
> **file1.py**:
>
> ```py
> import time
> import requests
> from lib.prometheus import REQUEST_COUNT
>
> def from_page(url):
>while True:
>  with requests.get(url) as rep:
>  REQUEST_COUNT().labels(store="stackoverflow",
> http_status=rep.status_code).inc()
>  print("Response: ", rep.status_code)
>  time.sleep(60)
>
> if __name__ == '__main__':
>from_page("https://stackoverflow.com;)
> ```
>
> **file2.py**
> ```py
> import time
> import requests
> from lib.prometheus import REQUEST_COUNT
>
> def from_page(url):
>while True:
>  with requests.get(url) as rep:
>  REQUEST_COUNT().labels(store="google",
> http_status=rep.status_code).inc()
>  print("Response: ", rep.status_code)
>  time.sleep(60)
>
> if __name__ == '__main__':
>from_page("https://google.com;)
> ```
>
> As you can see they both call the `lib.prometheus import REQUEST_COUNT`
> which is:
> ```py
> from prometheus_client import Counter, CollectorRegistry
>
> registery = CollectorRegistry()
>
> def REQUEST_COUNT():
> return Counter(
> namespace="scraper",
> name="request_count",
> documentation="Count the total requests",
> labelnames=['store', 'http_status'],
> registry=registery
> )
> ```
>
> The problem is that if I run this script simultaneously then I will get
> the error `ValueError: Duplicated timeseries in CollectorRegistry:
> {'scraper_request_count_created', 'scraper_request_count_total',
> 'scraper_request_count'}` and I wonder what can I do be able to push the
> data even if its in duplicated timeseries?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/00d236e4-6ca2-46a6-ab6b-92bd3f771f8en%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CADoMBNW4_Wu0yq_w653R8_e2YF8MMHpYbst2tMEXf-rTX0y8Mg%40mail.gmail.com.

Re: [prometheus-users] Error from Terraform apply when trying to add arangoDB external service monitor to Prometheus service monitor

[Ian Billett]

Shutima,

This issue does not relate to Prometheus, I would suggest raising this
question with whomever maintains the helm chart you are attempting to
install.

Best,

Ian

On Fri, Jun 25, 2021 at 3:47 PM Shutima Potivorakun <
shutima.potivora...@geotwin.io> wrote:

> Hello,
>
> I am having issue to try to add external ArangoDB service to Prometheus
> service monitor using bearer token as described in the link below. Could
> anyone who have experienced in this, please help me ? Thank you in advance.
>
>
> https://stackoverflow.com/questions/68132861/error-from-terraform-apply-when-trying-to-add-arangodb-external-service-monitor
>
> Rgds,
> Shutima
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/391c77f4-36aa-427f-9658-ffe5681db8afn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CADoMBNWatRHbRQWsDmJ3GHJMDGs7ssUyU3j6P3_X9a0YXJAA4g%40mail.gmail.com.

Re: [prometheus-users] Multiple Prometheus monitor the same targets, and each node's Prometheus will call the remote write, resulting in data duplication

[sayf eddine Hammemi]

Hello, I discovered this a while ago, https://github.com/beorn-/highlander,
it is a reverse proxy for remote_write that deduplicates from multiple
prometheis. this ofc in the case of the remote destination does not support
dedup out of the box or is expensive resource-wise

On Mon, Jun 28, 2021 at 9:41 AM Stuart Clark 
wrote:

> Deduplication is handled by the remote write destination, such as Thanos.
> Usually there is the requirement to add an external label identifying the
> cluster and the replica to allow this to happen.
>
> On 28 June 2021 08:07:44 BST, ronghao li 
> wrote:
>>
>> Multiple Prometheus monitor the same targets, and each node's Prometheus
>> will call the remote write, resulting in data duplication.How to solve this
>> problem？
>>
>>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/7A738BD7-D4EC-44A0-9343-BF4520A069C9%40Jahingo.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALnV8Wh3aFKH78RsuchbbGgBbfAiXmb8TuLY%2BFbA7mRbGm0_OQ%40mail.gmail.com.

Re: [prometheus-users] Multiple Prometheus monitor the same targets, and each node's Prometheus will call the remote write, resulting in data duplication

[Stuart Clark]

Deduplication is handled by the remote write destination, such as Thanos. 
Usually there is the requirement to add an external label identifying the 
cluster and the replica to allow this to happen. 

On 28 June 2021 08:07:44 BST, ronghao li  wrote:
>Multiple Prometheus monitor the same targets, and each node's
>Prometheus 
>will call the remote write, resulting in data duplication.How to solve
>this 
>problem？
>
>-- 
>You received this message because you are subscribed to the Google
>Groups "Prometheus Users" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to prometheus-users+unsubscr...@googlegroups.com.
>To view this discussion on the web visit
>https://groups.google.com/d/msgid/prometheus-users/682b0393-bb49-4a5f-a539-30032c854413n%40googlegroups.com.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/7A738BD7-D4EC-44A0-9343-BF4520A069C9%40Jahingo.com.

[prometheus-users] Multiple Prometheus monitor the same targets, and each node's Prometheus will call the remote write, resulting in data duplication

[ronghao li]

Multiple Prometheus monitor the same targets, and each node's Prometheus 
will call the remote write, resulting in data duplication.How to solve this 
problem？

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/682b0393-bb49-4a5f-a539-30032c854413n%40googlegroups.com.