Re: [prometheus-users] Automatic update in service discovery for prometheus

2020-06-25 Thread Stuart Clark
it would be enough to register in Consul on startup. However, if machine's IP address does change you'd need to detect that and trigger something to update Consul. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To u

Re: [prometheus-users] Automatic update in service discovery for prometheus

2020-06-25 Thread Stuart Clark
On 25/06/2020 12:18, pratyu...@gmail.com wrote: If my node instances (IP address) gets dynamically changed then how can it be automatically updated in service discovery so that prometheus can reload the changed instances on specified refresh interval. How can we do that (automatically updating

Re: [prometheus-users] Re: data points returned prometheus http query

2020-06-24 Thread Stuart Clark
>(not table). Want users to see X calls to my end point, etc. > >On Wednesday, June 24, 2020 at 10:46:23 AM UTC-4, Stuart Clark wrote: >> >> On 2020-06-24 15:33, Johny wrote: >> > Thanks. But how do you explain the 5 minute batches? >> > >>

Re: [prometheus-users] Is there any limits for prometheus monitoring?

2020-06-24 Thread Stuart Clark
On 24/06/2020 14:23, wang dong wrote: Hi prometheus expert, we have a production cluster, 5 masters, 20 workers. And we run our service in this cluster. And we install prometheus 2.8.0 with a helm chart. After one year running, we recently keep getting OOM of prometheus pod. From the

Re: [prometheus-users] Is Prometheus suitable for our use case?

2020-06-24 Thread Stuart Clark
. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web vi

Re: [prometheus-users] Does Prometheus automatically scrape metrics by default from all Namespaces?

2020-06-23 Thread Stuart Clark
On 23/06/2020 16:40, vikram yerneni wrote: Thanks Staurt for the reply here. And nope, I didn't added the scrape_config for the metrics for this new application that we deployed recently. Before I add the scrape configuration, I want to make sure that Prometheus doesn't scrape all metrics by

Re: [prometheus-users] Does Prometheus automatically scrape metrics by default from all Namespaces?

2020-06-23 Thread Stuart Clark
There is usually some sort of annotation needed to indicate that something should be scraped. Check you scrape config to see what you currently have configured. On 23 June 2020 15:50:56 BST, vikram yerneni wrote: >Fellas, >Quick question here: > >Does Prometheus automatically scrape metrics

Re: [prometheus-users] what is the lowest scrapping and alert interval values ?

2020-06-22 Thread Stuart Clark
l run like each 5 seconds. So I prefer to use the > >metrics that Prometheus will scrap anyway. > > > > >On Monday, June 22, 2020 at 3:42:14 PM UTC-4, Stuart Clark wrote: >> >> While it is definitely possible to have very low scrape intervals and >very >>

Re: [prometheus-users] Cloudwatch exporter scraping from multiple aws accounts

2020-06-22 Thread Stuart Clark
The general design of most exporters is that they deal with a single instance of the thing being monitored (single server, database, region, account), with multiple instances being handled by running multiple copies of the exporter. One reason this often works well is because you generally need

Re: [prometheus-users] what is the lowest scrapping and alert interval values ?

2020-06-22 Thread Stuart Clark
While it is definitely possible to have very low scrape intervals and very sensitive alerts often that results in poor outcomes. The reality is that reaction times to alerts are generally fairly long - an alert outside of office hours could easily take 30 minutes or longer to respond to. I'd

Re: [prometheus-users] Best suitable Service Discovery for prometheus

2020-06-22 Thread Stuart Clark
All three are useful methods, and you may find using several is the right answer for you. Ultimately it is about collecting the information about what to scrape from the sources of truth. File is the simplest, with information either hard coded or generated by a configuration management tool.

Re: [prometheus-users] Cassandra,kafka,hadoop and SQL service up/down monitoring using prometheus and grafana

2020-06-20 Thread Stuart Clark
n. https://www.robustperception.io/monitoring-cassandra-with-prometheus https://www.robustperception.io/monitoring-kafka-with-prometheus -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop r

Re: [prometheus-users] Prometheus + (Fluentd / Logstash) + (SNMP + Traps)

2020-06-18 Thread Stuart Clark
he counter for scraping) or use an event system such as Elasticsearch. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prome

Re: [prometheus-users] Resources limits

2020-06-17 Thread Stuart Clark
allocate to the pod in order to keep it running and avoid letting it to grow as it was? The amount of memory needed depends on the scrape interval, number of timeseries being ingested and the query load. -- Stuart Clark -- You received this message because you are subscribed to the Google

Re: [prometheus-users] block size calculation behaves unintuitively when only using size based retention

2020-06-17 Thread Stuart Clark
k-duration? What is the exact issue you are seeing, as it sounds like the storage usage is still within the 2.5T limit you set? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and s

Re: [prometheus-users] Preventing data loss from poor network communication

2020-06-15 Thread Stuart Clark
mplicated to >implement, but other than that? I'll do some testing this week, but I >was >wondering if there were anything that I was missing. > >Thanks for your help, it is really appreciated. > >Cheers, > >Mathieu > >On Sun, Jun 14, 2020 at 7:32 AM Stuart Clark >

Re: [prometheus-users] Preventing data loss from poor network communication

2020-06-14 Thread Stuart Clark
. > >Cheers, > >Mathieu > >On Sat, Jun 13, 2020 at 8:25 AM Stuart Clark >wrote: > >> On 12/06/2020 19:45, Mathieu Tétreault wrote: >> > We plan on using prometheus to fetch data from multiples servers >and >> > the link between the metrics'

Re: [prometheus-users] How to store my own key-value pairs in Prometheus?

2020-06-14 Thread Stuart Clark
On 14/06/2020 01:05, Aviral Srivastava wrote: Prometheus stores time-series data by default. In this default model, the x-axis is time and the y-axis is the value. I want x-axis to be a number(1,2,3,) and y-axis to be of some value(1000, 2000, 3000, ). How do I store that in

Re: [prometheus-users] Preventing data loss from poor network communication

2020-06-13 Thread Stuart Clark
On 12/06/2020 19:45, Mathieu Tétreault wrote: We plan on using prometheus to fetch data from multiples servers and the link between the metrics's server and the prometheus servers is known for not being that reliable. The instability can last a couples of minutes and there is nothing we can do

Re: [prometheus-users] Alertmanager v2

2020-06-10 Thread Stuart Clark
will be glad for support. * |*prometheus.yml: * |api_version: v1 rule_files: - "/alertmanager/alert.rules:/alertmanager/alert.rules" || * This line doesn't look to be correct. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Promethe

Re: [prometheus-users] Re: Why do my rules seem to fail (checking with routing tree editor)

2020-06-09 Thread Stuart Clark
ot;} I think it should give an error for that though. Also, it seems to ignore "continue: false/true" and just shows all the matches anyway. You need to take a look at inhibit rules for this. https://prometheus.io/docs/alerting/configuration/#inhibit_rule -- Stuart Clark -- You

Re: [prometheus-users] Question regarding Open Metrics format and how to handle errors

2020-06-09 Thread Stuart Clark
t;up" metric (e.g. mysq_up) which has a value of 0 if the metrics can't be fetched & 1 if all is OK. For the broken metrics themselves you could leave them out (no metric is returned at all during that scrape) or possibly return NaN. -- Stuart Clark -- You received this message b

Re: [prometheus-users] Re: Single Alertmanager for Multiple Prometheus Instances - Metrics Appearing In Other Instances

2020-06-08 Thread Stuart Clark
is the actual configuration in your Prometheus? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.co

Re: [prometheus-users] Re: Single Alertmanager for Multiple Prometheus Instances - Metrics Appearing In Other Instances

2020-06-08 Thread Stuart Clark
configuration? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion

Re: [prometheus-users] Single Alertmanager for Multiple Prometheus Instances - Metrics Appearing In Other Instances

2020-06-08 Thread Stuart Clark
space problem and fired an alert with the metric labels from DC1 but with DC2-4's job name. What do you mean by this? "Prometheus in DCs2-4 also detected DC1's space problem" -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "P

Re: [prometheus-users] Help with understanding relabel config

2020-06-08 Thread Stuart Clark
ics. You want labeldrop. You are also missing the "regex" parameter. https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users&q

Re: [prometheus-users] Alertmanager: how to split alerts in email notifications

2020-06-05 Thread Stuart Clark
o/docs/alerting/configuration/#configuration-file <https://prometheus.io/docs/alerting/configuration/#configuration-file> Kind regards, Christian -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

Re: [prometheus-users] Prometheus AlertManager integration with Microsoft MSTeams

2020-06-04 Thread Stuart Clark
from the local directory. We don't actually use that Helm chart. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-

Re: [prometheus-users] Prometheus AlertManager integration with Microsoft MSTeams

2020-06-04 Thread Stuart Clark
mage. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the we

Re: [prometheus-users] Prometheus AlertManager integration with Microsoft MSTeams

2020-06-04 Thread Stuart Clark
We successfully use https://github.com/idealista/prom2teams On 4 June 2020 18:20:25 BST, Zhang Zhao wrote: >Hi, >Anyone has tested this solution on GitHub for integrating AlertManager >with >MSTeams? > >https://github.com/prometheus-msteams/prometheus-msteams > > > > >Zhang > >-- >You received

Re: [prometheus-users] Windows Exporter SetUp

2020-06-04 Thread Stuart Clark
file to scrape metrics from each Windows machine containing the WMI Exporter. Then you'd add additional alerting rules & dashboards as needed. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe f

Re: [prometheus-users] Alertmanager: how to split alerts in email notifications

2020-06-04 Thread Stuart Clark
You want to look at the group options. On 4 June 2020 16:09:48 BST, "anya...@gmail.com" wrote: >Hi, colleagues! >I setup and configured Alertmanager for getting email notifications, >but >all of my alerts about different environments are gathered in one email >and >sorted by alphabetical

Re: [prometheus-users] Can service specific alert routing be defined with service deployment definition

2020-06-04 Thread Stuart Clark
- they might choose different things to send to Slack or might use Teams instead, etc. You'd normally have some labels/annotations set by the alert detailing some sort of severity, then the routing rules would match on those and determin where an alert is sent. -- Stuart Clark -- You received

Re: [prometheus-users] Re: Modeling for prometheus

2020-06-01 Thread Stuart Clark
rate() of each time series to see the data flows between agents. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-

Re: [prometheus-users] Re: Modeling for prometheus

2020-05-31 Thread Stuart Clark
. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the

Re: [prometheus-users] Modeling for prometheus

2020-05-31 Thread Stuart Clark
(Elasticsearch, InfluxDB, etc.) are probably better for you. If you want to show details of overall system status (current number of transfers, overall transfer bandwidth, etc.) possibly with dashboards & alerts then Prometheus would probably fit. -- Stuart Clark -- You received this mes

Re: [prometheus-users] Not get current value from Prometheus using Grok Exporter

2020-05-25 Thread Stuart Clark
10-12 >hrs back. > >Because of these i end up getting too many values on my dashboard >whereas i m only intrested in current one. > >Regards >Deependra > >Sent from my iPhone > >> On 25-May-2020, at 2:10 PM, Stuart Clark >wrote: >> >> Until a n

Re: [prometheus-users] Node_exporter in opensuse

2020-05-25 Thread Stuart Clark
P address of debian and opensuse. >Like this, >- target: ['194.167.200.146:9100','194.167.200.147:9100'] > >On Monday, 25 May 2020 14:36:34 UTC+5:30, Stuart Clark wrote: >> >> What configuration do you have set for the different node exporters >in >> your prom

Re: [prometheus-users] Node_exporter in opensuse

2020-05-25 Thread Stuart Clark
What configuration do you have set for the different node exporters in your prometheus.yml? On 25 May 2020 10:03:03 BST, Harsh bhadoria wrote: >I used chrome in opensuse machine to check the metrics. > >On Monday, 25 May 2020 14:30:12 UTC+5:30, Stuart Clark wrote: >> >>

Re: [prometheus-users] Node_exporter in opensuse

2020-05-25 Thread Stuart Clark
s side), >I'm >not getting access or see the metrics. On same time, same process I've >done >in my debian system, and by Prometheus is able to get the metrics. >Problem >is with node_exporter at suse side i think > >On Monday, 25 May 2020 14:21:14 UTC+5:30, Stuart Clar

Re: [prometheus-users] Node_exporter in opensuse

2020-05-25 Thread Stuart Clark
What is the exact command being used (with all the command line options)? On 25 May 2020 09:50:16 BST, Harsh bhadoria wrote: >./node_exporter. command starts the node_exporter server. >On Monday, 25 May 2020 14:18:43 UTC+5:30, Stuart Clark wrote: >> >> What command line

Re: [prometheus-users] Node_exporter in opensuse

2020-05-25 Thread Stuart Clark
g it from prometheus >> >> >> On Monday, 25 May 2020 14:00:52 UTC+5:30, Stuart Clark wrote: >>> >>> Are you able to use curl or wget to access the node exporter from >the box >>> or the Prometheus server? >>> >>> On 25 May

Re: [prometheus-users] Node_exporter in opensuse

2020-05-25 Thread Stuart Clark
rom opensuse while going through >localhost:9100, but not getting it from prometheus > > >On Monday, 25 May 2020 14:00:52 UTC+5:30, Stuart Clark wrote: >> >> Are you able to use curl or wget to access the node exporter from the >box >> or the Prometheus server? >

Re: [prometheus-users] Not get current value from Prometheus using Grok Exporter

2020-05-25 Thread Stuart Clark
Until a new value is scraped, that 48% is the current value. As a metrics system Prometheus scrapes values regularly. As a result it is unlikely that a scrape will fall exactly on the moment needed for a query to be processed. Instead Prometheus will look back to find the latest value and use

Re: [prometheus-users] Node_exporter in opensuse

2020-05-25 Thread Stuart Clark
Are you able to use curl or wget to access the node exporter from the box or the Prometheus server? On 25 May 2020 09:22:35 BST, Harsh bhadoria wrote: >I've installed Prometheus in my windows machine and installed Debian >and >OpenSuse in VMware. I wanted to check the system metrics so, I

Re: [prometheus-users] Business Process (Icinga/Nagios) on Prometheus?

2020-05-22 Thread Stuart Clark
alerting. You'd have an alert which fires if more than X (or X%) of workers are down. https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/ -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To u

Re: [prometheus-users] Business Process (Icinga/Nagios) on Prometheus?

2020-05-22 Thread Stuart Clark
emulate BP. Any advise here?. What sort of business processes? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prome

Re: [prometheus-users] Prometheus data storage above retension size

2020-05-22 Thread Stuart Clark
Is it staying at that size or varying? Are there temporarily files due to failed compaction? Do you have any backup snapshots in that directory? During compaction you can have various temporary files, but they should go once completed. On 22 May 2020 17:10:51 BST, "João Rua" wrote: >Hello >

Re: [prometheus-users] Need Help to get the history of alerts that triggered

2020-05-22 Thread Stuart Clark
You could add a webhook to your Alertmanager configuration which then logs alerts in a database of your choice. On 22 May 2020 08:27:46 BST, piyush sharma wrote: >Hello People, > >I see that there is no feature to have a history of alerts that came up >. >Please help to find a way to get the

Re: [prometheus-users] Re: How to optimize High cardinality labels in Prometheus

2020-05-19 Thread Stuart Clark
If you are doing large queries which touch a lot of timeseries you will need lots of memory and CPU. Ideally you would minimise such queries, or use pre-aggregated metrics (created with recording rules) to simplify what is being requested. I'd suggest looking at what you are try to achieve.

Re: [prometheus-users] Re: Slack integration issue

2020-05-18 Thread Stuart Clark
prometheus-users/4733a3b2-f303-4658-aaa1-f1ee034fac6a%40googlegroups.com?utm_medium=email_source=footer -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails fro

Re: [prometheus-users] Slack integration issue

2020-05-18 Thread Stuart Clark
eployed alertmanager? Is it via Prometheus Operator or your own Docker image? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email

Re: [prometheus-users] How to use the query expression as part of annotations in alert rule

2020-05-17 Thread Stuart Clark
As has been mentioned on this list recently an alert being resolved is due to the alert stopping being sent. There is no specific "alert resolution" message being sent. The Alertmanager sends the resolution alert and only has access to the data at the time the alert was still firing, so has no

Re: [prometheus-users] How to control duration for resolved condition

2020-05-16 Thread Stuart Clark
No. The alert clears as soon as the expression is no longer true On 16 May 2020 13:42:20 BST, Steve wrote: >Hi >The Server support the parameter "for" that controls the duration >between >first occurrence and alert firing. >Does the Server support a timer that control how long the condition

Re: [prometheus-users] Is it okay to post a Prometheus user survey here?

2020-05-16 Thread Stuart Clark
ail.com?utm_medium=email_source=footer>. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com &

Re: [prometheus-users] Prometheus server increases CPU usage beyond 200%

2020-05-14 Thread Stuart Clark
s-users/41576f93-da37-4524-aba8-8e5d0e595402%40googlegroups.com?utm_medium=email_source=footer -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an e

Re: [prometheus-users] Say no to yaml

2020-05-13 Thread Stuart Clark
. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web vi

Re: [prometheus-users] remote_read: inconsistency between remote storage(influxDB) and Prometheus results.

2020-05-11 Thread Stuart Clark
it is pretty unlikely that a scrape will fall exactly "now", and could have last been scraped several minutes ago. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop r

Re: [prometheus-users] Re: One Prometheus per Observed System vs. One Prometheus for Everything

2020-05-06 Thread Stuart Clark
ding longer term querying for capacity management, while others are just short term incident management)? Does the way you operate fit into a more hierarchical structure/process (e.g region -> environment -> service -> instance) or are things more "flat"? -- Stuart Clark -- You recei

Re: [prometheus-users] One Prometheus per Observed System vs. One Prometheus for Everything

2020-05-06 Thread Stuart Clark
The other big reason to split into multiple servers is organisational rather than technical. Having servers per team or per service allows them to be owned by the same people as the service, rather than by a central team. For larger organisations this can give a lot more control over

Re: [prometheus-users] label issue for metric obtained through federation

2020-05-05 Thread Stuart Clark
suggest maybe some recording rules in the scraping Prometheus, so you can ensure that the metric always exists for each label. but with 0 or 1 values. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubs

Re: [prometheus-users] label issue for metric obtained through federation

2020-05-05 Thread Stuart Clark
For metrics with different labels you generally want each scrape to be returning values for each label. For example: {state=ok} 1 {state=unknown} 0 {state=down} 0 Is this happening? If not, you are probably getting the most recent data point for the "missing" labels, until staleness kicks in

Re: [prometheus-users] Re: SNMP Docker-compose stack

2020-05-04 Thread Stuart Clark
As you have & in your query, which is a special character for your shell, you will need to enclose the URL in quotes. On 4 May 2020 19:22:58 BST, Parthey Khanderia wrote: >When I do curl I get > > >[1] 156655 >[2] 156656 >[$ 'target' parameter must be specified once > >[1]- Done

Re: [prometheus-users] Re: SNMP Docker-compose stack

2020-05-04 Thread Stuart Clark
. Please help That error means the scrape is taking longer than the configured scrape interval or timeout (whichever is shorter). If you use curl or similar to make the same request, how long does it take to reply? -- Stuart Clark -- You received this message because you are subscribed

Re: [prometheus-users] Re: Alert manager looping in firing -> resolved -> firing

2020-05-01 Thread Stuart Clark
to other systems which do. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.co

Re: [prometheus-users] Unable to access localhost from pushgateway pod on openshift

2020-04-29 Thread Stuart Clark
are hitting the container correctly. Have you enabled the web API using --web.enable-admin-api? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, sen

Re: [prometheus-users] Unable to access localhost from pushgateway pod on openshift

2020-04-29 Thread Stuart Clark
When you are running an application inside a container "localhost" refers to the container, not the underlying host. You will need to use a different method to reference other pods, such as a service DNS name. On 29 April 2020 07:09:56 BST, Nidhi Sharma wrote: >Hi, >Do i need to make any

Re: [prometheus-users] Re: rate() doesn't work

2020-04-28 Thread Stuart Clark
be best off reducing it from 5m. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.co

Re: [prometheus-users] Reg: query on api/v1/label//values

2020-04-26 Thread Stuart Clark
you please suggest how to refresh port values or any other better approach to find the current ports which the exporter is collecting stats . If you are just wanting current data, maybe use the query API instead? -- Stuart Clark -- You received this message because you are subscribed to the Goo

Re: [prometheus-users] Most of the time metrics are coming as empty

2020-04-23 Thread Stuart Clark
is recording the time series as stale and therefore not returning anything if queried. This is by design, so the main answer would be to reduce the scraping interval for your custom exporter. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups

Re: [prometheus-users] Storing and graphing the number of terminated EC2 Instances

2020-04-23 Thread Stuart Clark
generalised database. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view th

Re: [prometheus-users] problems in configuring prom2teams

2020-04-20 Thread Stuart Clark
8089/https://outlook.office.com/webhook/X/IncomingWebhook/X' The URL should contain the name of the connector, not the Teams URL. By default that is "Connector". -- Stuart Clark -- You received this message because you are subscribed to the Google Groups &quo

Re: [prometheus-users] Sum of different Time Series for common lable

2020-04-19 Thread Stuart Clark
that Counter B have extra label as target_base_url which is absent in Counter A Kindly let us know how to achieve this in Prometheus query. You can add them if they have the same labels, so use an aggregation operator (e.g. sum()) with "by" or "without". -- St

Re: [prometheus-users] Beginner Question to clarify some core concepts

2020-04-18 Thread Stuart Clark
to reset the values to zero or remove them after scraping them. Is this intentionally? Did I miss something in my configuration? Also, as I understood it, the summary is supposed to reset itself? Hope someone can give me some hints how to solve this -- Stuart Clark -- You received this

Re: [prometheus-users] Alertmanager and use of fields/labelshook revceiver

2020-04-18 Thread Stuart Clark
are wanting. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the

Re: [prometheus-users] Docker Metrics cannot be scraped.

2020-04-16 Thread Stuart Clark
ooglegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f6ad22c3-772a-4bec-b469-cda5869bdb4c%40googlegroups.com <https://groups.google.com/d/msgid/prometheus-users/f6ad22c3-772a-4bec-b469-cda5869bdb4c%40googlegroups.com?utm_medium=email_source

Re: [prometheus-users] Discrepancy in Resolved Alerts.

2020-04-14 Thread Stuart Clark
* is more than 90%. Here, my threshold is 90% but I am receiving the resolved alert at 95.73%. Can someone help? That looks like you are putting the current value in a label. As a result any time it changes a new alert will be created. Try moving that to an annotation instead. -- Stuart Clark

Re: [prometheus-users] Invalid Push metric issue while pushing metric

2020-04-13 Thread Stuart Clark
tual label names should be fixed for a particular metric. Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-

Re: [prometheus-users] Text-based histogram exposition format

2020-04-10 Thread Stuart Clark
1. There should be an inf bucket otherwise there might be entries that don't appear in any, but are included in the count. 2. You can have other labels like any other metric, for example by status. 3 & 4. Label and row ordering doesn't really matter. What language are you using, as many of the

Re: [prometheus-users] High scaping interval & timeout

2020-04-08 Thread Stuart Clark
On 2020-04-08 10:01, Ebenolt wrote: Yes, I tried to increase timeout and interval independently, and both at the same time, nothing works higher than 65s What happens if you manually hit the exporter endpoint (e.g. curl or a browser)? Le mercredi 8 avril 2020 10:59:48 UTC+2, Stuart Clark

Re: [prometheus-users] High scaping interval & timeout

2020-04-08 Thread Stuart Clark
On 2020-04-08 09:57, Ebenolt wrote: Hey Scrape interval can't go higher than 65s or i'll get a "context deadline exceeded" Did you also increase the timeout? Le mercredi 8 avril 2020 10:56:11 UTC+2, Stuart Clark a écrit : You can set your scrape interval to 2 minutes i

Re: [prometheus-users] High scaping interval & timeout

2020-04-08 Thread Stuart Clark
om/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com?utm_medium=email_source=footer -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving

Re: [prometheus-users] Re: Data retention policy

2020-04-01 Thread Stuart Clark
files hanging around in certain error scenarios (e.g. tmp files if there are issues loading the WAL on startup or during block rotation) Le mardi 31 mars 2020 17:51:31 UTC+2, Stuart Clark a écrit : No that sounds fairly normal. One thing to note is that those timestamps are not the times

Re: [prometheus-users] Too many scrape configs on prometheus config file

2020-03-31 Thread Stuart Clark
The general suggestion is to use a configuration management system such as Ansible to create the configuration file. Equally you can use the different service discovery mechanisms to simplify your configuration too, including the file_sd to pull targets into separate files (which themselves

Re: [prometheus-users] Re: Data retention policy

2020-03-31 Thread Stuart Clark
], > [ > 1585663620, > "3" > ], > [ > 1585663680, > "3" > ], > [ > 1585663740, >

Re: [prometheus-users] Re: Data retention policy

2020-03-31 Thread Stuart Clark
How are you storing the timestamp? Is that in a label or a metric value as the last call to the API? In general these are sounding like you are trying to store events within Prometheus rather than metrics. Normally you'd not have a timestamp but a counter of the number of calls to the API.

Re: [prometheus-users] Re: Data retention policy

2020-03-27 Thread Stuart Clark
etheus-users/c3550107-62af-4120-b2d1-13926b3fff14%40googlegroups.com?utm_medium=email_source=footer>>. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send a

Re: [prometheus-users] Removed target sends resolved alert

2020-03-26 Thread Stuart Clark
On 26/03/2020 19:09, Amanda H. L. de Andrade wrote: Hi, If a remove a target that has a alert in firing state, alertmanager sends a "resolved" alert to ours receivers. How can I prevent that? That's expected as the alert is no longer firing. -- Stuart Clark -- You received th

Re: [prometheus-users] Remote Write to another Prometheus

2020-03-19 Thread Stuart Clark
us. Does anyone have a hint for me ? You can't use remote write for this. Instead you can use federation, but in general it isn't recommended to copy all metrics from one Prometheus to another. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "P

Re: [prometheus-users] Monitoring multiple node instances on the same server

2020-03-16 Thread Stuart Clark
You'd normally have a job per exporter in your configuration, with different jobs using different ports. So your node exporter job would be scraping port 9100 and then other jobs for other exporters on other ports. On 16 March 2020 15:24:51 GMT, Silas Bempong wrote: >Hi, I'm a newbie to

Re: [prometheus-users] usage of rate function on recording metric

2020-03-10 Thread Stuart Clark
and rate() will return a spike at that point. Thanks n Regards, Chalapathi. On Mon, Mar 9, 2020 at 5:09 PM Julien Pivotto <mailto:roidelapl...@inuits.eu>> wrote: On 09 Mar 11:35, Stuart Clark wrote: > On 2020-03-09 11:21, Venkata Bhagavatula wrote: &

Re: [prometheus-users] usage of rate function on recording metric

2020-03-09 Thread Stuart Clark
be treated as Gauge? If the metric reduced (and it isn't a counter reset) then it suggest it isn't a counter, and therefore rate() shouldn't be used. Thanks n Regards, Chalapathi On Thu, Feb 27, 2020 at 12:59 PM Stuart Clark wrote: The graph showed a reduction at various points

Re: [prometheus-users] Working of Textfile Collector.

2020-03-07 Thread Stuart Clark
ooglegroups.com <https://groups.google.com/d/msgid/prometheus-users/71c47c5f-2c24-4714-a675-c3368b81086b%40googlegroups.com?utm_medium=email_source=footer>. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. T

Re: [prometheus-users] Prometheus Infra Cost

2020-03-02 Thread Stuart Clark
, Azure, VMware, etc.) and the number of metrics you are storing. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-user

Re: [prometheus-users] Read access log as metrics

2020-03-02 Thread Stuart Clark
ogle.com/d/msgid/prometheus-users/a5750a63-3da7-4496-bd40-4a3b6ea70e28%40googlegroups.com?utm_medium=email_source=footer -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop r

Re: [prometheus-users] Tracing metrics from only EC2 instances running JMX exporter

2020-03-01 Thread Stuart Clark
You need to have a way of determining which targets to scrape from the information returned from the EC2 service discovery call. The main details are tags, but you can also use AZ, instance type, platform, subnet, VPC. So as long as you can differentiate using those then you can create

Re: [prometheus-users] Prometheus HA strategies

2020-02-27 Thread Stuart Clark
Prometheus isn't a clustered system by design, so it expects to have complete control of the data files. If another process starts changing the files it would quickly result in data corruption. The large benefit of totally separate Prometheus servers without shared storage is simplicity. As

Re: [prometheus-users] Task metrics

2020-02-27 Thread Stuart Clark
Can the system which controls the tasks collect the information when a task finishes and then add it to metrics it exposes? Total number of tasks completed, total time spent, etc. On 27 February 2020 13:29:28 GMT, Paul van der Linden wrote: >I'm trying to scrape metrics from tasks. The

Re: [prometheus-users] usage of rate function on recording metric

2020-02-26 Thread Stuart Clark
tically adjusted for. > >Also why it worked on the original metric as on both dervied and >original >metric has reduction? > >Thanks n Regards, >Chalapathi > > >On Wed, Feb 26, 2020 at 10:49 PM Stuart Clark > >wrote: > >> Counters must only increase

Re: [prometheus-users] usage of rate function on recording metric

2020-02-26 Thread Stuart Clark
Counters must only increase. Any reduction is seen as a counter reset. If this isn't a counter use the derivative function rather than rate On 26 February 2020 14:05:31 GMT, Venkata Bhagavatula wrote: >Hi, > >In our application, there is one metric that we are deriving from >another >metric

<    1   2   3   4   5   6   >