Re: [prometheus-users] [blacbox_exporter - icmp] Multiple Source and Target

2021-03-26 Thread Stuart Clark
olon to __param_target and the bit after to __address__ (with the :9115 suffix). -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-

Re: [prometheus-users] Alerts sent to sms

2021-03-25 Thread Stuart Clark
would also have some level of configuration needed. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-user

Re: [prometheus-users] Why does increase() return fractional results for integer-valued metrics?

2021-03-23 Thread Stuart Clark
to force the returned value to be an integer. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@google

Re: [prometheus-users] Re: Seeing Wraning in Prometheus UI

2021-03-23 Thread Stuart Clark
account , which seems to be work allright . Do you maybe have something incorrect relating to the timezone settings? Something not picking the correct setting up can easily cause time to be several hours out between two systems, even if the actual UTC time is in sync. -- Stuart Clark -- You

Re: [prometheus-users] Re: Seeing Wraning in Prometheus UI

2021-03-23 Thread Stuart Clark
ilar) on all computers as unsynchronised clocks can cause huge issues (not just with Prometheus). -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it,

Re: [prometheus-users] Increasing scrape_interval causing missing data point for metrics

2021-03-22 Thread Stuart Clark
rather than being real-time). Using the Node exporter is also likely to be cheaper as the only costs are network bandwidth rather than the various API calls. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To u

Re: [prometheus-users] Increasing scrape_interval causing missing data point for metrics

2021-03-22 Thread Stuart Clark
? I would suggest only using the Cloudwatch exporter for things which you can't get elsewhere. So for example use node exporter, MySQL exporter, etc. for the main metrics, but the cloudwatch exporter for things like ALB or NAT gateway metrics (which aren't available elsewhere). -- Stuart Clark

Re: [prometheus-users] Increasing scrape_interval causing missing data point for metrics

2021-03-22 Thread Stuart Clark
and service tags). -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this

Re: [prometheus-users] Increasing scrape_interval causing missing data point for metrics

2021-03-22 Thread Stuart Clark
trying to increase the scrape interval above 2 minutes? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@

Re: [prometheus-users] scrape interval vs exporter internals

2021-03-22 Thread Stuart Clark
in the exact moment? The normal mechanism is to trigger a fetch/generate of any metrics that aren't continuous when a scrape comes in. If needed you might then cache those values within the exporter for a period if they are expensive/slow to produce. -- Stuart Clark -- You received

Re: [prometheus-users] Enterprises using Prometheus

2021-03-20 Thread Stuart Clark
try to keep things more separated, so using different storage for each team, which makes maintenance easier and allows each team to control their own configuration, at the cost of a bit more complexity/infrastructure. -- Stuart Clark -- You received this message because you are subscribed

Re: [prometheus-users] edit the default alertmanager global url like wechat pagerduty etc

2021-03-20 Thread Stuart Clark
editable in the config file: https://prometheus.io/docs/alerting/latest/configuration/#configuration-file For example the "opsgenie_api_url" or "victorops_api_url" under "global", or "api_url" within a wechat receiver section. -- Stuart Clark

Re: [prometheus-users] Count/display label dimensions with a certain value?

2021-03-18 Thread Stuart Clark
h have the value of 1. You can then format that within Grafana as a table for example with the subsystem label. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emai

Re: [prometheus-users] daily counteur increase:

2021-03-17 Thread Stuart Clark
you "align the Querry range and the Timestamp" in grafana? You need to ensure the query is being run between midnight & midnight rather than 24 hours since now. With API calls you can include the start/end timestamps to use while in Grafana you can set something similar. -- Stuar

Re: [prometheus-users] How to share task id between altert manager receivers

2021-03-17 Thread Stuart Clark
ngs like escalations, business rules, etc. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroup

Re: [prometheus-users] Show scrape errors in prometheus metrics

2021-03-16 Thread Stuart Clark
On 16/03/2021 10:14, Tom Liefheid wrote: yes, but running a HA prometheus doesn't let me see which prom-instance has the issues, as only 1 is failing How do you mean? You can query each instance or external labels should indicate which instance has the failing metric... -- Stuart Clark

Re: [prometheus-users] Show scrape errors in prometheus metrics

2021-03-16 Thread Stuart Clark
page). You should see labels for the job/target which is failing. It can be a useful metric to alert on, and then look at logs/the target page to try to figure out why the scrape is failing. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "P

Re: [prometheus-users] Up server alert

2021-03-16 Thread Stuart Clark
that indicate the outcome of that processing (e.g. as well as producing metrics via the Node Exporter for a ETL database job you could produce metrics from database queries, giving you the ability to alert if the job hasn't appeared to run or if it hasn't appeared to add records). -- Stua

Re: [prometheus-users] Delete old data in prometheus and grafana.

2021-03-16 Thread Stuart Clark
. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web vi

[prometheus-users] Re: monitor NFS mounts (despite NFS is in collector.filesystem.ignored-fs-types)

2021-03-15 Thread Stuart Clark
(on a different port). -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this

Re: [prometheus-users] Correct format in hours for alert rules for blackbox prometheus alerts

2021-03-11 Thread Stuart Clark
are wanting. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the

Re: [prometheus-users] Prometheus Node_Exporter SysV service metric supported?

2021-03-09 Thread Stuart Clark
other or better way to do that! That's one option. Another could be instrumenting the application itself or looking at the process exporter... -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from

Re: [prometheus-users] Prometheus Node_Exporter SysV service metric supported?

2021-03-09 Thread Stuart Clark
and their statuses, but for sysv the best you can really do is look in the runlevels directories and hope each one has a "status" option. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from

Re: [prometheus-users] What is a sample? Why different number of scrape_samples_scraped for node_exporter VMs?

2021-03-09 Thread Stuart Clark
On 2021-03-09 12:36, 'Evelyn Pereira Souza' via Prometheus Users wrote: On 09.03.21 13:32, Stuart Clark wrote: Different node exporter settings, different number of filesystems, network interfaces, etc.? Hi The node_exporter settings are exactly the same (Puppet rollout). Only nr

Re: [prometheus-users] What is a sample? Why different number of scrape_samples_scraped for node_exporter VMs?

2021-03-09 Thread Stuart Clark
On 2021-03-09 12:25, 'Evelyn Pereira Souza' via Prometheus Users wrote: Hi What is a sample exactly? Why different number of scrape_samples_scraped for node_exporter VMs? Different node exporter settings, different number of filesystems, network interfaces, etc.? -- Stuart Clark -- You

Re: [prometheus-users] federation will cache metrcis??

2021-03-04 Thread Stuart Clark
rts. If you have alerts which require there to be a value (e.g. checking memory is above a threshold) they would stop firing once the time series is marked as stale. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

Re: [prometheus-users] federation will cache metrcis??

2021-03-04 Thread Stuart Clark
any other scrape, so it fetches the metrics from the other Prometheus server and stores it within the TSDB. Therefore you will retain the metrics (and be able to query them) for however long your retention period is. -- Stuart Clark -- You received this message because you are subscribed

Re: [prometheus-users] Clarifying jobs/targets/instances when monitoring multiple systems from a single Prometheus server

2021-03-04 Thread Stuart Clark
On 03/03/2021 17:18, John Dexter wrote: -- Stuart Clark Stuart, this is totally bespoke C++ Windows system,  none of those :) The file-discovery seems able to pull out my targets but I can't find a good example how to structure my files. I suppose each

Re: [prometheus-users] Clarifying jobs/targets/instances when monitoring multiple systems from a single Prometheus server

2021-03-03 Thread Stuart Clark
On 03/03/2021 14:33, John Dexter wrote: On Wed, 3 Mar 2021 at 14:15, Stuart Clark <mailto:stuart.cl...@jahingo.com>> wrote: On 03/03/2021 13:00, John Dexter wrote: > While I understand the technical definitions I would appreciate some > real-world advice

Re: [prometheus-users] Clarifying jobs/targets/instances when monitoring multiple systems from a single Prometheus server

2021-03-03 Thread Stuart Clark
they would approach this in terms of job/scrape config? file-based discovery seems a great option but I'm struggling how to avoid a lot of copy-paste and duplicate information. How are your applications deployed and managed? Kubernetes, Docker, Ansible, Puppet, etc.? -- Stuart Clark -- You

Re: [prometheus-users] TSDB storage use constantly increasing

2021-03-03 Thread Stuart Clark
. Depending on the remote write destination used the amount of storage needed could be a lot larger than it would be to keep directly in Prometheus. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubs

Re: [prometheus-users] Staggering scrapes to different targets?

2021-03-03 Thread Stuart Clark
pe happens that varies between targets - you won't have all targets being scraped at the same time, but they all will be being scraped exactly the configured scrape interval after the previous one. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "P

Re: [prometheus-users] Staggering scrapes to different targets?

2021-03-02 Thread Stuart Clark
f 60s, you might find that the scraping happened at 10s past the minute for target 1 and 40s past the minute for target 2. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop r

Re: [prometheus-users] Removing namespace component in metric's keys

2021-03-01 Thread Stuart Clark
ted that might not be the case - a sum wouldn't mean anything and an average would be equally useless as the different systems do a totally different selection of work. However if you are looking at latencies end-to-end across multiple systems in a flow, or have multiple instances of a system, then it

Re: [prometheus-users] how to get metrics for activemq and mssql server using prometheus

2021-03-01 Thread Stuart Clark
for activemq and mssql server .can anyone suggest how to resolve those Are you able to give some details of the issues you are seeing? Are there any error messages? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users&q

Re: [prometheus-users] Not able to push the metrics from Prometheus on a Kubernetes cluster to another Prometheus server

2021-02-28 Thread Stuart Clark
reasonable. I hope you get it all working and it does what you are hoping for :-) -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to

Re: [prometheus-users] Per-host thresholds in prometheus alert

2021-02-26 Thread Stuart Clark
ld (which needs periodic checks & adjustment) you can use the predict_linear function (https://prometheus.io/docs/prometheus/latest/querying/functions/#predict_linear) to project how long until all swap is used based on recent usage increases. -- Stuart Clark -- You received this messa

Re: [prometheus-users] Scrape kube-state-metrics data/metrics from a URL as target

2021-02-26 Thread Stuart Clark
the cluster you just need to add a job to your prometheus.yaml with the target URL/IP address (probably using the static_sd option) of the load balancer. You don't need any relabeling configuration generally. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups

Re: [prometheus-users] Append new labels to existing list of lables in prometheus metrics

2021-02-25 Thread Stuart Clark
the metric is presented could be confusing, so in those situations it might make sense to change the name to explicitly split things into a "before" and "after". -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Use

Re: [prometheus-users] Not able to push the metrics from Prometheus on a Kubernetes cluster to another Prometheus server

2021-02-25 Thread Stuart Clark
hich would likely require a lot more resources than would reasonably be available). -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email

Re: [prometheus-users] Not able to push the metrics from Prometheus on a Kubernetes cluster to another Prometheus server

2021-02-25 Thread Stuart Clark
ervers too, allowing both aggregated and specific queries. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prome

Re: [prometheus-users] Not able to push the metrics from Prometheus on a Kubernetes cluster to another Prometheus server

2021-02-25 Thread Stuart Clark
cesses (such as cron jobs) which can't be scraped directly, and uses a different API (not the remote write API). Maybe it would help if you could describe what you are trying to do from a non-technical perspective? Why are you trying to send metrics from one server to another? -- Stuart Clark -- You re

Re: [prometheus-users] Not able to push the metrics from Prometheus on a Kubernetes cluster to another Prometheus server

2021-02-24 Thread Stuart Clark
metrics via federation, although this is designed for a limited subset of aggregate metrics and not a complete copy. Alternatively a system such as Thanos could be used as an alternative global metrics store which can be accessed by all your Prometheus servers. -- Stuart Clark -- You received

Re: [prometheus-users] Prometheus is failing to scrape a target pod, but able to "wget" it inside Prometheus pod

2021-02-23 Thread Stuart Clark
- default According to this: https://blog.getambassador.io/understanding-envoy-proxy-and-ambassador-http-access-logs-fee7802a2ec5 The UC response flags indicate "Upstream connection termination", which suggests something on the destination service. I would suggest looking at the logs

Re: [prometheus-users] Understanding Prometheus

2021-02-23 Thread Stuart Clark
from that data, so Prometheus could still have a place in the overall system, but as Ben says it is really about monitoring the overall state of the system, with alerts or dashboards showing number of online users, amounts of sales, CPU usage, database connections or network bandwidth. --

Re: [prometheus-users] Prometheus Federation between AWS and GCP

2021-02-23 Thread Stuart Clark
hich can then be used to give a global query capability. There are various options available which generally store the data in some form of database or object storage system. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Use

Re: [prometheus-users] prometheus cannot access target behind haproxy

2021-02-23 Thread Stuart Clark
X-XXX-     static_configs:       - targets:         - service.haproxy.local:443 But in the prometheus I get 503 service unavailable What do the HA Proxy logs say when Prometheus requests are made? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Promethe

Re: [prometheus-users] Memory Uses Very High

2021-02-21 Thread Stuart Clark
of complex dashboards which are being viewed by many people? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-user

Re: [prometheus-users] Memory Uses Very High

2021-02-21 Thread Stuart Clark
are you scraping? What level of query complexity and frequency are you seeing? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to promet

Re: [prometheus-users] Discuss Prmetheus alerts suppressing

2021-02-19 Thread Stuart Clark
or connect alertmanagers to the non-production servers. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-user

Re: [prometheus-users] Prometheus is failing to scrape a target pod, but able to "wget" it inside Prometheus pod

2021-02-19 Thread Stuart Clark
light on where that error is coming from? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegr

Re: [prometheus-users] limit on time series per metric

2021-02-17 Thread Stuart Clark
metrics with 10 series each. The more time series you are touching for a query, the more data you have to load into memory, so the larger the resource requirements. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users&q

Re: [prometheus-users] limit on time series per metric

2021-02-17 Thread Stuart Clark
datacenters or AWS regions) as well as different services/applications/areas (whatever makes sense based on organisation or technical structures). You can then use tools such as federation or a remote read/write system (such as Thanos) to construct global views and alerts if needed. -- Stuart

Re: [prometheus-users] Re: how to drop "le="+Inf" label from kube-apiserver metrics

2021-02-16 Thread Stuart Clark
-users/7600d164-feb3-4716-b062-7348b7a1e470n%40googlegroups.com <https://groups.google.com/d/msgid/prometheus-users/7600d164-feb3-4716-b062-7348b7a1e470n%40googlegroups.com?utm_medium=email_source=footer>. -- Stuart Clark -- You received this message because you are subscribed to th

Re: [prometheus-users] Reg: Prometheus aborting continuously

2021-02-16 Thread Stuart Clark
of Prometheus but the the issue still exists. Are you running out of memory / hitting a memory limit? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails fro

Re: [prometheus-users] How to scrape K8s pod metrics from prometheus running in VM instance

2021-02-12 Thread Stuart Clark
for setting those up within the cluster and updating Prometheus if the token changes. You might also need to set tls_config section depending on what CA you are using (to supply the CA certificate file). -- Stuart Clark -- You received this message because you are subscribed to the Google Groups

Re: [prometheus-users] Exporting custom metrics to remote storage from python code.

2021-02-12 Thread Stuart Clark
are just basic "write this value" or "run this query" type actions. You could look at the graphite bridge code and create something similar with your chosen storage client library. -- Stuart Clark -- You received this message because you are subscribed to the Google Gro

Re: [prometheus-users] Change in format of "startsAt" time.

2021-02-11 Thread Stuart Clark
olz PromLabs - promlabs.com -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion

Re: [prometheus-users] Prometheus.service startup failed

2021-02-11 Thread Stuart Clark
with incorrect command line options, invalid configuration file or corrupt data. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to promet

Re: [prometheus-users] Exporting custom metrics to remote storage from python code.

2021-02-11 Thread Stuart Clark
are looking to be using the Python Prometheus client library already what is the issue with scraping your application in the normal way? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from

Re: [prometheus-users] Federated Prometheus - Clarifications

2021-02-09 Thread Stuart Clark
act of server 3 breaking would be a gap in your query (or lower than expected aggregate values) while it was unavailable. There is no impact for any historical data before that time or data once the server is back. -- Stuart Clark -- You received this message because you are subscribed to the Go

Re: [prometheus-users] Trouble scraping Custom Python Metric in OpenShift 3.11 + Prometheus

2021-02-05 Thread Stuart Clark
ometheus_io_port]           action: replace           target_label: __address__           regex: (.+)(?::\d+);(\d+)           replacement: $1:$2         - action: labelmap           regex: __meta_kubernetes_service_label_(.+)         - source_labels: [__meta_kubernetes_namespace]           acti

Re: [prometheus-users] Re: err="opening storage failed: block dir: unexpected end of JSON input"

2021-02-02 Thread Stuart Clark
will need to remove that block and lose the data contained. If you have a backup you could try a recovery. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving e

Re: [prometheus-users] Re: err="opening storage failed: block dir: unexpected end of JSON input"

2021-02-02 Thread Stuart Clark
324267Z caller=main.go:517 msg="Scrape manager stopped" Feb 02 20:35:45 Hydrogen prometheus[437]: level=error ts=2021-02-02T20:35:45.347679163Z caller=main.go:686 err="opening storage failed: block dir: \"/var/lib/prometheus/metrics2/01EWX0Y64BRZ749

Re: [prometheus-users] Setting custom threshold for Disk Storage Space.

2021-02-02 Thread Stuart Clark
han simple % thresholds the predict_linear function can be very useful - alerting instead when space is predicted to be exhausted within a particular time period. See https://www.robustperception.io/tag/predict_linear -- Stuart Clark -- You received this message because you are subscrib

Re: [prometheus-users] looks like rate/irate/delta/increase are not calculating consistent

2021-02-01 Thread Stuart Clark
have a scrape that sometimes takes an extra second to return data). -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-user

Re: [prometheus-users] Remove the labels

2021-01-31 Thread Stuart Clark
& pod_template_hash) on the prometheus UI so these labels will not appear for any query. So as mentioned you can use the aggregation operators. For example: |sum without (instance, job, pod_template_hash) () | Replacing sum with max, min, avg, etc as appropriate. -- Stuart Clark --

Re: [prometheus-users] White box monitoring for Cloudera services

2021-01-28 Thread Stuart Clark
to remember is that once a scrape web request is received from Prometheus the exporter should connect to the external system to obtain the latest metric values to return (normally the values for "now"). It isn't possible to load in historical data. -- Stuart Clark -- You received th

Re: [prometheus-users] Remove the labels

2021-01-28 Thread Stuart Clark
, count, etc.) and the without() and by() clauses. Take a look here: https://prometheus.io/docs/prometheus/latest/querying/operators/#aggregation-operators -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubs

Re: [prometheus-users] Prometheus does not return divide result as a list

2021-01-26 Thread Stuart Clark
On 25/01/2021 23:42, Ricardo Estalder wrote: I think this is very complicated due The first query A returns 10 rows with label: metric_name A B C ... and the other query B, for example returns the sum of values of query A without labels so how will these match each other? If there are no

Re: [prometheus-users] Prometheus does not return divide result as a list

2021-01-25 Thread Stuart Clark
On 25/01/2021 23:08, ricardo@edge.com.py wrote: Hell Guys I have a doubts about using arithmetic operations in Prometheus Currently I have a metric *A* that return more than a single row with values, so when I try to divide these values of *A* by using the single result of another metric

Re: [prometheus-users] Trouble scraping Custom Python Metric in OpenShift 3.11 + Prometheus

2021-01-25 Thread Stuart Clark
On 25/01/2021 22:15, Jeff Tippey wrote: Hi,     I have a custom python container I wrote that exports metrics on port 8001 at "/metrics". The setup used to work in Openshift 3.6 with Prometheus.   I'm using the Prometheus configuration from github at prometheus github link

Re: [prometheus-users] Recording rule eval interval how longe should I can set? less than 5m?

2021-01-25 Thread Stuart Clark
On 25/01/2021 12:50, Allenzh li wrote: I want to count the average value of each hour of a metric and I have created a recording rule as follow: ```     - name: inspection-platform-CVMAndPhysical       interval: 1h       rules:         - record: vm:xxx_cpu_usage:avg1h           expr:

Re: [prometheus-users] Filtering data based on unique values

2021-01-25 Thread Stuart Clark
On 25/01/2021 04:28, Rohit Bharati wrote: Hi Everyone, I am setting up monitoring for my bamboo Job(CI/CD tool). For which I am using a nodeJS exporter ( I am new to it) which returns the following feilds:

Re: [prometheus-users] Behavior of Prometheus wrt to scraping and evaluating interval.

2021-01-23 Thread Stuart Clark
On 23/01/2021 13:15, akshay sharma wrote: Thanks for the clarification, It makes sense, but why we are deleting the data as soon as Prometheus scrapes becuase next time (at t15minutes) we can't register same metric ,as it is already registered at time t1. We are using Prometheus go client

Re: [prometheus-users] Behavior of Prometheus wrt to scraping and evaluating interval.

2021-01-23 Thread Stuart Clark
On 23/01/2021 10:55, akshay sharma wrote: Thanks for your reply. Actually the issue which are facing is , Let the exporter expose metric at every 15minutes, and let the Prometheus to scrape data every 2mintues. But suppose at t1, Prometheus scrapes data and gets the metrics with some value

Re: [prometheus-users] Behavior of Prometheus wrt to scraping and evaluating interval.

2021-01-23 Thread Stuart Clark
Removing the metric data shouldn't be done. The expectation is that you can query an exporter at any point and receive metrics. Ideally that would be instant data from the time of the scrape, but for processes that might only happen occasionally or queries which take a while to produce you

Re: [prometheus-users] Routes and receivers

2021-01-21 Thread Stuart Clark
On 21/01/2021 16:58, steve waldron wrote: Is it possible to add a label with a dynamic value? for example - the amount of time an alert has been firing?  if so, any examples of PROMQL queries to accomplish this? I'm trying to determine if routes can be created using a label defined by length

Re: [prometheus-users] Textfile Collector Using Multiple Shell Script

2021-01-21 Thread Stuart Clark
It should work with multiple files. Are there any errors logged by the node exporter? What are the command line options in use, and what is the content of both files? On 21 January 2021 14:48:20 GMT, Julian Ade Putra wrote: >Hi Expert, > >Currently, i have 2 shell script using textfile

Re: [prometheus-users] Pushgateway - any way to save/restore metrics

2021-01-15 Thread Stuart Clark
The push gateway supports a persistence file via the --persistence.file parameter to allow cached metrics to survive restarts. On 15 January 2021 09:46:18 GMT, Peter Toft wrote: >Hi guys > >I use a pushgateway -> prometheus -> grafana system to display >simulation >metrics, and when I reboot

Re: [prometheus-users] Uptime when no up metric is available

2021-01-14 Thread Stuart Clark
On 13/01/2021 20:31, Tom Gregory wrote: Hi Stuart. The metrics are coming from different scrapes. Each service is a different endpoint. I'm doing the relabelling using /metric_relabel_configs./ metric relabeling is really more for removing metrics (if you can't turn them off on the

Re: [prometheus-users] Re: amtool alertmanager config yml

2021-01-10 Thread Stuart Clark
On 10/01/2021 04:27, kiran wrote: Thank you, @Stuart Clark <mailto:stuart.cl...@jahingo.com> So it cannot work for multiple slacks. You will need to have multiple receivers as previously mentioned. -- You received this message because you are subscribed to the Google Groups "Prome

Re: [prometheus-users] Re: amtool alertmanager config yml

2021-01-09 Thread Stuart Clark
On 08/01/2021 22:32, kiran wrote: @Stuart Clark <mailto:stuart.cl...@jahingo.com> Here is what I tried. I am also attaching the version(*with slack variables*) which is not working. Any help resolving this is appreciated. *TEST#1* Using {{ .GroupLabels.slack_url }} for api_url witho

[prometheus-users] Re: amtool alertmanager config yml

2021-01-08 Thread Stuart Clark
Could you show the configuration you tried with the double quotes in place, as well as the error returned? On 8 January 2021 13:57:40 GMT, kiran wrote: >Yes I did both single and double it’s complaining about quotes in url > >On Friday, January 8, 2021, Stuart Clark >wrote: > &

Re: [prometheus-users] amtool alertmanager config yml

2021-01-08 Thread Stuart Clark
On 08/01/2021 06:22, kiran wrote: All I am trying to validate my alertmanager config yml file using *amtool*. amtool is throwing an error message 'cannot unmarshall !!map' with the following, but when I replace *api_url* with a slack_url(e.g https://hooks.slack.com/services///

Re: [prometheus-users] Disable blocks generation

2021-01-08 Thread Stuart Clark
On 08/01/2021 02:17, Mohan Nagandlla wrote: Hi team I have integrated the promethus with influxdb that is success a good in working on each and every second the metrics is exporting to influx so I don't need to store the metrics in prometheus db and don't need the blocks generation inside the

Re: [prometheus-users] Get the last value of a metric with indefinite time to look back

2021-01-05 Thread Stuart Clark
On 05/01/2021 15:35, Roman Dodin wrote: Hello community, I have a telemetry system that sends data into Prometheus when the data changes (i.e. its a push-on-event mechanism, not a sample/interval push) That means, if a system is stable, then the last reported value can be reported quite some

Re: [prometheus-users] Re: How to config prometheus to discovery ec2 instance.

2021-01-05 Thread Stuart Clark
On 05/01/2021 05:11, 辛二 wrote: config now: - *job_name*: 'ec2-sd'   proxy_url: '172.31.2.224:3001' *scrape_interval*: 5s *ec2_sd_configs*:       - *region*: cn-north-1 *access_key*: x *secret_key*: x *port*: 9100 *relabel_configs*:       - *source_labels*:

Re: [prometheus-users] Node Exporter - HTTP Security Headers Not Detected

2020-12-29 Thread Stuart Clark
quot; other than not returning them at all (the supported values indicate some sort of blocking), so adding them could quite easily break functionality. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe fr

Re: [prometheus-users] Querying prometheus data/series.

2020-12-24 Thread Stuart Clark
On 24/12/2020 11:37, akshay sharma wrote: 5:04 PM (0 minutes ago) I understood your point Now i've one metric and with multiple values updating like every minute, if you range over you will see all the values.

Re: [prometheus-users] Querying prometheus data/series.

2020-12-24 Thread Stuart Clark
On 24/12/2020 05:53, akshay sharma wrote: Thanks for the clarification, What if,there is multiple metrics with same labels but different values ex: STAT{METype="NT",ResourceID="t",queue="rx",time="1608786795151775007"} 31 STAT{METype="NT",ResourceID="t",queue="rx",time="1608786795151813374"}

Re: [prometheus-users] Querying prometheus data/series.

2020-12-23 Thread Stuart Clark
On 23/12/2020 15:18, akshay sharma wrote: Hi Christian, Thanks for you reply, ONTSTAT{IDF="false",Interval="0",METype="NT",ResourceID="t",instance="localhost:8999 ",job="prometheus",rxByte="33",time="1608711202557257990",txByte="18"}  4 what i've understood

Re: [prometheus-users] Timestamp value with custom metrics

2020-12-23 Thread Stuart Clark
On 23/12/2020 14:59, Pramod Kumar wrote: Hi Clark, Thank you for quick reply. Actually, in my case, the application is generating some numbers (with timestamp) and pushing inside file. Now, this number has to be consider as counters and shown in prometheus along with the below timestamp.

Re: [prometheus-users] Timestamp value with custom metrics

2020-12-23 Thread Stuart Clark
You will get timing information from Prometheus automatically. Every time your application is scraped the current value of the counter is stored against the scrape time, so if the previous scrape had a value of 100 and it is now 200, you know the increase happened between the two scrapes. For

Re: [prometheus-users] Error encoding and sending metrics family - Node Exporter.

2020-12-22 Thread Stuart Clark
On 22/12/2020 13:25, yagyans...@gmail.com wrote: Hi. I am running node_exporter version 0.18.1 on around 2500 servers. On some of the servers I am observing that node_exporter is throwing error intermittently for sending the metrics to Prometheus server. Dec 22 17:10:29 hostname

Re: [prometheus-users] Replicas more than 2 corrupting the wal directory

2020-12-22 Thread Stuart Clark
On 22/12/2020 04:52, Mohan Nagandlla wrote: HI team I am using the Prometheus instance having the more than 1 replica, When replica as 1 there is no wal corruption in data directory and now for the sake of zero down time updates for instance I make the replicas count as 2 the instance is up

Re: [prometheus-users] Pushing custom metrics to avoid AKS load balancer

2020-12-21 Thread Stuart Clark
, Marcin Burakiewicz wrote: >What do you mean by "scraping"? How the architecture of this solution >should look like? > >poniedziałek, 21 grudnia 2020 o 14:01:26 UTC+1 Stuart Clark napisał(a): > >> On 21/12/2020 10:34, Marcin Burakiewicz wrote: >> >> Exa

Re: [prometheus-users] Pushing custom metrics to avoid AKS load balancer

2020-12-21 Thread Stuart Clark
On 21/12/2020 10:34, Marcin Burakiewicz wrote: Exactly. So how can this be done? I searched for answer here https://prometheus.io/docs/instrumenting/pushing/, but I could not find information how to send to PushGateway the metrics which are already exposed via http in prometheus format.

Re: [prometheus-users] Best practice for Prometheus + GCP Cloud Run

2020-12-18 Thread Stuart Clark
On 18/12/2020 14:50, Rohit Ramkumar wrote: Hi, I'm running a service in Cloud Run (https://cloud.google.com/run) and wondering what the best practice is here for setting up Prometheus. Specifically, I'm wondering how to handle the case when there are multiple container instances running

Re: [prometheus-users] Scraping Kubernetes metrics from standalone Prometheus Instance

2020-12-18 Thread Stuart Clark
On 18/12/2020 06:03, Sakthi Raam wrote: Hi All, We have restrictions to deploy Prometheus inside Kubernetes environment due to shared infrastructure but we still need to scrape all the node/services/endpoints/pod level metrics using service discovery approach. When I checked almost all the

<    1   2   3   4   5   6   >