owed by exactly 2
numbers. So the first one would also match "50" or "5frogs" which aren't
valid status codes, but in reality your application would have to be
having serious problems to be setting those values anyway.
--
Stuart Clark
--
You received this message because you a
h;
Not really a Prometheus thing, but that isn't a valid ConfigMap. It is
expecting top level fields called "metatdata" and "data", with a
filename within the data section that then contains whatever data you
are wanting (which for Prometheus is a YAML config file).
--
) the Prometheus server will either need to be
inside the cluster or connected to the same network mechanism as the
pods.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiv
that the average job runtime is 5 minutes over
the past 3 hours, but you'd then use the events system to find the exact
durations for each run (or the number of events processed, or the error
message returned, etc.).
--
Stuart Clark
--
You received this message because you are subscribed to the Google
to 2 minutes. If you need to know when that set of metrics was
created a common pattern is to include a metric who's value is the
timestamp.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from
could have had a web scan
producing millions of combinations)
* lots of changes to the targets, such as new instances of software or
high churn of applications restarting
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" g
e? Perhaps pull the Activity.Current trace id that is
used for exemplars? Not sure what the best practices are here.
This sounds like you might actually want traces rather than just
metrics, which can be added using Tempo or other tools.
--
Stuart Clark
--
You received this message b
On 25/05/2023 15:59, Yogita Bhardwaj wrote:
How can i apply rate or increase while using http api.
You just need to include it in the query you are sending the Prometheus API.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus
, so you
don't need to do anything. Prometheus handles counter resets
automatically when you are using functions like rate().
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop
* source_labels: [__param_target]
target_label: instance * target_label: ADDRESS
Samples still every 15s.
2023-05-15 09:45:00
0.0624
2023-05-15 09:45:15
How are you producing that data?
Did you remember to reload/restart Prometheus after making the change to
your config?
--
Stuart Clark
t system, such as Loki, Elasticsearch or a SQL database.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googleg
to be running multiple instances of the exporter.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@google
&
details, etc.) then no, as Prometheus is a metric system. Tempo is the
tracing system from Grafana that can work alongside Prometheus (and
there are other OS and commercial offerings too).
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"P
due to different job names or use of labels).
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups
That's definitely something that can be done _using_ Prometheus, but not
something done _within_ it.
You'd have an application which uses the query API (or is sent live data via
remote write) to fetch metrics and then does whatever calculations are needed
(for example using machine learning
The timestamp feature is only really designed for limited situations, such as
ingesting metrics from another metric system. Scraping is generally expected to
be returning the metric value from "now" (or in the case of coming via another
metric system within the last few minutes). As Prometheus
at I missed in the config?
How have you configured the Windows machine? Have you just imported the
CA into Windows, or did you generate a client certificate and import /
configure that too?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"
have to wait a short amount of time before metrics
start appearing - although often you need to wait for at least a few
scrapes to be able to do things like looking at counter increase rates,
etc.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
ch you can then
include in email/ticket subjects.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googl
what you are doing, what is happening & what you
are wanting instead someone might be able to suggest something...
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receivi
. Can it be removed?
What makes it problematic?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-user
anything longer than that will likely result in what
you are seeing.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-user
here .
I would suggest using the file SD option and then having a tool creating
a JSON/YAML file containing the servers to scrape.
Is there really no list of those servers? For example are you using
something like Ansible or Chef to provision/maintain them?
--
Stuart Clark
--
You received
Could anyone help me?
What is the contents of those files in /var/lib/node_exporter?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an emai
computer is away?
Do you mean you are switching off the machine which is running Blackbox
Exporter or Prometheus?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving e
etric like this:
avg_over_time(up{job="node"}[30d]) * 100
You would need to adjust the query to be
avg_over_time(up{job="node"}[28d]) * 100
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe f
relabelling rules applied for that
particular job only. The only way you can set relabelling rules for a
job is by editing the job config. For the Prometheus Operator this is
done by adjusting the PodMonitor/ServiceMonitor objects.
--
Stuart Clark
--
You received this message because you
ures.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the we
g jobs were added
as part of loading prometheus-operator. I'm not sure I can change those.
The other jobs are probably from PodMonitor & ServiceMonitor objects, so
you'd need to adjust those.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"P
On 20/02/2023 22:33, Jihui Yang wrote:
I think these metrics are being scraped from another job. What I want
is to drop any scraped metrics with names match the regex I provided
Then you need to add the relabel config to that other job.
--
Stuart Clark
--
You received this message because you
s msg="Scrape failed"
err="server returned HTTP status 400 Bad Request"
```
The metrics are still not dropped
I'm not really following exactly what your config is.
Those errors suggest that at least some of the scrapes are failing.
When you say "the metrics are still
o tell Prometheus where to scrape from.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegro
be different.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion
config file as well still getting the same error 404
I'm not quite sure what you are meaning by "without a port number"? All
connections are to a specific port. If you don't specify a port when
doing a web request it will default to 80 or 443 (depending on if it is
HTTP or HTTPS).
time the service is
stop/start. The hostname stays the same.
Is there another exporter that I can use? or might know of an
workaround that would work in this case. Since the url change each
time, blackbox exporter cannot be used in this instance.
How would you know what port to use?
--
Stuart
e="603k09311-9-bjsimu01",instance="10.253.108.171:9100",ip="10.253.108.171",job="node-exporter",mode="iowait",namespace="product-coc-monitor",pod="coc-monitor-prometheus-node-exporter-c2plp",service="coc-monitor-prometheus-nod
a
YAML/JSON file or changing AWS/Kubernetes tags.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@google
better fit for you than a timeseries database
and ecosystem that is designed for handling metrics.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, sen
every
>> 10 days that alert is firing, meaning you'd get a message at 60, 50, 40,
>> etc days until expiry.
>>
>> More complex alerting routing decisions are generally out of scope for
>> Alertmanager and would be expected to be managed by a more complex
>&
, Grafana On-Call, etc.). This would
cover you example of wanting to escalate an alert after a period of
time, but would also cover things like having on-call rotas where
different people would be contacted by looking at a rota calendar.
--
Stuart Clark
--
You received this message because you
aml in the directory
/etc/prometheus-shared doesn't exist, which is the configuration file
for Prometheus.
You need to either ensure the file does exist at that location or adjust
the command line to give the correct location if different.
--
Stuart Clark
--
You received this message becaus
ting & what you actually see?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To vi
It is a requirement that counters end with _total which is likely why it is
being added for you.
On 14 December 2022 17:50:32 GMT, "arnav...@gmail.com"
wrote:
>Hi,
>
>I am exposing two counter metrics from my spring boot application. The
>metrics are getting generated and the counters
't listing a metric here as you are using !~. You need to ensure
you are only using = in any labels.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails fr
ric as labels. These need splitting out into their own separate
metrics (e.g. NNI_RTT_Min and Packet_Loss). As it stands you are
creating new time series for basically every scrape, which results in
the lines you are seeing in your graph. Also as it is a cardinality
issue it will cause problems
al it would actually be more useful to
alert if that threshold is breached for over say 5 minutes, or even more
usefully when an SLO is failed or is projected to fail within the next
30 minutes.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
&quo
metrics_path: '/federate'
scheme : http
params:
'match[]':
- '{job!=""}'
The alert rules are set in each Prometheus, so if you are not wanting
them you'd need to look at the configuration and change/remove.
--
Stuart Clark
--
You received this message b
://promcon.io/2022-munich/talks/why-is-it-so-big-analysing-the-m/)
which is available on YouTube, but basically the main memory usage is
generally due to the number of time series, with only a very minor usage
due to queries.
--
Stuart Clark
--
You received this message because you are subscribed
point with the Blackbox Exporter.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view
?
Where are you looking in the UI and what is showing?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-user
, which are very
different and not limited to number (e.g. logs).
Maybe it would be best to describe what you are trying to achieve?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this grou
lobal
storage) it makes sense to support both, there are plenty of use cases
which only require one or the other.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving email
Khazamov wrote:
>Thanks, I'm importing metrics from our internal metrics system. Do you have
>any advice on how to push withthe explicit timestamps?
>
>Le ven. 7 oct. 2022 à 13:30, Stuart Clark a
>écrit :
>
>> On 07/10/2022 10:16, Omar Khazamov wrote:
>>
(which
then would allow you to alert if one method stops working), but order ID
isn't something you should ever put in the metrics. Instead once you
were alerted about a potential issue you might query your order database
directly or look at log files to dig into the detail and figure out what
is
. You would need to look
at the server configuration to whitelist the IP of the Blackbox Exporter
so it doesn't get blocked.
Alternatively look at using the MySQL Exporter instead of using Blackbox
Exporter, which will give you more insight above just availability too?
--
Stuart Clark
--
You
always be publishing the "latest" value of any metric when
Prometheus scrapes the endpoint (or the push gateway in this case).
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from t
cal "error" and "debug"
levels? Are you meaning some metrics would only be exposed on some
environments?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop
/querying/api/#delete-series
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.co
ystem was configured.
If you have something stored in the local Prometheus store (rather than
something you are using remote read with) and don't want to wait until
it expires (14 days or whatever you configured) you can use the deletion
API.
--
Stuart Clark
--
You received this message becaus
alert grouping config.
In general if you are sending alerts through to an external incident
management system (PagerDuty, OpsGenie, JIRA, etc.) you are probably
best not doing any grouping at all:
|group_by: ['...']|
--
Stuart Clark
--
You received this message because you are subscribed to the Goog
hy you think resetting metrics is important / needed?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@g
completely in an unreachable state.
So you are only wanting it if every single scrape failed over the past 2
days?
Try sum() instead of max_over_time().
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubsc
specific HTTP response code was returned from a web call, etc.
What is the value of the metric app_version supposed to signify?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and st
:
https://prometheus.io/docs/prometheus/latest/querying/api/
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-user
On 12/08/2022 08:29, nina guo wrote:
So if with push gateway or textfile collector, we need to also to
customize our metrics, am I right?
What do you mean?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To u
f the last time the metric was updated.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googl
onger interval you will end up with stale time
series and "gaps" in all your graphs.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
+'='+ldap_search_base+','+'}
'+str(query_check),
file=open("/var/log/node_exporter/filecollector/ldap_query.prom", "a+"))
File mode "a" will open for appending (so preserve anything already in
the file). Instead to fully replace the file you'd need to use file mode
&qu
a random non *.prom filename and then rename it into the correct
name - renames are generally atomic whereas file updates often aren't,
meaning if you modify the file directly you could end up reading part
finished data.
--
Stuart Clark
--
You received this message because you are subscribe
t able to produce a
histogram in the application (or externally via processing an event
feed, such as logs)? By design a simple histogram is pretty low impact,
being a set of counters for each bucket.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prom
le - which is only possible in the application, or if you store
events elsewhere (e.g. in log files) in a tool that connects to your
event store system.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubsc
t; would be returned, and you'd have gaps
in your graphs. The recommended maximum of about 2-2.5 minutes is to
allow for a single scrape failure not to result in gaps as well as all
the various processing times to actually do the scrape.
--
Stuart Clark
--
You received this message becaus
On 25/07/2022 08:28, nina guo wrote:
And one more question pls, I checked the log that the probe is sending
every 2-3 seconds, can I adjust this frequency to about 1min?
Yes that's the scrape frequency, so you can adjust the job configuration
up to a maximum of about 2.5 minutes.
--
Stuart
On 25/07/2022 01:08, nina guo wrote:
Thank you Brian. " up to T - 5 minutes ", this 5 mins is the scraping
interval?
No. The scraping interval doesn't matter. Prometheus will by default
look back at most 5 minutes for a value.
--
Stuart Clark
--
You received this message b
that helps.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on
for a container.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion
queries won't have any data to query.
However Prometheus itself creates certain metrics for all scrape
targets, including one called "up" which is either 0 or 1 - where 0
means the scrape failed. You can therefore create dashboards and alerts
that list the servers which aren't accessi
ld it be that it takes a while and your load balancer is configured
with a shorter timeout?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails fro
eve it would help us advise better?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To
wouldn't actually help BTW as the statusNumber is a
label so you could only use regex matches rather than >/<. If you wanted
that as well you'd need a separate metric
(idrac_amperage_probe_status_number or something) that has no labels and
just the 1-10 value.
The value of that purely numeri
sing agent mode?
If you are just going to copy everything to the "central" instance on
the same cluster, why not just do without the extra three clusters and
have just the one instance that monitors everything?
--
Stuart Clark
--
You received this message because you are subscribed
a good or bad thing depending on team responsibilities/timings.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-user
). It is a one-way process - the webhook
is triggered when an alert fires (and optionally clears). I'm not really
sure what you are hoping for?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this grou
Two alerts suggests that the two instances aren't talking to each other. How
have you configured them? Does the UI show the "other" instance?
On 5 July 2022 08:34:45 BST, Venkatraman Natarajan wrote:
>Thanks Brian. I have used last_over_time query in our expression instead of
>turning off
What are you meaning by "AM executor"? Are you meaning the Alertmanager
application?
If so, there are both community Helm charts and a Docker image available, so
running within Kubernetes should be no problem.
On 5 July 2022 08:03:45 BST, Test Kumar wrote:
>Hi Team,
>
>I installed the AM
uld
you expect timestamps to be attached to the returned metrics.
It would be good to hear a bit more about what you are trying to do, as
it is highly likely that the use of timestamps in your use case is
probably not the right option and they should just be dropped.
--
Stuart Clark
--
You rec
HTTP status code, section of website,
etc.). What is it about this system that you think it needs very high
cardinality labels?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and
.
Would you be able to explain a bit more about what you are wanting to be
able to achieve and we can suggest an alternative to Prometheus/metrics
that would help?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To u
, and you are unlikely to recenive any
free/paid support in that architecture.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to promet
it won't work. However
there are issues that people find, and if you do find those the
recommendation would always be to switch the filesystem to something
local. So basically you'd be fairly on your own if you did have problems.
--
Stuart Clark
--
You received this message because you
no
Internet access? There is nothing within Prometheus which should require
such access. What isn't working? What does the target page show
regarding the things you are scraping?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users&q
asn't been exposed to
the outside host.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googl
an error, which should be visible via
something like the "up" metric).
As the standard staleness interval is 5 minutes it is recommended that
the maximum scrape period should be no more that 2 minutes (to allow for
a failed scrape without the time series being marked as stale).
--
S
your Prometheus retention period to? By default it is
2 weeks.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-user
not be an integer (due
to the way interpolation happens).
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-user
that isn't changing is next to nothing,
so I wouldn't worry about it. What you describe would be exactly how I'd
expect Pushgateway to be behaving - some metrics are updated more
frequently and others less, but they are always there and being scraped
at the same frequency.
--
Stuart Clark
, due to staleness
- less frequent scrapes would likely cause series to be marked as stale
and therefore disappear from alerts/dashboards.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this grou
Teams), but
I'm not sure if a Sendgrid one has already been written.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-user
details of what is going on
on the target server itself, not so concerned about what the
Prometheus server has log wise.
I was talking about logs from the application being scraped.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus
.
I'm not clear if you are getting logs from these requests or not? I'd
expect any request logs to include the path being requested, time &
source IP. What do you see?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Use
1 - 100 of 509 matches
Mail list logo
