Re: [prometheus-users] Create custom metrics and add it to prometheus

[BHARATH KUMAR]

The main issue is that I am not able to see the metrics in Prometheus UI. I 
am unable to understand the root cause.

Am I missing anything?

On Tuesday, 28 February 2023 at 23:44:59 UTC+5:30 Brian Candler wrote:

> Aside: you might also want to look at kube-state-metrics 
> , which can give you a 
> range of pod metrics 
> 
>  
> including the pod start time. Then an expression like 
> time()-kube_pod_start_time should give you the pod age.
>
> On Tuesday, 28 February 2023 at 17:55:20 UTC Brian Candler wrote:
>
>> The problem is that the filename must end with ".prom", i.e. you need to 
>> create "apt1.prom" not "apt1-prom"
>>
>> Also, you need to enable the functionality with a flag to node_exporter:
>>
>> node_exporter --collector.textfile.directory=/var/lib/node_exporter
>>
>> See the documentation here: 
>> https://github.com/prometheus/node_exporter#textfile-collector
>>
>> On Tuesday, 28 February 2023 at 17:16:06 UTC Stuart Clark wrote:
>>
>>> On 28/02/2023 17:12, BHARATH KUMAR wrote: 
>>> > Hello All, 
>>> > 
>>> > I want to calculate the pod age running on every server. I wrote a 
>>> > shell script. I added this script under the folder 
>>> /var/lib/node_exporter. 
>>> > 
>>> > I created a cronjob for this script to run every minute 
>>> > * * * * * root bash /var/lib/node_exporter/custom_metrics.sh > 
>>> > /var/lib/node_exporter/apt1-prom 
>>> > 
>>> > I store the above cronjob in the /etc/cron.d folder with filename: 
>>> > prom-apt1. 
>>> > 
>>> > But I am not able to see the metrics I created in Prometheus UI. 
>>> > 
>>> > 
>>> > 
>>> > But similarly, I created another shell script file to fetch some 
>>> > metrics. I followed the same procedure as above. 
>>> > 
>>> > * * * * * root bash /var/lib/node_exporter/custom1.sh > 
>>> > /var/lib/node_exporter/apt-prom 
>>> > 
>>> > I store the above cronjob in the /etc/cron.d folder with filename: 
>>> > prom-apt. 
>>> > 
>>> > The metrics which I mentioned in custom1.sh, I am able to see those 
>>> > metrics in Prometheus UI. 
>>> > 
>>> > 
>>> > Could anyone help me? 
>>> > 
>>> What is the contents of those files in /var/lib/node_exporter? 
>>>
>>> -- 
>>> Stuart Clark 
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/248cb5ff-0c32-43e6-88cc-e16600e6ed18n%40googlegroups.com.

Re: [prometheus-users] Create custom metrics and add it to prometheus

[BHARATH KUMAR]

In /var/lib/node_exporter --> In this folder, the scripts(custom metrics 
script) will be available.


On Tuesday, 28 February 2023 at 22:46:06 UTC+5:30 Stuart Clark wrote:

> On 28/02/2023 17:12, BHARATH KUMAR wrote:
> > Hello All,
> >
> > I want to calculate the pod age running on every server. I wrote a 
> > shell script. I added this script under the folder 
> /var/lib/node_exporter.
> >
> > I created a cronjob for this script to run every minute
> > * * * * * root bash /var/lib/node_exporter/custom_metrics.sh > 
> > /var/lib/node_exporter/apt1-prom
> >
> > I store the above cronjob in the /etc/cron.d folder with filename: 
> > prom-apt1.
> >
> > But I am not able to see the metrics I created in Prometheus UI.
> >
> >
> >
> > But similarly, I created another shell script file to fetch some 
> > metrics. I followed the same procedure as above.
> >
> > * * * * * root bash /var/lib/node_exporter/custom1.sh > 
> > /var/lib/node_exporter/apt-prom
> >
> > I store the above cronjob in the /etc/cron.d folder with filename: 
> > prom-apt.
> >
> > The metrics which I mentioned in custom1.sh, I am able to see those 
> > metrics in Prometheus UI.
> >
> >
> > Could anyone help me?
> >
> What is the contents of those files in /var/lib/node_exporter?
>
> -- 
> Stuart Clark
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/85673771-c5fb-45c1-bb27-f71cca079407n%40googlegroups.com.

Re: [prometheus-users] Create custom metrics and add it to prometheus

[BHARATH KUMAR]

Sorry for the mistake in the above message. I created the filename with 
".prom" only.

I added the --collector.textfile.directory=/var/lib/node_exporter as one of 
the scripts was working fine. but another file was issue.
On Tuesday, 28 February 2023 at 23:25:20 UTC+5:30 Brian Candler wrote:

> The problem is that the filename must end with ".prom", i.e. you need to 
> create "apt1.prom" not "apt1-prom"
>
> Also, you need to enable the functionality with a flag to node_exporter:
>
> node_exporter --collector.textfile.directory=/var/lib/node_exporter
>
> See the documentation here: 
> https://github.com/prometheus/node_exporter#textfile-collector
>
> On Tuesday, 28 February 2023 at 17:16:06 UTC Stuart Clark wrote:
>
>> On 28/02/2023 17:12, BHARATH KUMAR wrote: 
>> > Hello All, 
>> > 
>> > I want to calculate the pod age running on every server. I wrote a 
>> > shell script. I added this script under the folder 
>> /var/lib/node_exporter. 
>> > 
>> > I created a cronjob for this script to run every minute 
>> > * * * * * root bash /var/lib/node_exporter/custom_metrics.sh > 
>> > /var/lib/node_exporter/apt1-prom 
>> > 
>> > I store the above cronjob in the /etc/cron.d folder with filename: 
>> > prom-apt1. 
>> > 
>> > But I am not able to see the metrics I created in Prometheus UI. 
>> > 
>> > 
>> > 
>> > But similarly, I created another shell script file to fetch some 
>> > metrics. I followed the same procedure as above. 
>> > 
>> > * * * * * root bash /var/lib/node_exporter/custom1.sh > 
>> > /var/lib/node_exporter/apt-prom 
>> > 
>> > I store the above cronjob in the /etc/cron.d folder with filename: 
>> > prom-apt. 
>> > 
>> > The metrics which I mentioned in custom1.sh, I am able to see those 
>> > metrics in Prometheus UI. 
>> > 
>> > 
>> > Could anyone help me? 
>> > 
>> What is the contents of those files in /var/lib/node_exporter? 
>>
>> -- 
>> Stuart Clark 
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/96ebb6eb-31b3-44e5-b8cc-adf6b94cf1a7n%40googlegroups.com.

Re: [prometheus-users] Custometrics 503 error

[Rohit Mittapalli]

Hey Kumar,

Were you able to get this resolved? Seeing a similar issue on our end.

On Sunday, November 22, 2020 at 7:38:51 PM UTC-8 kumar k wrote:

> When i do describe API,I am seeing 404 error.
> Can someone please help
>
> kubectl describe apiservice v1beta1.custom.metrics.k8s.io
> Name: v1beta1.custom.metrics.k8s.io
> Namespace:
> Labels:   
> Annotations:  
> API Version:  apiregistration.k8s.io/v1
> Kind: APIService
> Metadata:
>   Creation Timestamp:  2020-11-20T22:20:54Z
>   Resource Version:126789181
>   Self Link:   /apis/
> apiregistration.k8s.io/v1/apiservices/v1beta1.custom.metrics.k8s.io
>   UID: 06b5e355-8a67-4f17-b1ec-7c655d33e83e
> Spec:
>   Group: custom.metrics.k8s.io
>   Group Priority Minimum:100
>   Insecure Skip TLS Verify:  true
>   Service:
> Name:monitoring-base-prometheus-adapter
> Namespace:   dock
> Port:443
>   Version:   v1beta1
>   Version Priority:  100
> Status:
>   Conditions:
> Last Transition Time:  2020-11-20T22:20:54Z
> Message:   failing or missing response from 
> https://:6443/apis/custom.metrics.k8s.io/v1beta1: bad status from 
> https://:6443/apis/custom.metrics.k8s.io/v1beta1: 404
> Reason:FailedDiscoveryCheck
> Status:False
> Type:  Available
> Events:
>
> On Sun, Nov 22, 2020 at 10:29 PM kumar k  wrote:
>
>> Hi,
>>
>> I am getting below error for custom metrics on prometheus adaptor.
>> Can someone please help?
>>
>> kubectl get apiservice
>>
>> v1beta1.custom.metrics.k8s.io  dock/prometheus-adapter   False 
>> (FailedDiscoveryCheck)   2d4h
>>
>> kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 -v9
>> I1122 22:12:38.509257   17365 loader.go:375] 
>> I1122 22:12:38.511514   17365 round_trippers.go:423] curl -k -v -XGET  -H 
>> "Accept: application/json, */*" -H "User-Agent: kubectl/v1.18.3 
>> (darwin/amd64) kubernetes/2e7996e" '
>> https://XX.us-east-1.eks.amazonaws.com/apis/custom.metrics.k8s.io/v1beta1
>> '
>> I1122 22:12:39.614349   17365 round_trippers.go:443] GET 
>> https://XX.us-east-1.eks.amazonaws.com/apis/custom.metrics.k8s.io/v1beta1
>>  
>> 503 Service Unavailable in 1102 milliseconds
>> I1122 22:12:39.614434   17365 round_trippers.go:449] Response Headers:
>> I1122 22:12:39.614457   17365 round_trippers.go:452] Content-Length: 
>> 20
>> I1122 22:12:39.614474   17365 round_trippers.go:452] Date: Mon, 23 
>> Nov 2020 03:12:39 GMT
>> I1122 22:12:39.614492   17365 round_trippers.go:452] Audit-Id: 
>> f167be23-5f10-451c-b423-9ed6362cb672
>> I1122 22:12:39.614515   17365 round_trippers.go:452] Cache-Control: 
>> no-cache, private
>> I1122 22:12:39.614537   17365 round_trippers.go:452] Content-Type: 
>> text/plain; charset=utf-8
>> I1122 22:12:39.614566   17365 round_trippers.go:452]
>>  X-Content-Type-Options: nosniff
>> I1122 22:12:39.615355   17365 request.go:1068] Response Body: service 
>> unavailable
>> I1122 22:12:39.616357   17365 helpers.go:216] server response object: [{
>>   "metadata": {},
>>   "status": "Failure",
>>   "message": "the server is currently unable to handle the request",
>>   "reason": "ServiceUnavailable",
>>   "details": {
>> "causes": [
>>   {
>> "reason": "UnexpectedServerResponse",
>> "message": "service unavailable"
>>   }
>> ]
>>   },
>>   "code": 503
>> }]
>> F1122 22:12:39.616722   17365 helpers.go:115] Error from server 
>> (ServiceUnavailable): the server is currently unable to handle the request
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/ce7ff7eb-cf6d-47a2-88cf-7323e805c732n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/e17f3ff1-98db-492c-8a2a-3b855078a5b0n%40googlegroups.com.

Re: [prometheus-users] Create custom metrics and add it to prometheus

[Brian Candler]

Aside: you might also want to look at kube-state-metrics 
, which can give you a 
range of pod metrics 

 
including the pod start time. Then an expression like 
time()-kube_pod_start_time should give you the pod age.

On Tuesday, 28 February 2023 at 17:55:20 UTC Brian Candler wrote:

> The problem is that the filename must end with ".prom", i.e. you need to 
> create "apt1.prom" not "apt1-prom"
>
> Also, you need to enable the functionality with a flag to node_exporter:
>
> node_exporter --collector.textfile.directory=/var/lib/node_exporter
>
> See the documentation here: 
> https://github.com/prometheus/node_exporter#textfile-collector
>
> On Tuesday, 28 February 2023 at 17:16:06 UTC Stuart Clark wrote:
>
>> On 28/02/2023 17:12, BHARATH KUMAR wrote: 
>> > Hello All, 
>> > 
>> > I want to calculate the pod age running on every server. I wrote a 
>> > shell script. I added this script under the folder 
>> /var/lib/node_exporter. 
>> > 
>> > I created a cronjob for this script to run every minute 
>> > * * * * * root bash /var/lib/node_exporter/custom_metrics.sh > 
>> > /var/lib/node_exporter/apt1-prom 
>> > 
>> > I store the above cronjob in the /etc/cron.d folder with filename: 
>> > prom-apt1. 
>> > 
>> > But I am not able to see the metrics I created in Prometheus UI. 
>> > 
>> > 
>> > 
>> > But similarly, I created another shell script file to fetch some 
>> > metrics. I followed the same procedure as above. 
>> > 
>> > * * * * * root bash /var/lib/node_exporter/custom1.sh > 
>> > /var/lib/node_exporter/apt-prom 
>> > 
>> > I store the above cronjob in the /etc/cron.d folder with filename: 
>> > prom-apt. 
>> > 
>> > The metrics which I mentioned in custom1.sh, I am able to see those 
>> > metrics in Prometheus UI. 
>> > 
>> > 
>> > Could anyone help me? 
>> > 
>> What is the contents of those files in /var/lib/node_exporter? 
>>
>> -- 
>> Stuart Clark 
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/84408ccb-6753-41ec-94db-95cc404baa6en%40googlegroups.com.

Re: [prometheus-users] Create custom metrics and add it to prometheus

[Brian Candler]

The problem is that the filename must end with ".prom", i.e. you need to 
create "apt1.prom" not "apt1-prom"

Also, you need to enable the functionality with a flag to node_exporter:

node_exporter --collector.textfile.directory=/var/lib/node_exporter

See the documentation 
here: https://github.com/prometheus/node_exporter#textfile-collector

On Tuesday, 28 February 2023 at 17:16:06 UTC Stuart Clark wrote:

> On 28/02/2023 17:12, BHARATH KUMAR wrote:
> > Hello All,
> >
> > I want to calculate the pod age running on every server. I wrote a 
> > shell script. I added this script under the folder 
> /var/lib/node_exporter.
> >
> > I created a cronjob for this script to run every minute
> > * * * * * root bash /var/lib/node_exporter/custom_metrics.sh > 
> > /var/lib/node_exporter/apt1-prom
> >
> > I store the above cronjob in the /etc/cron.d folder with filename: 
> > prom-apt1.
> >
> > But I am not able to see the metrics I created in Prometheus UI.
> >
> >
> >
> > But similarly, I created another shell script file to fetch some 
> > metrics. I followed the same procedure as above.
> >
> > * * * * * root bash /var/lib/node_exporter/custom1.sh > 
> > /var/lib/node_exporter/apt-prom
> >
> > I store the above cronjob in the /etc/cron.d folder with filename: 
> > prom-apt.
> >
> > The metrics which I mentioned in custom1.sh, I am able to see those 
> > metrics in Prometheus UI.
> >
> >
> > Could anyone help me?
> >
> What is the contents of those files in /var/lib/node_exporter?
>
> -- 
> Stuart Clark
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/fd4d0b99-2828-4ac0-a34f-585e5cfef00cn%40googlegroups.com.

Re: [prometheus-users] Create custom metrics and add it to prometheus

[Stuart Clark]

On 28/02/2023 17:12, BHARATH KUMAR wrote:

Hello All,

I want to calculate the pod age running on every server. I wrote a 
shell script. I added this script under the folder /var/lib/node_exporter.


I created a cronjob for this script to run every minute
* * * * * root bash /var/lib/node_exporter/custom_metrics.sh > 
/var/lib/node_exporter/apt1-prom


I store the above cronjob in the /etc/cron.d folder with filename: 
prom-apt1.


But I am not able to see the metrics I created in Prometheus UI.



But similarly, I created another shell script file to fetch some 
metrics. I followed the same procedure as above.


* * * * * root bash /var/lib/node_exporter/custom1.sh > 
/var/lib/node_exporter/apt-prom


I store the above cronjob in the /etc/cron.d folder with filename: 
prom-apt.


The metrics which I mentioned in custom1.sh, I am able to see those 
metrics in Prometheus UI.



Could anyone help me?


What is the contents of those files in /var/lib/node_exporter?

--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/34cc064c-d700-931b-d912-1f9dd256b14c%40Jahingo.com.

[prometheus-users] Create custom metrics and add it to prometheus

[BHARATH KUMAR]

Hello All,

I want to calculate the pod age running on every server. I wrote a shell 
script. I added this script under the folder /var/lib/node_exporter.

I created a cronjob for this script to run every minute 
* * * * * root bash /var/lib/node_exporter/custom_metrics.sh > 
/var/lib/node_exporter/apt1-prom

I store the above cronjob in the /etc/cron.d folder with filename: 
prom-apt1.

But I am not able to see the metrics I created in Prometheus UI.



But similarly, I created another shell script file to fetch some metrics. I 
followed the same procedure as above.

* * * * * root bash /var/lib/node_exporter/custom1.sh > 
/var/lib/node_exporter/apt-prom

I store the above cronjob in the /etc/cron.d folder with filename: prom-apt.

The metrics which I mentioned in custom1.sh, I am able to see those metrics 
in Prometheus UI.


Could anyone help me?

Thanks & regards,
Bharath Kumar.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/20ea101b-f81e-47d8-be22-543030ca5771n%40googlegroups.com.

[prometheus-users] transitivity of remote read/write api

[Johny]

experimentally, I'm able to read data from downstream Prometheus instances 
with 3-4 levels between them. For example, consider this for simplicity
PromA   *-remote-read> *PromB *remote-read->* 
ExternalStorage

I am able to get data points in PromA that are fetched remotely from 
ExternalStorage. Similarly, I am able to write data points from PromA to 
ExternalStraoge.

Is there any caveat with this approach besides reliability issues in 
introducing multiple points of failure on the read/write path?


-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/d9ad025b-fbf8-4910-91d9-78aeebaf5dbcn%40googlegroups.com.

Re: [prometheus-users] fading out sample resolution for samples from longer ago possible?

[Ben Kochie]

On Tue, Feb 28, 2023 at 1:45 AM Christoph Anton Mitterer 
wrote:

> Hy Stuart, Julien and Ben,
>
> Hope you don't mind that I answer all three replies in one... don't
> wanna spam the list ;-)
>

Thanks!


>
>
>
> On Tue, 2023-02-21 at 07:31 +, Stuart Clark wrote:
> > Prometheus itself cannot do downsampling, but other related projects
> > such as Cortex & Thanos have such features.
>
> Uhm, I see. Unfortunately neither is packaged for Debian. Plus it seems
> to make the overall system even more complex.
>

I do not recommend using Debian packages for Prometheus. Debian release
cycles are too slow for the pace of Prometheus development. Every release
brings new improvements. In the last year we've made improvements to memory
use and query performance. Debian also ignores the Go source vendor
versions we provide which leads to bugs.

You'd be better off running Prometheus using podman, or deploying official
binaries with Ansible[0].

[0]: https://github.com/prometheus-community/ansible


>
> I want to Prometheus merely or monitoring a few hundred nodes (thus it
> seems a bit overkill to have something like Cortex, which sounds like a
> system for really large number of nodes) at the university, though as
> indicated before, we'd need both:
> - details data for a like the last week or perhaps two
> - far less detailed data for much longer terms (like several years)
>
> Right now my Prometheus server runs in a medium sized VM, but when I
> visualise via Grafana and select a time span of a month, it already
> takes considerable time (like 10-15s) to render the graph.
>
> Is this expected?
>

No, but It depends on your queries. Without seeing what you're graphing
there's no way to tell. Your queries could be complex or inefficient. Kinda
like writing slow SQL queries.

There are ways to speed up graphs for specific things, for example you can
use recording rules to pre-render parts of the queries.

For example, if you want to graph node CPU utilization you can have a
recording rule like this:

groups:
  - name: node_exporter
interval: 60s
rules:
  - record: instance:node_cpu_utilization:ratio_rate1m
expr: >
  avg without (cpu) (
sum without (mode) (

rate(node_cpu_seconds_total{mode!="idle",mode!="iowait",mode!="steal"}[1m])
)
  )

This will give you a single metric per node that will be faster to render
over longer periods of time. It also effectively down-samples by only
recording one point per minute.

Also "Medium sized VM" doesn't give us any indication of how much CPU or
memory you have. Prometheus uses page cache for database access. So maybe
your system is lacking enough memory to effectively cache the data you're
accessing.


>
>
>
>
> On Tue, 2023-02-21 at 11:45 +0100, Julien Pivotto wrote:
> > We would love to have this in the future but it would require careful
> > planning and design document.
>
> So native support is nothing on the near horizon?
>
> And I guess it's really not possible to "simply" ( ;-) ) have different
> retention times for different metrics?
>

No, we've talked about having variable retention times, but nobody has
implemented this. It's possible to script this via the DELETE endpoint[1].
It would be easy enough to write a cron job that deletes specific metrics
older than X, but I haven't seen this packaged into a simple tool. I would
love to see something like this created.

[1]:
https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series


>
>
>
>
> On Tue, 2023-02-21 at 15:52 +0100, Ben Kochie wrote:
> > This is mostly unnecessary in Prometheus because it uses compression
> > in the TSDB samples. What would take up a lot of space in an RRD file
> > takes up very little space in Prometheus.
>
> Well right now I scrape only the node-exporter data from 40 hosts at a
> 15s interval plus the metrics from prometheus itself.
> I'm doing this on test install since the 21st of February.
> Retention time is still at it's default.
>
> That gives me:
> # du --apparent-size -l -c -s --si /var/lib/prometheus/metrics2/*
> 68M /var/lib/prometheus/metrics2/01GSST2X0KDHZ0VM2WEX0FPS2H
> 481M/var/lib/prometheus/metrics2/01GSVQWH7BB6TDCEWXV4QFC9V2
> 501M/var/lib/prometheus/metrics2/01GSXNP1T77WCEM44CGD7E95QH
> 485M/var/lib/prometheus/metrics2/01GSZKFK53BQRXFAJ7RK9EDHQX
> 490M/var/lib/prometheus/metrics2/01GT1H90WKAHYGSFED5W2BW49Q
> 487M/var/lib/prometheus/metrics2/01GT3F2SJ6X22HFFPFKMV6DB3B
> 498M/var/lib/prometheus/metrics2/01GT5CW8HNJSGFJH2D3ADGC9HH
> 490M/var/lib/prometheus/metrics2/01GT7ANS5KDVHVQZJ7RTVNQQGH
> 501M/var/lib/prometheus/metrics2/01GT98FETDR3PN34ZP59Y0KNXT
> 172M/var/lib/prometheus/metrics2/01GT9X2BPN51JGB6QVK2X8R3BR
> 60M /var/lib/prometheus/metrics2/01GTAASP91FSFGBBH8BBN2SQDJ
> 60M /var/lib/prometheus/metrics2/01GTAHNDG070WXY8WGDVS22D2Y
> 171M/var/lib/prometheus/metrics2/01GTAHNHQ587CQVGWVDAN26V8S
> 102M/var/lib/prometheus/metrics2/chunks_head
> 

[prometheus-users] Re: Add parameters in rules files expression

[sayf.eddi...@gmail.com]

FWIW, we use saltstack based jinja2 templating with a configured systemd 
reload (ExecReload=/bin/kill -HUP $MAINPID) to update the thresholds, and 
it works fine


On Tuesday, February 28, 2023 at 9:31:12 AM UTC+1 Brian Candler wrote:

> After changing the rules file, you'll need to trigger a prometheus reload, 
> either by hitting a special API endpoint or sending it a SIGHUP. See 
> https://prometheus.io/docs/prometheus/latest/management_api/#reload
>
> (Note: *don't* restart the prometheus process! That's a much slower 
> operation as it has to dump out and reload its WAL and can take several 
> minutes on a busy server)
>
> On Tuesday, 28 February 2023 at 06:55:24 UTC inmar tzemach wrote:
>
>> Thanks!
>>
>> How can I change the threshold from outside for giving expression while 
>> Prometheus running? 
>>
>> On Monday, February 27, 2023 at 5:06:27 PM UTC+2 Brian Candler wrote:
>>
>>> No, but you can expand the rules files using a preprocessor of your 
>>> choice.
>>>
>>> If you want the threshold to be different for different time series, 
>>> then see 
>>> https://www.robustperception.io/using-time-series-as-alert-thresholds
>>>
>>> On Monday, 27 February 2023 at 13:58:47 UTC inmar tzemach wrote:
>>>
 Is it possible to add parameters in rules files expression?
 Something like:
 expr: (100 - (avg by (instance) 
 (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)) > {{ $treshhold}}

 Thanks

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/e3c57e07-094c-4d71-af0e-c9b0e0b34378n%40googlegroups.com.

[prometheus-users] Re: Add parameters in rules files expression

[Brian Candler]

After changing the rules file, you'll need to trigger a prometheus reload, 
either by hitting a special API endpoint or sending it a SIGHUP. 
See https://prometheus.io/docs/prometheus/latest/management_api/#reload

(Note: *don't* restart the prometheus process! That's a much slower 
operation as it has to dump out and reload its WAL and can take several 
minutes on a busy server)

On Tuesday, 28 February 2023 at 06:55:24 UTC inmar tzemach wrote:

> Thanks!
>
> How can I change the threshold from outside for giving expression while 
> Prometheus running? 
>
> On Monday, February 27, 2023 at 5:06:27 PM UTC+2 Brian Candler wrote:
>
>> No, but you can expand the rules files using a preprocessor of your 
>> choice.
>>
>> If you want the threshold to be different for different time series, then 
>> see https://www.robustperception.io/using-time-series-as-alert-thresholds
>>
>> On Monday, 27 February 2023 at 13:58:47 UTC inmar tzemach wrote:
>>
>>> Is it possible to add parameters in rules files expression?
>>> Something like:
>>> expr: (100 - (avg by (instance) 
>>> (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)) > {{ $treshhold}}
>>>
>>> Thanks
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/6a6f9b4f-e41f-4a6c-9814-89b978704c75n%40googlegroups.com.

Re: [prometheus-users] fading out sample resolution for samples from longer ago possible?

[Brian Candler]

On Tuesday, 28 February 2023 at 00:45:36 UTC Christoph Anton Mitterer wrote:

I want to Prometheus merely or monitoring a few hundred nodes (thus it 
seems a bit overkill to have something like Cortex, which sounds like a 
system for really large number of nodes) at the university


Thanos may be simpler. Although I've not used it myself, it looks like it 
can be deployed incrementally starting with the sidecars.

 

, though as 
indicated before, we'd need both: 
- details data for a like the last week or perhaps two 
- far less detailed data for much longer terms (like several years)


I can offer a couple more options:

(1) Use two servers with federation.
- server 1 does the scraping and keeps the detailed data for 2 weeks
- server 2 scrapes server 1 at lower interval, using the federation endpoint

(2) Use recording rules to generate lower-resolution copies of the primary 
timeseries - but then you'd still have to remote-write them to a second 
server to get the longer retention, since this can't be set at timeseries 
level.

Either case makes the querying more awkward.  If you don't want separate 
dashboards for near-term and long-term data, then it might work to stick 
promxy in front of them.

Apart from saving disk space (and disks are really, really cheap these 
days), I suspect the main benefit you're looking for is to get faster 
queries when running over long time periods.  Indeed, I believe Thanos 
creates downsampled timeseries for exactly this reason, whilst still 
continuing to retain all the full-resolution data as well.

Right now my Prometheus server runs in a medium sized VM, but when I 
visualise via Grafana and select a time span of a month, it already 
takes considerable time (like 10-15s) to render the graph.


Ah right, then that is indeed your concern.
 

Is this expected?


That depends.  What PromQL query does your graph use? How many timeseries 
does it touch? What's your scrape interval?  Is your VM backed by SSDs?

For example, I have a very low performance (Celeron N2820, SATA SSD, 8GB 
RAM) test box at home.  I scrape data at 15 second intervals. Prometheus is 
running in an lxd container, alongside many other lxd containers.  The 
query:

rate(ifHCInOctets{instance="gw2",ifName="pppoe-out2"}[2m])

run over a 30 day range takes less than a second - but that only touches 
one timeseries. (With 2-hour chunks, I would expect a 30 day period to read 
360 chunks, for a single timeseries).  But it's possible that when I tested 
it, it already had the relevant data cached in RAM.

If you are doing something like a Grafana dashboard, then you should 
determine exactly what queries it's doing.  Enabling the query log 
 can also help you identify 
the slowest running queries.

Another suggestion: running netdata  
within the VM will give you performance metrics at 1 second intervals, 
which can help identify what's happening during those 10-15 seconds: e.g. 
are you bottlenecked on CPU, or disk I/O, or something else.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/43cea4c0-31a8-4dd6-8d98-3fed327ccf39n%40googlegroups.com.