Re: [prometheus-users] Create custom metrics and add it to prometheus

[Brian Candler]

Also: you should clarify where exactly you're running these scripts, i.e. 
which host has the /var/lib/node_exporter/*.prom files

* If it's on the prometheus server itself, then you'll need to run 
node_exporter on the prometheus server, *and* you'll need to get prometheus 
to scrape itself - i.e. a scrape job that collects from 
localhost:9100/metrics

  - job_name: 'node'
static_configs:
  - targets: ['localhost:9100']

* If it's on some remote host, then you'll need to get prometheus to scrape 
*that* host on x.x.x.x:9100/metrics

If you're already scraping node_exporter on all your hosts - for example, 
you can see metrics like "node_filesystem_avail_bytes", on the instance 
where your custom metrics are being created - then you should be fine.

On Wednesday, 1 March 2023 at 08:58:25 UTC Brian Candler wrote:

> *> The main issue is that I am not able to see the metrics in Prometheus 
> UI. I am unable to understand the root cause.*
>
> Almost certainly because they are not being collected.  So you need to 
> break this down further to find out why.
>
> (1) Show the output of this command:
>
> *ls -l /var/lib/node_exporter*
>
> * If the filenames don't end with ".prom" then that is your root cause.  
> node_exporter only picks up files whose names match the pattern *.prom
>
> * If the files aren't readable by the user that node_exporter is running 
> as, then that's the root cause.
>
> * Are you running under RedHat/CentOS with SELinux?  If so, try putting it 
> into permissive mode.  Maybe there's some SELinux policy which prevents 
> access to files in /var/lib/node_exporter.
>
> (2) Then show the contents of one of those files:
>
> *head /var/lib/node_exporter/foobar.prom*
>
> If the files are not in valid openmetrics format, then they won't be 
> picked up either.  You can test them with "promtool check metrics":
>
> /path/to/promtool check metrics 
> If that shows an error, then modify your script to generate files in the 
> correct format. There are examples you can copy from here: 
> https://github.com/prometheus-community/node-exporter-textfile-collector-scripts
>
> (3) Show the command line that you're running node_exporter with:
>
> *ps auxwww | grep node_exporter*
>
> If the flag "--collector.textfile.directory=/var/lib/node_exporter" is not 
> present, then that's your root cause.
>
> (4) Show the output of this command:
>
> *curl localhost:9100/metrics | grep foobar*
>
> where "foobar" is one of the custom metrics you are generating.  Does it 
> appear here?  If not, then you can rule out any issues with prometheus 
> itself, it's definitely node_exporter not picking up your custom metrics.  
> If it does, then you'll need to dig further onto prometheus side (e.g. your 
> scrape job config)
>
>
> *> In /var/lib/node_exporter --> In this folder, the scripts(custom 
> metrics script) will be available.*
>
> You don't want the scripts in that directory - only the *output* from 
> those scripts.  However, as long as the scripts themselves don't end with 
> ".prom" then they will be ignored, so that won't be a problem.
>
> On Wednesday, 1 March 2023 at 05:24:40 UTC BHARATH KUMAR wrote:
>
>> The main issue is that I am not able to see the metrics in Prometheus UI. 
>> I am unable to understand the root cause.
>>
>> Am I missing anything?
>>
>> On Tuesday, 28 February 2023 at 23:44:59 UTC+5:30 Brian Candler wrote:
>>
>>> Aside: you might also want to look at kube-state-metrics 
>>> , which can give you 
>>> a range of pod metrics 
>>> 
>>>  
>>> including the pod start time. Then an expression like 
>>> time()-kube_pod_start_time should give you the pod age.
>>>
>>> On Tuesday, 28 February 2023 at 17:55:20 UTC Brian Candler wrote:
>>>
 The problem is that the filename must end with ".prom", i.e. you need 
 to create "apt1.prom" not "apt1-prom"

 Also, you need to enable the functionality with a flag to node_exporter:

 node_exporter --collector.textfile.directory=/var/lib/node_exporter

 See the documentation here: 
 https://github.com/prometheus/node_exporter#textfile-collector

 On Tuesday, 28 February 2023 at 17:16:06 UTC Stuart Clark wrote:

> On 28/02/2023 17:12, BHARATH KUMAR wrote: 
> > Hello All, 
> > 
> > I want to calculate the pod age running on every server. I wrote a 
> > shell script. I added this script under the folder 
> /var/lib/node_exporter. 
> > 
> > I created a cronjob for this script to run every minute 
> > * * * * * root bash /var/lib/node_exporter/custom_metrics.sh > 
> > /var/lib/node_exporter/apt1-prom 
> > 
> > I store the above cronjob in the /etc/cron.d folder with filename: 
> > prom-apt1. 
> > 
> > But I am not able to see the metrics I created in Prometheus UI. 
> > 
> > 
> > 
> > But similarly, I 

Re: [prometheus-users] Create custom metrics and add it to prometheus

[Brian Candler]

*> The main issue is that I am not able to see the metrics in Prometheus 
UI. I am unable to understand the root cause.*

Almost certainly because they are not being collected.  So you need to 
break this down further to find out why.

(1) Show the output of this command:

*ls -l /var/lib/node_exporter*

* If the filenames don't end with ".prom" then that is your root cause.  
node_exporter only picks up files whose names match the pattern *.prom

* If the files aren't readable by the user that node_exporter is running 
as, then that's the root cause.

* Are you running under RedHat/CentOS with SELinux?  If so, try putting it 
into permissive mode.  Maybe there's some SELinux policy which prevents 
access to files in /var/lib/node_exporter.

(2) Then show the contents of one of those files:

*head /var/lib/node_exporter/foobar.prom*

If the files are not in valid openmetrics format, then they won't be picked 
up either.  You can test them with "promtool check metrics":

/path/to/promtool check metrics https://github.com/prometheus-community/node-exporter-textfile-collector-scripts

(3) Show the command line that you're running node_exporter with:

*ps auxwww | grep node_exporter*

If the flag "--collector.textfile.directory=/var/lib/node_exporter" is not 
present, then that's your root cause.

(4) Show the output of this command:

*curl localhost:9100/metrics | grep foobar*

where "foobar" is one of the custom metrics you are generating.  Does it 
appear here?  If not, then you can rule out any issues with prometheus 
itself, it's definitely node_exporter not picking up your custom metrics.  
If it does, then you'll need to dig further onto prometheus side (e.g. your 
scrape job config)


*> In /var/lib/node_exporter --> In this folder, the scripts(custom metrics 
script) will be available.*

You don't want the scripts in that directory - only the *output* from those 
scripts.  However, as long as the scripts themselves don't end with ".prom" 
then they will be ignored, so that won't be a problem.

On Wednesday, 1 March 2023 at 05:24:40 UTC BHARATH KUMAR wrote:

> The main issue is that I am not able to see the metrics in Prometheus UI. 
> I am unable to understand the root cause.
>
> Am I missing anything?
>
> On Tuesday, 28 February 2023 at 23:44:59 UTC+5:30 Brian Candler wrote:
>
>> Aside: you might also want to look at kube-state-metrics 
>> , which can give you a 
>> range of pod metrics 
>> 
>>  
>> including the pod start time. Then an expression like 
>> time()-kube_pod_start_time should give you the pod age.
>>
>> On Tuesday, 28 February 2023 at 17:55:20 UTC Brian Candler wrote:
>>
>>> The problem is that the filename must end with ".prom", i.e. you need to 
>>> create "apt1.prom" not "apt1-prom"
>>>
>>> Also, you need to enable the functionality with a flag to node_exporter:
>>>
>>> node_exporter --collector.textfile.directory=/var/lib/node_exporter
>>>
>>> See the documentation here: 
>>> https://github.com/prometheus/node_exporter#textfile-collector
>>>
>>> On Tuesday, 28 February 2023 at 17:16:06 UTC Stuart Clark wrote:
>>>
 On 28/02/2023 17:12, BHARATH KUMAR wrote: 
 > Hello All, 
 > 
 > I want to calculate the pod age running on every server. I wrote a 
 > shell script. I added this script under the folder 
 /var/lib/node_exporter. 
 > 
 > I created a cronjob for this script to run every minute 
 > * * * * * root bash /var/lib/node_exporter/custom_metrics.sh > 
 > /var/lib/node_exporter/apt1-prom 
 > 
 > I store the above cronjob in the /etc/cron.d folder with filename: 
 > prom-apt1. 
 > 
 > But I am not able to see the metrics I created in Prometheus UI. 
 > 
 > 
 > 
 > But similarly, I created another shell script file to fetch some 
 > metrics. I followed the same procedure as above. 
 > 
 > * * * * * root bash /var/lib/node_exporter/custom1.sh > 
 > /var/lib/node_exporter/apt-prom 
 > 
 > I store the above cronjob in the /etc/cron.d folder with filename: 
 > prom-apt. 
 > 
 > The metrics which I mentioned in custom1.sh, I am able to see those 
 > metrics in Prometheus UI. 
 > 
 > 
 > Could anyone help me? 
 > 
 What is the contents of those files in /var/lib/node_exporter? 

 -- 
 Stuart Clark 



-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/2c04d777-3ced-41c9-bbda-f4e10ba4fd3bn%40googlegroups.com.

Re: [prometheus-users] Create custom metrics and add it to prometheus

[BHARATH KUMAR]

The main issue is that I am not able to see the metrics in Prometheus UI. I 
am unable to understand the root cause.

Am I missing anything?

On Tuesday, 28 February 2023 at 23:44:59 UTC+5:30 Brian Candler wrote:

> Aside: you might also want to look at kube-state-metrics 
> , which can give you a 
> range of pod metrics 
> 
>  
> including the pod start time. Then an expression like 
> time()-kube_pod_start_time should give you the pod age.
>
> On Tuesday, 28 February 2023 at 17:55:20 UTC Brian Candler wrote:
>
>> The problem is that the filename must end with ".prom", i.e. you need to 
>> create "apt1.prom" not "apt1-prom"
>>
>> Also, you need to enable the functionality with a flag to node_exporter:
>>
>> node_exporter --collector.textfile.directory=/var/lib/node_exporter
>>
>> See the documentation here: 
>> https://github.com/prometheus/node_exporter#textfile-collector
>>
>> On Tuesday, 28 February 2023 at 17:16:06 UTC Stuart Clark wrote:
>>
>>> On 28/02/2023 17:12, BHARATH KUMAR wrote: 
>>> > Hello All, 
>>> > 
>>> > I want to calculate the pod age running on every server. I wrote a 
>>> > shell script. I added this script under the folder 
>>> /var/lib/node_exporter. 
>>> > 
>>> > I created a cronjob for this script to run every minute 
>>> > * * * * * root bash /var/lib/node_exporter/custom_metrics.sh > 
>>> > /var/lib/node_exporter/apt1-prom 
>>> > 
>>> > I store the above cronjob in the /etc/cron.d folder with filename: 
>>> > prom-apt1. 
>>> > 
>>> > But I am not able to see the metrics I created in Prometheus UI. 
>>> > 
>>> > 
>>> > 
>>> > But similarly, I created another shell script file to fetch some 
>>> > metrics. I followed the same procedure as above. 
>>> > 
>>> > * * * * * root bash /var/lib/node_exporter/custom1.sh > 
>>> > /var/lib/node_exporter/apt-prom 
>>> > 
>>> > I store the above cronjob in the /etc/cron.d folder with filename: 
>>> > prom-apt. 
>>> > 
>>> > The metrics which I mentioned in custom1.sh, I am able to see those 
>>> > metrics in Prometheus UI. 
>>> > 
>>> > 
>>> > Could anyone help me? 
>>> > 
>>> What is the contents of those files in /var/lib/node_exporter? 
>>>
>>> -- 
>>> Stuart Clark 
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/248cb5ff-0c32-43e6-88cc-e16600e6ed18n%40googlegroups.com.

Re: [prometheus-users] Create custom metrics and add it to prometheus

[BHARATH KUMAR]

In /var/lib/node_exporter --> In this folder, the scripts(custom metrics 
script) will be available.


On Tuesday, 28 February 2023 at 22:46:06 UTC+5:30 Stuart Clark wrote:

> On 28/02/2023 17:12, BHARATH KUMAR wrote:
> > Hello All,
> >
> > I want to calculate the pod age running on every server. I wrote a 
> > shell script. I added this script under the folder 
> /var/lib/node_exporter.
> >
> > I created a cronjob for this script to run every minute
> > * * * * * root bash /var/lib/node_exporter/custom_metrics.sh > 
> > /var/lib/node_exporter/apt1-prom
> >
> > I store the above cronjob in the /etc/cron.d folder with filename: 
> > prom-apt1.
> >
> > But I am not able to see the metrics I created in Prometheus UI.
> >
> >
> >
> > But similarly, I created another shell script file to fetch some 
> > metrics. I followed the same procedure as above.
> >
> > * * * * * root bash /var/lib/node_exporter/custom1.sh > 
> > /var/lib/node_exporter/apt-prom
> >
> > I store the above cronjob in the /etc/cron.d folder with filename: 
> > prom-apt.
> >
> > The metrics which I mentioned in custom1.sh, I am able to see those 
> > metrics in Prometheus UI.
> >
> >
> > Could anyone help me?
> >
> What is the contents of those files in /var/lib/node_exporter?
>
> -- 
> Stuart Clark
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/85673771-c5fb-45c1-bb27-f71cca079407n%40googlegroups.com.

Re: [prometheus-users] Create custom metrics and add it to prometheus

[BHARATH KUMAR]

Sorry for the mistake in the above message. I created the filename with 
".prom" only.

I added the --collector.textfile.directory=/var/lib/node_exporter as one of 
the scripts was working fine. but another file was issue.
On Tuesday, 28 February 2023 at 23:25:20 UTC+5:30 Brian Candler wrote:

> The problem is that the filename must end with ".prom", i.e. you need to 
> create "apt1.prom" not "apt1-prom"
>
> Also, you need to enable the functionality with a flag to node_exporter:
>
> node_exporter --collector.textfile.directory=/var/lib/node_exporter
>
> See the documentation here: 
> https://github.com/prometheus/node_exporter#textfile-collector
>
> On Tuesday, 28 February 2023 at 17:16:06 UTC Stuart Clark wrote:
>
>> On 28/02/2023 17:12, BHARATH KUMAR wrote: 
>> > Hello All, 
>> > 
>> > I want to calculate the pod age running on every server. I wrote a 
>> > shell script. I added this script under the folder 
>> /var/lib/node_exporter. 
>> > 
>> > I created a cronjob for this script to run every minute 
>> > * * * * * root bash /var/lib/node_exporter/custom_metrics.sh > 
>> > /var/lib/node_exporter/apt1-prom 
>> > 
>> > I store the above cronjob in the /etc/cron.d folder with filename: 
>> > prom-apt1. 
>> > 
>> > But I am not able to see the metrics I created in Prometheus UI. 
>> > 
>> > 
>> > 
>> > But similarly, I created another shell script file to fetch some 
>> > metrics. I followed the same procedure as above. 
>> > 
>> > * * * * * root bash /var/lib/node_exporter/custom1.sh > 
>> > /var/lib/node_exporter/apt-prom 
>> > 
>> > I store the above cronjob in the /etc/cron.d folder with filename: 
>> > prom-apt. 
>> > 
>> > The metrics which I mentioned in custom1.sh, I am able to see those 
>> > metrics in Prometheus UI. 
>> > 
>> > 
>> > Could anyone help me? 
>> > 
>> What is the contents of those files in /var/lib/node_exporter? 
>>
>> -- 
>> Stuart Clark 
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/96ebb6eb-31b3-44e5-b8cc-adf6b94cf1a7n%40googlegroups.com.

Re: [prometheus-users] Create custom metrics and add it to prometheus

[Brian Candler]

Aside: you might also want to look at kube-state-metrics 
, which can give you a 
range of pod metrics 

 
including the pod start time. Then an expression like 
time()-kube_pod_start_time should give you the pod age.

On Tuesday, 28 February 2023 at 17:55:20 UTC Brian Candler wrote:

> The problem is that the filename must end with ".prom", i.e. you need to 
> create "apt1.prom" not "apt1-prom"
>
> Also, you need to enable the functionality with a flag to node_exporter:
>
> node_exporter --collector.textfile.directory=/var/lib/node_exporter
>
> See the documentation here: 
> https://github.com/prometheus/node_exporter#textfile-collector
>
> On Tuesday, 28 February 2023 at 17:16:06 UTC Stuart Clark wrote:
>
>> On 28/02/2023 17:12, BHARATH KUMAR wrote: 
>> > Hello All, 
>> > 
>> > I want to calculate the pod age running on every server. I wrote a 
>> > shell script. I added this script under the folder 
>> /var/lib/node_exporter. 
>> > 
>> > I created a cronjob for this script to run every minute 
>> > * * * * * root bash /var/lib/node_exporter/custom_metrics.sh > 
>> > /var/lib/node_exporter/apt1-prom 
>> > 
>> > I store the above cronjob in the /etc/cron.d folder with filename: 
>> > prom-apt1. 
>> > 
>> > But I am not able to see the metrics I created in Prometheus UI. 
>> > 
>> > 
>> > 
>> > But similarly, I created another shell script file to fetch some 
>> > metrics. I followed the same procedure as above. 
>> > 
>> > * * * * * root bash /var/lib/node_exporter/custom1.sh > 
>> > /var/lib/node_exporter/apt-prom 
>> > 
>> > I store the above cronjob in the /etc/cron.d folder with filename: 
>> > prom-apt. 
>> > 
>> > The metrics which I mentioned in custom1.sh, I am able to see those 
>> > metrics in Prometheus UI. 
>> > 
>> > 
>> > Could anyone help me? 
>> > 
>> What is the contents of those files in /var/lib/node_exporter? 
>>
>> -- 
>> Stuart Clark 
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/84408ccb-6753-41ec-94db-95cc404baa6en%40googlegroups.com.

Re: [prometheus-users] Create custom metrics and add it to prometheus

[Brian Candler]

The problem is that the filename must end with ".prom", i.e. you need to 
create "apt1.prom" not "apt1-prom"

Also, you need to enable the functionality with a flag to node_exporter:

node_exporter --collector.textfile.directory=/var/lib/node_exporter

See the documentation 
here: https://github.com/prometheus/node_exporter#textfile-collector

On Tuesday, 28 February 2023 at 17:16:06 UTC Stuart Clark wrote:

> On 28/02/2023 17:12, BHARATH KUMAR wrote:
> > Hello All,
> >
> > I want to calculate the pod age running on every server. I wrote a 
> > shell script. I added this script under the folder 
> /var/lib/node_exporter.
> >
> > I created a cronjob for this script to run every minute
> > * * * * * root bash /var/lib/node_exporter/custom_metrics.sh > 
> > /var/lib/node_exporter/apt1-prom
> >
> > I store the above cronjob in the /etc/cron.d folder with filename: 
> > prom-apt1.
> >
> > But I am not able to see the metrics I created in Prometheus UI.
> >
> >
> >
> > But similarly, I created another shell script file to fetch some 
> > metrics. I followed the same procedure as above.
> >
> > * * * * * root bash /var/lib/node_exporter/custom1.sh > 
> > /var/lib/node_exporter/apt-prom
> >
> > I store the above cronjob in the /etc/cron.d folder with filename: 
> > prom-apt.
> >
> > The metrics which I mentioned in custom1.sh, I am able to see those 
> > metrics in Prometheus UI.
> >
> >
> > Could anyone help me?
> >
> What is the contents of those files in /var/lib/node_exporter?
>
> -- 
> Stuart Clark
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/fd4d0b99-2828-4ac0-a34f-585e5cfef00cn%40googlegroups.com.

Re: [prometheus-users] Create custom metrics and add it to prometheus

[Stuart Clark]

On 28/02/2023 17:12, BHARATH KUMAR wrote:

Hello All,

I want to calculate the pod age running on every server. I wrote a 
shell script. I added this script under the folder /var/lib/node_exporter.


I created a cronjob for this script to run every minute
* * * * * root bash /var/lib/node_exporter/custom_metrics.sh > 
/var/lib/node_exporter/apt1-prom


I store the above cronjob in the /etc/cron.d folder with filename: 
prom-apt1.


But I am not able to see the metrics I created in Prometheus UI.



But similarly, I created another shell script file to fetch some 
metrics. I followed the same procedure as above.


* * * * * root bash /var/lib/node_exporter/custom1.sh > 
/var/lib/node_exporter/apt-prom


I store the above cronjob in the /etc/cron.d folder with filename: 
prom-apt.


The metrics which I mentioned in custom1.sh, I am able to see those 
metrics in Prometheus UI.



Could anyone help me?


What is the contents of those files in /var/lib/node_exporter?

--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/34cc064c-d700-931b-d912-1f9dd256b14c%40Jahingo.com.

[prometheus-users] Create custom metrics and add it to prometheus

[BHARATH KUMAR]

Hello All,

I want to calculate the pod age running on every server. I wrote a shell 
script. I added this script under the folder /var/lib/node_exporter.

I created a cronjob for this script to run every minute 
* * * * * root bash /var/lib/node_exporter/custom_metrics.sh > 
/var/lib/node_exporter/apt1-prom

I store the above cronjob in the /etc/cron.d folder with filename: 
prom-apt1.

But I am not able to see the metrics I created in Prometheus UI.



But similarly, I created another shell script file to fetch some metrics. I 
followed the same procedure as above.

* * * * * root bash /var/lib/node_exporter/custom1.sh > 
/var/lib/node_exporter/apt-prom

I store the above cronjob in the /etc/cron.d folder with filename: prom-apt.

The metrics which I mentioned in custom1.sh, I am able to see those metrics 
in Prometheus UI.


Could anyone help me?

Thanks & regards,
Bharath Kumar.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/20ea101b-f81e-47d8-be22-543030ca5771n%40googlegroups.com.