Re: [prometheus-users] Prom QL
Thanks Sir On Monday, 25 July 2022 at 12:26:43 UTC+5:30 Stuart Clark wrote: > On 25/07/2022 07:30, BHARATH KUMAR wrote: > > I am talking about grafana dashboard. I created a custom variable as > > follows: > > > > CPU: > > All : <=100, > > lt 10 && gt 0 : >0<=10, > > lt 30 && gt 10 : >10<=30 > > > > So this CPU filter will be added at the top of grafana dashboard. Now > > If I select ALL in CPU filter I am not able to find the unreachable > > servers list in the grafana dashboard as those servers are not present > > in prometheus Data Source(since these are in unreachable state). > > > > My query: > > > ((1 - avg(irate(node_cpu_seconds_total{mode="idle",instance=~"$ip"}[5m])) by > (instance)) * 100) > > > $CPU > > > > Here we are comparing the value in $cpu > > > > Is there any other way to compare the values that are not present in > > prometheus data source so that when I click ALL option I can able to > > see all servers data. > > If you are not scraping something (because the scrape is failing) then > no data exists about that server. > > For the drop down selector in Grafana you could use the "up" metric, > which will still have values for servers that weren't able to be > scraped, but selecting one of those would then result in an empty > dashboard as there wouldn't be any CPU values. > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3a2e28b6-800a-4586-9822-323d874d385fn%40googlegroups.com.
Re: [prometheus-users] Prom QL
On 25/07/2022 07:30, BHARATH KUMAR wrote: I am talking about grafana dashboard. I created a custom variable as follows: CPU: All : <=100, lt 10 && gt 0 : >0<=10, lt 30 && gt 10 : >10<=30 So this CPU filter will be added at the top of grafana dashboard. Now If I select ALL in CPU filter I am not able to find the unreachable servers list in the grafana dashboard as those servers are not present in prometheus Data Source(since these are in unreachable state). My query: ((1 - avg(irate(node_cpu_seconds_total{mode="idle",instance=~"$ip"}[5m])) by (instance)) * 100) $CPU Here we are comparing the value in $cpu Is there any other way to compare the values that are not present in prometheus data source so that when I click ALL option I can able to see all servers data. If you are not scraping something (because the scrape is failing) then no data exists about that server. For the drop down selector in Grafana you could use the "up" metric, which will still have values for servers that weren't able to be scraped, but selecting one of those would then result in an empty dashboard as there wouldn't be any CPU values. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b9017c3d-2464-084d-419f-394d0e3d5e5f%40Jahingo.com.
Re: [prometheus-users] Prom QL
I am talking about grafana dashboard. I created a custom variable as follows: CPU: All : <=100, lt 10 && gt 0 : >0<=10, lt 30 && gt 10 : >10<=30 So this CPU filter will be added at the top of grafana dashboard. Now If I select ALL in CPU filter I am not able to find the unreachable servers list in the grafana dashboard as those servers are not present in prometheus Data Source(since these are in unreachable state). My query: ((1 - avg(irate(node_cpu_seconds_total{mode="idle",instance=~"$ip"}[5m])) by (instance)) * 100) $CPU Here we are comparing the value in $cpu Is there any other way to compare the values that are not present in prometheus data source so that when I click ALL option I can able to see all servers data. Thanks & regards, Bharath Kumar. On Wednesday, 20 July 2022 at 18:56:58 UTC+5:30 Brian Candler wrote: > No idea. You haven't said what dashboard software you're using, nor what > queries you're using to build that dashboard. > > > I want to add one filter so that I can be able to know which servers are > least used or most used. > > Not sure what you mean by a "filter" in this context. A PromQL query > using min() or max() will work over all the values which are present in the > instant vector, which as I said before, is of variable size. It doesn't > have to have a fixed number of inputs. > > e.g. given this data > > [ > node_blah{instance="foo"} 123 > node_blah{instance="bar"} 456 > node_blah{instance="baz"} 789 > ] > > then > > min(node_blah) => 123 > > On Wednesday, 20 July 2022 at 10:28:58 UTC+1 chembakay...@gmail.com wrote: > >> Thanks, Clark and Brian for your reply. >> >> I am using two data sources in my case. i.e Prometheus and Postgres. >> >> In my dashboard, there is a table that contains both Prometheus and >> Postgres data. In this table, there is a column name %cpu used which will >> be obtained from Prometheus. >> >> As Brain said, if the server goes down, we will not get the node level >> metrics and for that particular server, we will have Postgres data but, not >> Prometheus as the server was down. >> >> for example, my dashboard table is as follows: >> >> IPCPU %cpu memory memory_used column1 >> column2column3 >> 1.1.1.1 4 0.4% 40gb 60% >> a b c >> 1.1.1.2 8 10%80gb30% >> d e f >> 1.1.1.3 >>h i j >> >> >> the third server goes down, so we are not able to see the CPU and memory >> values, my question was I want to add one filter so that I can be able to >> know which servers are least used or most used. >> >> The CPU used for the third server will be no data as that server was >> down. can we do any comparison for these servers(servers who went down) so >> that I can filter these servers whose value will be null/no_data. >> >> Thanks & regards, >> Bharath Kumar. >> >> On Wednesday, 20 July 2022 at 14:22:17 UTC+5:30 Brian Candler wrote: >> >>> And just to clarify slightly, there aren't really "null values" in >>> prometheus. A query like "node_blah" returns a *vector* of results, that >>> is, a variable number of values. e.g. >>> >>> [ >>> node_blah{instance="foo"} 123 >>> node_blah{instance="bar"} 456 >>> node_blah{instance="baz"} 789 >>> ] >>> >>> If node "baz" goes down, then a query at a later point in time may return >>> >>> [ >>> node_blah{instance="foo"} 124 >>> node_blah{instance="bar"} 457 >>> ] >>> >>> If you want to test for this specific condition, i.e. there is no >>> "node_blah" metric present for a specific instance "baz", then you can form >>> a rather awkward join query using absent() in conjunction with the "up" >>> metric as Stuart described. >>> >>> But usually, you just want to query the "up" metric itself. >>> >>> On Wednesday, 20 July 2022 at 09:38:58 UTC+1 Stuart Clark wrote: >>> On 20/07/2022 08:49, BHARATH KUMAR wrote: Hello all, I installed node exporters on many servers (around 300). Few of the servers are unreachable. So because of that, we are unable to get the CPU, and memory values of those servers. Now I want to add a filter in the Grafana dashboard to check the least CPU used, most CPU used servers. But due to unreachability, we are not getting values for a few servers. My question is "*how to compare the output of the Prometheus query is NULL"* Generally, I am comparing the output of the prom query like I) if the CPU usage is less than 10% then I am comparing like query >=0<=10% ii) if the CPU usage is greater than 10% and less than 30% then I am comparing like query >10<=30 *similarly how to check the null values using the Promethe
Re: [prometheus-users] Prom QL
No idea. You haven't said what dashboard software you're using, nor what queries you're using to build that dashboard. > I want to add one filter so that I can be able to know which servers are least used or most used. Not sure what you mean by a "filter" in this context. A PromQL query using min() or max() will work over all the values which are present in the instant vector, which as I said before, is of variable size. It doesn't have to have a fixed number of inputs. e.g. given this data [ node_blah{instance="foo"} 123 node_blah{instance="bar"} 456 node_blah{instance="baz"} 789 ] then min(node_blah) => 123 On Wednesday, 20 July 2022 at 10:28:58 UTC+1 chembakay...@gmail.com wrote: > Thanks, Clark and Brian for your reply. > > I am using two data sources in my case. i.e Prometheus and Postgres. > > In my dashboard, there is a table that contains both Prometheus and > Postgres data. In this table, there is a column name %cpu used which will > be obtained from Prometheus. > > As Brain said, if the server goes down, we will not get the node level > metrics and for that particular server, we will have Postgres data but, not > Prometheus as the server was down. > > for example, my dashboard table is as follows: > > IPCPU %cpu memory memory_used column1 > column2column3 > 1.1.1.1 4 0.4% 40gb 60% > a b c > 1.1.1.2 8 10%80gb30% > d e f > 1.1.1.3 > h i j > > > the third server goes down, so we are not able to see the CPU and memory > values, my question was I want to add one filter so that I can be able to > know which servers are least used or most used. > > The CPU used for the third server will be no data as that server was down. > can we do any comparison for these servers(servers who went down) so that I > can filter these servers whose value will be null/no_data. > > Thanks & regards, > Bharath Kumar. > > On Wednesday, 20 July 2022 at 14:22:17 UTC+5:30 Brian Candler wrote: > >> And just to clarify slightly, there aren't really "null values" in >> prometheus. A query like "node_blah" returns a *vector* of results, that >> is, a variable number of values. e.g. >> >> [ >> node_blah{instance="foo"} 123 >> node_blah{instance="bar"} 456 >> node_blah{instance="baz"} 789 >> ] >> >> If node "baz" goes down, then a query at a later point in time may return >> >> [ >> node_blah{instance="foo"} 124 >> node_blah{instance="bar"} 457 >> ] >> >> If you want to test for this specific condition, i.e. there is no >> "node_blah" metric present for a specific instance "baz", then you can form >> a rather awkward join query using absent() in conjunction with the "up" >> metric as Stuart described. >> >> But usually, you just want to query the "up" metric itself. >> >> On Wednesday, 20 July 2022 at 09:38:58 UTC+1 Stuart Clark wrote: >> >>> On 20/07/2022 08:49, BHARATH KUMAR wrote: >>> >>> Hello all, >>> >>> I installed node exporters on many servers (around 300). Few of the >>> servers are unreachable. So because of that, we are unable to get the CPU, >>> and memory values of those servers. >>> >>> Now I want to add a filter in the Grafana dashboard to check the least >>> CPU used, most CPU used servers. But due to unreachability, we are not >>> getting values for a few servers. >>> >>> My question is >>> "*how to compare the output of the Prometheus query is NULL"* >>> >>> Generally, I am comparing the output of the prom query like >>> I) if the CPU usage is less than 10% then I am comparing like >>> query >=0<=10% >>> ii) if the CPU usage is greater than 10% and less than 30% then I am >>> comparing like >>> query >10<=30 >>> *similarly how to check the null values using the Prometheus query.* >>> >>> For servers which can't be scraped there will be no metrics, so any >>> queries won't have any data to query. >>> >>> However Prometheus itself creates certain metrics for all scrape >>> targets, including one called "up" which is either 0 or 1 - where 0 means >>> the scrape failed. You can therefore create dashboards and alerts that list >>> the servers which aren't accessible (up == 0). >>> >>> -- >>> Stuart Clark >>> >>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2c15e3e8-b878-45a4-9386-ef2f0c0b1ccen%40googlegroups.com.
Re: [prometheus-users] Prom QL
Thanks, Clark and Brian for your reply. I am using two data sources in my case. i.e Prometheus and Postgres. In my dashboard, there is a table that contains both Prometheus and Postgres data. In this table, there is a column name %cpu used which will be obtained from Prometheus. As Brain said, if the server goes down, we will not get the node level metrics and for that particular server, we will have Postgres data but, not Prometheus as the server was down. for example, my dashboard table is as follows: IPCPU %cpu memory memory_used column1 column2column3 1.1.1.1 4 0.4% 40gb 60% a b c 1.1.1.2 8 10%80gb30% d e f 1.1.1.3 h i j the third server goes down, so we are not able to see the CPU and memory values, my question was I want to add one filter so that I can be able to know which servers are least used or most used. The CPU used for the third server will be no data as that server was down. can we do any comparison for these servers(servers who went down) so that I can filter these servers whose value will be null/no_data. Thanks & regards, Bharath Kumar. On Wednesday, 20 July 2022 at 14:22:17 UTC+5:30 Brian Candler wrote: > And just to clarify slightly, there aren't really "null values" in > prometheus. A query like "node_blah" returns a *vector* of results, that > is, a variable number of values. e.g. > > [ > node_blah{instance="foo"} 123 > node_blah{instance="bar"} 456 > node_blah{instance="baz"} 789 > ] > > If node "baz" goes down, then a query at a later point in time may return > > [ > node_blah{instance="foo"} 124 > node_blah{instance="bar"} 457 > ] > > If you want to test for this specific condition, i.e. there is no > "node_blah" metric present for a specific instance "baz", then you can form > a rather awkward join query using absent() in conjunction with the "up" > metric as Stuart described. > > But usually, you just want to query the "up" metric itself. > > On Wednesday, 20 July 2022 at 09:38:58 UTC+1 Stuart Clark wrote: > >> On 20/07/2022 08:49, BHARATH KUMAR wrote: >> >> Hello all, >> >> I installed node exporters on many servers (around 300). Few of the >> servers are unreachable. So because of that, we are unable to get the CPU, >> and memory values of those servers. >> >> Now I want to add a filter in the Grafana dashboard to check the least >> CPU used, most CPU used servers. But due to unreachability, we are not >> getting values for a few servers. >> >> My question is >> "*how to compare the output of the Prometheus query is NULL"* >> >> Generally, I am comparing the output of the prom query like >> I) if the CPU usage is less than 10% then I am comparing like >> query >=0<=10% >> ii) if the CPU usage is greater than 10% and less than 30% then I am >> comparing like >> query >10<=30 >> *similarly how to check the null values using the Prometheus query.* >> >> For servers which can't be scraped there will be no metrics, so any >> queries won't have any data to query. >> >> However Prometheus itself creates certain metrics for all scrape targets, >> including one called "up" which is either 0 or 1 - where 0 means the scrape >> failed. You can therefore create dashboards and alerts that list the >> servers which aren't accessible (up == 0). >> >> -- >> Stuart Clark >> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/acd5139b-258d-48ab-a767-990132c052e7n%40googlegroups.com.
Re: [prometheus-users] Prom QL
And just to clarify slightly, there aren't really "null values" in prometheus. A query like "node_blah" returns a *vector* of results, that is, a variable number of values. e.g. [ node_blah{instance="foo"} 123 node_blah{instance="bar"} 456 node_blah{instance="baz"} 789 ] If node "baz" goes down, then a query at a later point in time may return [ node_blah{instance="foo"} 124 node_blah{instance="bar"} 457 ] If you want to test for this specific condition, i.e. there is no "node_blah" metric present for a specific instance "baz", then you can form a rather awkward join query using absent() in conjunction with the "up" metric as Stuart described. But usually, you just want to query the "up" metric itself. On Wednesday, 20 July 2022 at 09:38:58 UTC+1 Stuart Clark wrote: > On 20/07/2022 08:49, BHARATH KUMAR wrote: > > Hello all, > > I installed node exporters on many servers (around 300). Few of the > servers are unreachable. So because of that, we are unable to get the CPU, > and memory values of those servers. > > Now I want to add a filter in the Grafana dashboard to check the least CPU > used, most CPU used servers. But due to unreachability, we are not getting > values for a few servers. > > My question is > "*how to compare the output of the Prometheus query is NULL"* > > Generally, I am comparing the output of the prom query like > I) if the CPU usage is less than 10% then I am comparing like > query >=0<=10% > ii) if the CPU usage is greater than 10% and less than 30% then I am > comparing like > query >10<=30 > *similarly how to check the null values using the Prometheus query.* > > For servers which can't be scraped there will be no metrics, so any > queries won't have any data to query. > > However Prometheus itself creates certain metrics for all scrape targets, > including one called "up" which is either 0 or 1 - where 0 means the scrape > failed. You can therefore create dashboards and alerts that list the > servers which aren't accessible (up == 0). > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/645e6766-818b-474d-a5c2-ad6b61b2789an%40googlegroups.com.
Re: [prometheus-users] Prom QL
On 20/07/2022 08:49, BHARATH KUMAR wrote: Hello all, I installed node exporters on many servers (around 300). Few of the servers are unreachable. So because of that, we are unable to get the CPU, and memory values of those servers. Now I want to add a filter in the Grafana dashboard to check the least CPU used, most CPU used servers. But due to unreachability, we are not getting values for a few servers. My question is "*how to compare the output of the Prometheus query is NULL"* Generally, I am comparing the output of the prom query like I) if the CPU usage is less than 10% then I am comparing like query >=0<=10% ii) if the CPU usage is greater than 10% and less than 30% then I am comparing like query >10<=30 *similarly how to check the null values using the Prometheus query.* For servers which can't be scraped there will be no metrics, so any queries won't have any data to query. However Prometheus itself creates certain metrics for all scrape targets, including one called "up" which is either 0 or 1 - where 0 means the scrape failed. You can therefore create dashboards and alerts that list the servers which aren't accessible (up == 0). -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/53716e29-ceb2-7304-b003-17eba5262684%40Jahingo.com.