[rrd-users] NaN values in percentile calculations

[Jacques du Toit]

Hi guys

I'm having this issue with some of our data for a customer’s billing. Wondering 
if there is an elegant solution? What I really want is the ability to exclude 
NaN values from percentile calculations. Consider the following data series:

foo = 1,NaN,2,NaN,3,NaN,4,NaN,5,NaN,6,NaN,7,NaN,8,NaN,9,NaN,10,NaN

VDEF:90perc=foo,90,PERCENT

Seems like rrdtool includes the NaN values in the calculation. So I get 
90perc=8. While technically correct according to the series, its not really 
useful in determining real 90th percentile values in an every-day use-case. 
Particularly with billing, just because you have no data it’s not reasonable to 
assume it’s 0. Most likely the traffic profile by “connecting-the-dots” would 
have actually looked like this:

1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9,10,10

In this case I get 90perc=9 which means the billing would be skewed because of 
the loss of data. Now, it’s debatable what is mathematically really the right 
thing to do here. But the bottom line is that that we don’t know if the NaN 
values would have been above or below the 90th percentile value, it’s better to 
exclude them rather than assume they are below, IMHO.

The customer would also not be too happy as NaN values being included always 
pushes the percentile value down by definition and this mean they might end up 
with slightly “incorrect” billing.

So anyone know of a way to exclude those NaN values from the PERCENT 
calculation?

Thanks,
  Jacques___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] How to generate nicely formatted columns in the legend graph?

[Jacques du Toit]

>> Quick question, how can I create nice columns that won't mess up the layout 
>> if numbers increment or decrements? Something like this: 
>> http://martybugs.net/linux/rrdtool/images/traffic_detail.png
>> 
>> I'm not trying stuff like here below, but it doesn't handle changing values, 
>> the columns shift position then...
> 
> As you've found out ... it's tricky sometimes.
> Make sure that your labels are all the same length - so pad out "User" and 
> "System" so they are the same length as "User (nice)". Then make sure that 
> all your value formats are of sufficient length that all values you might see 
> will fit in the given size of display format. Between them, I find that these 
> will line up the columns nicely, except ...
> 
> The biggest problem I find is getting the column headers to line up. I end up 
> using trial and error to get the column headers "near enough" as no matter 
> what I do, I have never managed to get them to line up exactly - as there's 
> no label (like there is for the data lines), that throws the origin of the 
> text off and the rest goes off as well.


In addition to padding out the the text you should use Pango markup (you might 
have to specify arg for this) and make all the GRPINT output use fixed-width 
font by adding  tags...

 57   "LINE1:cpu_user_avg$RED:User\: "\
 58   "GPRINT:cpu_user_avg:AVERAGE:%3.1lf" \   
  

You can also pad to the left of the numbers by increasing the number eg %8.1f 
will leave quite a large space to the left of the value allowing you to make 
your columns consistent. A bit of trial and error, but play around and see what 
looks nice.

Cheers,
  Jacques



 
___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] CDEF effects entire timeline

[Jacques du Toit]

Hi

If I understand you correctly….I think you’re having issues trying to use NOW 
because rrdtool will snap to the 1 second grid skewing your calculations ever 
so slightly. I don’t think it’s really designed for that. It would probably 
expect DEF like ( TMPOUT, TMPIN, AMOUNT_OF_WATER) and then just graph the 
values with something like:

CDEF:PWR=TMPOUT,TMPIN,-,AMOUNT_OF_WATER,*,0.06978,*

And then let rrdtool calculate the value per second (which is what you have 
with your 1 second step).

But I can see that unless you measure at precisely 1 second intervals down to 
the millisecond your values will be slightly erratic as the amount of water 
will vary for each 1-second RRA.

Something I would try to do is to compensate for this using python to store the 
millisecond value since the last measurement (period = time_now - 
last_measured_time). Then do something like:

DEF:TMPOUT=…
DEF:TMPIN=…
DEF:FLOWRT=…
DEF:PERIOD=…
CDEF:PWR=TMPOUT,TMPIN,-,FLOWRT,*,0.06978,*,PERIOD,1000,/,/

But it’s a bit counter rrdtool to save the “step” in an rrd in this way. You 
might actually just be better off doing all the maths in python and just 
writing the PWR as a GAUGE value in rrdtool…that’s probably what I’d do, but 
I’m lazy with RPN that way… :)

Cheers,
  Jacques

> On 14 Feb 2017, at 20:31, SolarMan  wrote:
> 
> I have a simple application to determine the power generated by a water
> heater. One sensor (TEMPIN) measures the in-flowing cold water temperature
> while a second sensor (TEMPOUT) does the same for the hot water exiting the
> heater. The flow rate is constant and samples are stored in an rrd database.
> Using rrdgraph I want to display the time series of the temperatures plus
> the generated power in Watt. I planned to use a CDEF to calculate the power
> by multiplying the temperature difference by the flowrate and a constant and
> divide that by the time between two successive samples. A python script does
> it all for me. Samples come in nominally - but not exactly - at a one-second
> interval, therefore the CDEF does the division using a value calculated in
> python using rrdtool.last() just before every rrdtool.update() call and NOW
> in the CDEF as you might guess. Seems not all too complex you would say
> BUT: I noticed that the entire Power graph (PWR) jumps up and down over the
> entire time range when the time-interval (hence the division) changes and
> that is clearly not what I want.
> Here is the CDEF (as a python snippet):   
> "CDEF:PWR=TMPOUT,TMPIN,-," + str(flowrate) + ",*,0.06978,*,NOW," +
> str(prevtime) + ",-,/",
> I spent many hours reading all the available material on CDEF, VDEF etc. but
> this fenomenon apparently has never come up - or this is a feature "by
> design".
> How can I accomplish that the power is calculated and displayed truly
> momentary per step even when the measuring interval varies?
> Thanks everyone!
> 
> 
> 
> 
> 
> --
> View this message in context: 
> http://rrd-mailinglists.937164.n2.nabble.com/CDEF-effects-entire-timeline-tp7583504.html
> Sent from the RRDtool Users Mailinglist mailing list archive at Nabble.com.
> 
> ___
> rrd-users mailing list
> rrd-users@lists.oetiker.ch
> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] CDEF 0 in rrdtool graph

[Jacques du Toit]

Hi Oliver

I’m afraid you’ll always need a DEF from somewhere. I’ve done plenty of 
workarounds like this, including having an empty rrdfile that is never updated. 
Usually I end up doing something like this to get just a 0 value...

DEF:foo=/some/empty/rrdfile.rrd:foo:AVERAGE
CDEF:myval=foo,POP,0

Hope that helps :)

Cheers,
  Jacques

> On 14 Feb 2017, at 20:49, Olivier Doucet  wrote:
> 
> Hi,
> 
> 
> 2017-02-14 19:44 GMT+01:00 SolarMan  >:
> Hello Olivier,
> The error message is confusing and should actually read:
> rrdtool.OperationalError: rpn expressions without DEF or CDEF variables are
> not supported
> as it does in other cases.
> The meaning is clear: you must use a DEF or CDEF in your CDEF definition.
> Regards,
> hans
> 
> Thank you Hans for your quick response. I understand the documentation is 
> wrong then.
> 
> Any idea how to easily create a DEF with 0 value all along ? I'm using it in 
> a automatized script that write the whole 'rrdtool graph' line, and when 
> original DEF file is not present, I would like to replace it with 0 to keep 
> the same logic.
> 
> Olivier
> 
> 
> ___
> rrd-users mailing list
> rrd-users@lists.oetiker.ch
> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

[rrd-users] time component of a VDEF

[Jacques du Toit]

Hi

I see this in the rrdtool_rpn man page under the VDEF section:
MAXIMUM, MINIMUM, AVERAGE <>
Return the corresponding value, MAXIMUM and MINIMUM also return the first 
occurrence of that value in the time component.
Example: VDEF:avg=mydata,AVERAGE

What is the “time component” referred to here? Is there a way to reference it?

The reason I’m asking is because I’m trying to determine during which time of 
day traffic is the busiest - the so-called “busy hour”. My approach was to:

1. Force step=3600 to make it hourly averages
2. Get the hourly MAX values from the DEF
3. Determine the LTIME of the first occurence of MAX

So this is what I tried (paraphrasing a bit to simplify the rpn):

DEF:bytes_in_max=/some/path:bytes_in:MAX:step=3600:reduce=AVERAGE
DEF:bytes_out_max=/some/path:bytes_out:MAX:step=3600:reduce=AVERAGE
CDEF:max_util=bytes_in_max,bytes_out_max,MAX,8,*
VDEF:busy_time=max_util,MAXIMUM,LTIME

But seems that the VDEF is not valid with the LTIME added… I get “Unknown 
function MAXIMUM,LTIME” error. :-/

Any idea how I can find the time value for the hour at which MAXIMUM was for 
the VDEF? This is rrdtool-1.4.8, I haven’t tried with the latest rrdtool yet, 
but just want to know if I’m on the right track of if there is a better way...

Thanks in advance. :)

Cheers,
  Jacques

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users