Re: [rrd-users] How to graph slow changing counters

[Simon Hobson]

Thorsten Erdmann  wrote:

> you are right (and I know) that with such a low resolution it is difficult to 
> draw sensefull data. But I want to understand what RRD does. I think the 
> aggregation is wrong:
> 
> If I sample the data every 5 minutes (1/12h) and get a single 1kWh step in 
> one of these 5 minute intervals then I get a peak about 0,3kW in the dayly 
> graph. But why? I get headache with these calculation. I would say the 
> formular is 1kWh / (1/12)h but that gives 12kW, which is definitely wrong.
> 
> Another weird thing is what happens on the weekly graph. I get some "nice" 
> step lines up and down. Shouldn't it be simply a flattened out curve just as 
> you said. I cannot give you a picture right now, because my server is down 
> for implementing the S0 photosensor, just as you suggested. I will implent 
> both. So I can adjust the S0 pulse counter once a day or so to the real 
> counter value.



> This is how I defined the data:
>  
> ret = rrdtool.create(rrdname + '.rrd', '--step', '300', 
> '--start', str(starttime),
> 'DS:Gauge:GAUGE:600:U:U',
> 'DS:Counter:COUNTER:600:U:U',
> 'RRA:AVERAGE:0.5:5m:24h',
> 'RRA:AVERAGE:0.5:2h:31d',
> 'RRA:AVERAGE:0.5:6h:31d',
> 'RRA:AVERAGE:0.5:1d:1y')
>  
> And this is how I graph them
>  
> ret = rrdtool.graph(rrdname + '24h.png', '--start', 'end-24h', '--end', 
> 'now', 
> '--upper-limit', '650', '--lower-limit', '600', '--rigid', 
> '--right-axis', '0.1:-0.006',
> 'DEF:gauge=./' + rrdname + '.rrd:Gauge:AVERAGE',
> 'DEF:counter=./' + rrdname + '.rrd:Counter:AVERAGE',
> 'CDEF:cnt1000=600,counter,1,*,+',
> 'LINE1:gauge#00ff00:Gauge',
> 'LINE1:cnt1000#ff:Counter')
>  
> The gauge line looks good, but the counter line looks weird.


Looking again I have some questions and observations ...

You have a GAUGE and a COUNTER in your rrd - what are you collecting, and how ? 
Counter would be correct for the meter, what are you putting in the gauge ?
Knowing this may affect the advice offered.

I'd recommend you store a bit more data than you are (e.g. 25h for daily 
graphs) - and that links in with graphing.
As you've written them here, I think you'll find the wrong aggregate data being 
used. If you do "now-24h" to "now" then you'll find that the first data point 
in that period is just outside your 24 hours @ 5 minute - so instead you'll 
find that the 2 hour aggregation is used (see more below). For that reason I've 
always calculated the exact end time of the last period - i.e. in bash, 
something like :

step=300
durn=87600
etime=`date +%s`
etime=$((${etime}/${step}*${step}))
stime=$((${etime}-${durn}))
... rrdtool ... "--start ${stime} --end ${etime} --step ${step} ...

That way, your strt and end times are aligned to the bucket boundaries of the 
RRD file, and you've explicitly defined the data resolution to be used. This 
gives more reliable selection of data !
I tend to set step and durn in a case statement, along the lines of :
case period in
  daily ( step=300;durn=87600 );;
  weekly ( step-7200; ...
and then the graphing command is common to all resolutions.


A bit more on how RRDtool selects data. Bear in mind this is from memory which 
may have faded a bit ...
If you don't tell it otherwise, it will select the highest resolution data that 
supplies the entire graphing period selected. So selecting "now" as the end 
point, you will be selecting an incomplete bucket at the end of the graph. From 
memory, I think this also means that "now - 24h" goes part of a bucket past the 
first bucket in the RRA - because when the current incomplete bucket is 
started, it takes the place of the one from 24h ago.
As this goes beyond the 300s stored data, the next resolution is selected - 
hence you day graph that should show 300s buckets, is actually drawn with 2hr 
buckets - so coarser data AND potentially a gap at the end (e.g. if "now" is 
11:55 UTC, you'll only see data up to 10:00 UTC because the 10:00 to 12:00 UTC 
bucket is not yet complete).


Simon

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] How to graph slow changing counters

[Simon Hobson]

Thorsten Erdmann  wrote:

> I want to graph the data of my energy meter. But it is measuring only in 1kWh 
> raster. So with normal power usage I have every 2..4 hours a step of 1kWh. 
> The meter displays it’s data as kWh, so it gives e.g 600 for the absolute 
> counter value of 600kWh.
> I sample the absolute energy every 5 minutes. So I took this as step and 600s 
> as heartbeat.
>  
> What I get is short peaks with a height which is displayed as 330u over a 
> period of 24h. With longer periods like a week the values get smaller and 
> smaller and there are weird stepping curves. So how can I measure/graph such 
> low resolution measures in a senseful way?

Realistically, there's not a lot you can do with such low resolution data. 
You'll get a spike each time a unit clocks up, then as you zoom out for a 
longer scale view, it will flatten out to (as you've found) a rather low value. 
If you are using a unit every (say) 3 hours (taking a mean between your 2 and 4 
hours), then that equates to an average of 333W - or if you are just storing 
kWH, 0.333kW. Again depending on how you store and calculate, this then equates 
to around 93µJ/s - at least, if my brain is working right this afternoon !


You don't say how you are interrogating the meter - is it via a data source it 
provides ?
If the meter is providing you with a counter value when asked, then this will 
give you the best long term accuracy - your counter will always match what is 
in the meter.

As an alternative, many meters provide a pulsed output of some sort (even if 
it's only an LED flashing). A bit like the old Ferraris disk meters with their 
"X revolutions/kWH", the meter pulses will give you an indication when a much 
lower quantity of energy has been used - it should say on the face of the meter 
how many pulses/kWH. The downside to counting these pulses is that if your 
counter isn't running or otherwise misses them, then you lose sight of energy 
consumed and your RRD calculations will show a lower figure than the meter.

You could, of course, collect both sources - use the pulses for short term 
indication (ie better graphing up to a few days duration) and the register 
count for longer term.
___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] rrdtool graph: COMMENT and empty lines(lf) - comprehension question

[Simon Hobson]

Daniel Schlifka  wrote:

> I have a little comprehension question, i'm still struggling/fighting with 
> rrdtool graph.
> Today i tried to create an empty line within the plotted graphs legend(to 
> separate two lines of text optically).
> Following older ML records and StackOverflow Threads i found some variants to 
> achieve that.
> i.e. "COMMENT:\n" && COMMENT:" " && COMMENT:"\n" && COMMENT:"\s\n"
> 
> Unfortunately none of these worked for me. I tried in different variations 
> without success.
> 
> What interestingly did the trick was:
> COMMENT:"\t\n"
> 
> 
> So here's the question - is the related to how COMMENT handle strings? Are 
> linebreaks considered as part of the string or does COMMENT fail because it 
> considers the 0x0a as terminator and therefore not part of the string(unlike 
> tab) and "believes" there is nothing to print?
> 
> Another theory would be this has nothing to do with rrdtool but is caused by 
> some bash-expansion.

The latter sounds a likely issue. Checking a script from the past, I see I've 
used these lines in BASH scripts :
COMMENT:\"min  ave  max  last\n\"
COMMENT:\"  Data to ${PrintedTime}\n\

It may be something specific to my scripts, in these cases it was a matter of 
doing :
echo "" | /usr/bin/rrdcgi --filter - 2>&1 > /dev/null
where  could amount to thousands of lines of script 
generated output !

It's was a long time ago that I wrote those scripts, one thing I do recall was 
struggling at times to get the right quoting for quote marks.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] How to upgrade RRDtool?

[Simon Hobson]

Stage  wrote:

> I have a small question for you. I work as an Intern,and i got the assignment 
> to upgrade the monitoring solution.
> Since i cannot find a guide to upgrade RRDtool on the internet. I felt urged 
> to ask the mailing list.
> The plan is to upgrade RRDtool from 1.3.8 to 1.5.6 but i don't know how.
> I am using CentOS 6 64-bit, with Cacti 0.8.8h
>  
> Does anyone know how to do this?

It depends largely on how it was originally installed.

If it was installed by way of the distribution's package manager then it's 
normally just a case of telling the package manager to upgrade that package - 
which usually also means upgrading packages it depends on.

If installed from source, then it would be a case of downloading the source, 
compiling it, and installing the binary.

Regardless, you need to plan the upgrade. Have a plan for what you are going to 
do, when you are going to do it, how you are going to test that it all works, 
and what you are going to do if it doesn't ! Having a good, recent, backup is 
always a good idea ! But these should be covered by policies at whatever 
establishment you are working at - and you should consult your supervisor for 
guidance on this.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Making RRDs longer

[Simon Hobson]

Benjamin Benziger  wrote:

> 3 years ago I've created some rrds that contain 10 minute values for 3 years. 
> To prevent data to get overwritten I want to enlarge the database to let's 
> say 10 years. How do I do that?
> 
> rrdresize is only to change the number of rows in a specific rra?

Which is what you want to do.
RRD does not store a set length of time of values - it stores a preset number 
of consolidated values. You can have multiple RRAs, eachof which covers a 
different timescale.

It sounds like you have just one RRA, and all you need to do is set it to store 
more values.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] formatting the rrdtool graph commands/scripts

[Simon Hobson]

Andreas Schuldei  wrote:

> Hi,
> 
> I am looking for inspiration for formatting my rrdtool graph commands.
> given that this is some form of prgramming, i would like to extract common 
> parts like the preamble which is similar in many of my graphs, or format the 
> script so the meaning becomes more obvious. Many of my graph commands are 
> longish (80lines) and are hard to grock after a while. How do you deal with 
> this?

A lot of my graphs are (CGI) script generated, using parameters for timescale 
etc. So parsing options (eg day, week, month, year) from the HTML request and 
setting (eg) the start and end accordingly. Some of the scripts (eg traffic by 
IP address) loop through a number of items building a stacked graph.

In one script I might have one function to generate the common preamble/header, 
a number of functions to generate graph bodies, and one to generate the trailer 
- and the output from them is piped through rrd_cgi. Along these lines :

do_header () {
  echo "graph title=${title} start=${start} 
  
}


And finally :
( do_header
  do_body
  do_footer ) | rrd_cgi 2>/dev/null > ${graphfile}


But in reality, there are lots of ways to do it - which is best for you depends 
on your requirements, skills, and what's already installed on the system.
___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Intermittent data causing stepped graphs

[Simon Hobson]

jamiedg  wrote:

> Recently though I've hit a problem that maybe someone can help with;
> My temp sensors would normally save data at least once a min which worked
> fine, but to extend battery life the newer sensors are only transmitting
> data when the temp changes or at most every hour. This causes intermittent
> data causing the charts to sometimes have a stepped output.
> 
> 
>  
> 
> When I post the same data to the cloud service EmonCMS I get a more smoothed
> output.
> 
> 
>  

The obvious observation is that the latter is wrong FOR THAT DATA. The data you 
are feeding in doesn't have the data to show a sloping line, so drawing a 
straight line between the points is strictly speaking incorrect FOR THAT DATA.

Your real problem is that the data itself is stepped. When you say that the 
sensor only sends data when the temperature changes, it presumably has some 
quantisation - it appears to be reporting to a resolution of 0.1˚. I suspect 
that the older sensors are reporting to a higher resolution - eg if it reports 
to 0.01˚ then you'd not see the steps as they'd be smaller than the pixels on 
the graph, both in the Y axis due to the resolution, and in the X axis due to 
the frequent reports.

Do you still have any of the old sensors ? If so then I'd suggest comparing the 
raw data. My guess is that the old ones report to a higher resolution, which 
combined with frequent updates will give you a smooth graph. The new ones are 
almost certainly quantising to a much lower resolution (otherwise there'd be 
little or no battery saving) which is what's giving you your problem.


> If the data is Undefined I use the Prev value (sen3,UN,PREV,sen3,1,*,IF)

That is probably wrong as well. You are explicitly telling RRD to draw a flat 
line there.

> Can anyone please offer any suggestions or help to smooth the stepped graph
> output.

Fix the input ?

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Diurnal average

[Simon Hobson]

Helios Solaris  wrote:

>> Over how many days ?
> 
> Over the entire database (2 years).

Ouch - as already mentioned by Alex, that could be a LOT of processing and also 
prevents you using consolidation.

> Yes, using time shift may work, but it's not really flexible when I want
> to change parameters.

Depends on what you want to change. I would script it so that the parameters 
are easy to change - that's how most of my graphs work, passing (eg) start, 
end, and step values that are generated by a case statement in a shell script 
that takes "day", "week", "month", "year" as a parameter.

I do have another thought ...
How many time periods do you want during the day ? If it's not too many (where 
"too many" is somewhat subjective), then you might want to look at turning it 
around - store (say) midnight-1am in one rrd file, 1am to 2am in another, and 
so on (in this example, using 24 rrd files). Each rrd file can then use 
consolidation.

And (I'm sort of typing as my thought process flows here), I'm led to another 
option - but which limits consolidation again.
For midnight to 1am, use a time of day function to set a CDEF to the stored 
value if the function is true and to either zero or unknown if not. Then 
another time function and cdef for 1am to 2am, and so on. Then you can use a 
VDEF to get a consolidation across the whole CDEF to get a single value - which 
you can then print with a PRINT (not GPRINT) statement.
However, a quick look at the docs suggest that there isn't a simple "time of 
day" function which complicates matters somewhat.
I **think** this may do it !
DEF:x=somefile.rrd...
CDEF:tod=TIME,DUP,86400,/,FLOOR,86400,*,-
# get time of sample, divide by 86400 (1 day), take integer part, multiply by 
86400 (get time value of midnight at start of day) and subtract from time of 
sample to get the number of seconds since midnight.

CDEF:h0=tod,0,GE,x,3600,LT,[0|UNKN],IF,[0|UNKN],IF
# If 0<=tod<3600 then get x else get [0|UNKN]
VDEF:h0ave=h0,AVERAGE
# Get an average value.
PRINT:h0ave,"%6.2lf'

CDEF:h1=tod,3600,GE,x,7200,LT,[0|UNKN],IF,[0|UNKN],IF
and so on ...


This is the sort of thing I'd just script in Bash - almost trivial to generate 
an arbitrary number of statements covering appropriate timescales

It's so long since I've used the TIME function, dunno if it should be 
tod=TIME... or tod=x,TIME... (ie get x, then get the TIME value for the current 
sample.

Whether you use zero or unknown for the times outside of each window depends a 
little on your requirements. I suspect that unknown (UNKN) is probably the 
correct one to use.

Whatever happens, I don't think you can calculate and graph in one go. You'll 
have a "graph" going back an arbitrary time to generate just one day's worth of 
data - and I don't think you can have a graph covering (say) 2 years of data 
but only drawing a line for one day. Something like gnuplot might be a better 
fit for that.

I'd also expect this to be "quite slow" and resource intensive. Lets consider 
the case where you do it only by hours. You are creating 1 duplicate of your 
dataset with time of day calculated, then another 24 duplicates. So that's 
effectively 25 copies of your database in memory at once.
Go down to (say) 5 minute periods, and it then means 289 (1 + 288) copies of 
the database in memory at once !
AND you must store all the data you want to use. That means, if you want 5 
minute slots for a full 2 years, you need to store 5 minute consolidated data 
points for 2 years - that's 210240 samples to store (though I guess that's not 
so much data).

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Diurnal average

[Simon Hobson]

Helios Solaris  wrote:

> Is there an easy way to plot a diurnal average?
> I.e. the average from all days at 00:00, 00:01, 00:02 etc. so that I
> have a plot of an average day?

Over how many days ?
You might be able to do it with some of the time shift functions. If, for 
example, you wanted to do one day and average over the last , then I think you 
might do it like this :

Assuming START and END are already defined variables which give the start and 
end of our 7th day, and wiring "sort of shell script" (I'm used to dynamically 
generating graphs from Bash scripts and piping the resulting rrd command 
through rrd) ...

graph ... start=$START end=$END
def:day7=datafile.rrd:data
def:day6=datafile.rrd:data:start=$START-1day:end=$END-1day
...
def:day1=datafile.rrd:data:start=$START-6days:end=$END-6days

def:dayave=day1,day2,day3,day4,day5,day6,day7,add,add,add,add,add,add,7,div


You should now have a cdef with values which are the averages of the 
corresponding time slots over the last 7 days. I would suggest having a good 
read of the docs as the syntax above may not be completely accurate.


___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] How to generate nicely formatted columns in the legend graph?

[Simon Hobson]

K. de Jong  wrote:

> Quick question, how can I create nice columns that won't mess up the layout 
> if numbers increment or decrements? Something like this: 
> http://martybugs.net/linux/rrdtool/images/traffic_detail.png
> 
> I'm not trying stuff like here below, but it doesn't handle changing values, 
> the columns shift position then...

As you've found out ... it's tricky sometimes.
Make sure that your labels are all the same length - so pad out "User" and 
"System" so they are the same length as "User (nice)". Then make sure that all 
your value formats are of sufficient length that all values you might see will 
fit in the given size of display format. Between them, I find that these will 
line up the columns nicely, except ...

The biggest problem I find is getting the column headers to line up. I end up 
using trial and error to get the column headers "near enough" as no matter what 
I do, I have never managed to get them to line up exactly - as there's no label 
(like there is for the data lines), that throws the origin of the text off and 
the rest goes off as well.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Quantisation issues - how do i get the math right?

[Simon Hobson]

Andreas Schuldei  wrote:

> I just got an idea that there might be some confusion about the word "Power". 
> 
> I am interested in both the engergy (Energie) and the power (Leistung) 
> output. Currently I am dealing with the power graph. Energy is the sum (or 
> integral over time) of the power. So ABSOLUTE (implemented as an ever 
> increasing staircase) would emulate the absorbed energy, but not the power.  

Just to be clear, RRD **ONLY* does rates - so power flow. To get totals (ie 
accumulated energy) you multiply the rate (where appropriate, an averaged 
value) by time. SO whatever you feed in, you need to arrange that it's a rate.

In your case, I think you have no option other than to use the counter as it 
is, and either multiply or divide by an appropriate factor to get the power 
instead of some arbitrary number of "somethings".
In your case, I suspect the best you'll get is to store : flow rate (f) and 
treat the values you read as a counter, t1 and t2, and use derived/computed DSs 
for "t2-t1" (temperature difference) if that might be useful in future, and 
"f*(t2-t1)*k" (where f is the flow rate and k is the "fudge factor" to store 
actual power (energy flow rate). It's going to be very uneven due to the 
chunkiness of the data updates you can feed in - there isn't really anything 
you can do about that. Once you've done some aggregation (eg to daily figures) 
then this lumpiness should reduce. However, you will get false data for MAX 
values.

Simple answer - if you only get some data in big chunks, there's not much you 
can do to make it "not chunky". IIRC it's come up before, possibly in relation 
to a weather station and a coarse reading from a rain gauge.
The only other thing you could do is simply not look at fine grained data. If 
(say) on a typical day the counter might step (say) once in half an hour - then 
have your step at (say) 2 hours. You'll never be able to look at data finer 
grained than 2 hours. But your normalised data will vary between (say) 3 per 2 
hours when flow is down, to (say) 6 per 2 hours when it's higher. Better than 
it doing zero, zero, zero, , zero, zero, ...

But you will still want to feed in data fairly regularly, because you'll want 
the average temperature difference over each 2 hour period - assuming it varies 
significantly over that sort of timescale (eg clouds appearing/disappearing) 
then you don't want just a snapshot each 2 hours.


The other option is to change the flow meter to one giving a useful output !

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] rrdtool fetch with a resolution != rrd_step ?

[Simon Hobson]

Marek Schimara  wrote:

> I'm trying to get data out of an RRD database (created with step=10) with 30s 
> resolution. I have read 
> http://oss.oetiker.ch/rrdtool/doc/rrdfetch.en.html#RESOLUTION_INTERVAL and 
> I'm using -start, -end timestamps aligned to 30. However I can't get rrdtool 
> fetch to give me data with the requested resolution:

AIUI, you can only fetch a resolution that exists - it won't do any 
consolidation/interpolation to transform from what is stored to an arbitrary 
resolution. I see this isn't mentioned on that page.
___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Using Granular RRA Automatically

[Simon Hobson]

Stephen R Guglielmo  wrote:

> If I were to do something
> like the following:
> 
> rrdtool create filename.rrd \
>  --step '86400' \
>  'DS:Sites:GAUGE:172800:0:U' \
>  'DS:DeletedSites:GAUGE:172800:0:U' \
>  'DS:OutdatedSites:GAUGE:172800:0:U' \
>  'DS:Users:GAUGE:172800:0:U' \
>  'RRA:LAST:0.3:1:3660'
> 
> This will basically keep every single value entered for 3660 days (~10
> years), correct? It won't consolidate any values?

Correct.

> I understand this
> will have increased disk usage in the .rrd file, but are there any
> other downsides to doing this?

For this application, I don't think so.
In general, it's a trade-off between the amount of data retained (ie how far 
back you can go vs how detailed it is), disk usage, and resources needed to 
draw graphs.
So if you were to keep (say) traffic for every 5 seconds for the last 10 years 
with no consolidation, then drawing graphs would be very I/O and processor 
intensive as it would have to do a lot of consolidation to generate the graph 
(as well as taking a lot of disk space). In your case, even keeping 
unconsolidated data for 10 years, it's not a large amount of data, and unless 
youre system is very resource constrained, I doubt it's going to make any 
difference.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Using Granular RRA Automatically

[Simon Hobson]

Stephen R Guglielmo  wrote:

> With the above database, I update it once a day with four data points.
> I record MIN, MAX, AVERAGE, and LAST on a weekly basis for 10 years. I
> also record MIN, MAX, AVERAGE, and LAST on a daily basis for 35 days.
> 
> My question is: Since the RRAs are not named, if I create 1 month, 1
> year, and 2 year graphs, will the 1 month graph automatically use the
> more granular RRAs (the "daily" 35-day ones as opposed to the "weekly"
> 10-year ones)?

Yes, it'll automatically use the "best fitting" RRA - which in most cases will 
be the right thing. You can also specify the resolution in your graph command.


> Here is an example `graph` command that I received from the RRD Wizard:
> 
> rrdtool graph 'sites-1month.png' \
> --title 'Sites, 1 Month' \
> --vertical-label 'Number of Sites' \
> --width '800' \
> --height '300' \
> --full-size-mode \
> --start end-1m \
> ...


It is recommended to explicitly specify the end time, and to round it to an 
integer multiple of the RRA resolution. So, for an RRA with a resolution of 1 
day (86400 seconds), specify the end time as an integer multiple of 86400.
And specify the start time as end-(integer multiple of resolution), and ideally 
the multiple will be the number of pixels in the graph which avoids rescaling 
the data to fit.

Longer explanation, RRD will use the RRA which best fits - specifically it'll 
pick the RRA(s) providing the most data within the specified timescale, and if 
more than one, the one with the highest resolution. So in your case, if you 
plot a graph over 36 days then the 1 day RRA won't fill it and it'll switch to 
a coarser RRA. for 35 days and below, it'll pick the 1 day RRA as having the 
higher resolution.
But there are cases where a coarser resolution may provide more data - though I 
don't think they are very common.

Consider if you aggregated at 2 day and 5 day resolution - contrived but 
illustrates the point. There will be times where the 5 day RRA covers a day 
later than the 2 day RRA (basically where it's a multiple of 5 days since unix 
epoch) - and so the "use the RRA with the most data in the desired period" 
method will switch to the 5 day RRA.
For a case like this, you'd need to explicitly specify the RRA to be used, and 
your data would stop a day earlier while showing more detail.
___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] RRDs::graph and DEF... MIN

[Simon Hobson]

Joachim Hartmann  wrote:

> Hi,
> I am monitoring my heating by a raspberry-pi using rrd data base:
> - TMP_V = flow temperature
> - TMP_R = return temperature
> - WOHNZ = indoor temperature
> - AUSEN = outdoor temperature
> - PUMPE = circulating pump (running or not)
...

> All looks fine as lpng as the outdoor temperatrure is above 0 degree.
> Otherwise the pump graph ends at the O line, it looks ugly!
> 
> My idea: drawing first something like this:
> 
>  ...
>  "DEF:Off=/home/pi/heizung.rrd:ausen:MIN",
>  "AREA:Off#F0EFEF",
>  "LINE1:Off#424242:",
>  ...
> 
> 
> but - my - RRDs::graph does not exept the line
> "DEF:Off=/home/pi/heizung.rrd:ausen:MIN",
> !!
> 
> Anyone an idea?

How about running through an RRD Graph once, but rather than drawing anything, 
use PRINT (NOT GPRINT) to get the minimum value of ausen and then you have a 
value (Perl variable) you can use as a constant in a second RRD Graph run.

So use VDEF to get the min value, then do some maths (further CDEFs) to draw 
the pumpe value between this min value and 1


> Sorry English is not my native language and school is long ago

No need to apologise, your English is a lot better than my German from school 
(also a lot of years ago) !

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Daily consumption

[Simon Hobson]

Guillaume Betous  wrote:

> But do you confirm that I can update with a higher freq than de DB ? Let's 
> say update every 30 sec still have sense ?

Yes, you can submit updates more or less as often as you want - all that 
happens is the intermediate data is collated until the end of the step.
Have you read Alex's excellent tutorials http://rrdtool.vandenbogaerdt.nl ? 
Rates, normalizing and consolidating is really essential reading.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Daily consumption

[Simon Hobson]

Bram Langen  wrote:

> ... but for full flexibility and data analysis (such as comparing same period 
> year over year) I let the Raspberry Pi write the counter values once a day to 
> a text file. Every so often I import the CSV data into MS-Excel, allowing any 
> data comparison you want.

You can (AIUI) do time period comparisons by using the time offset functions.
I've not done it myself, but AIUI you can create two CDEFs and offset one of 
them by (say) 365 days, then plot them on the same graph.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Question regarding missing data points

[Simon Hobson]

Alex van den Bogaerdt  wrote:

> You are not writing points, you are writing rates which are measured
> during some interval.
> The rate "8" is for time 1477433700..1477434300. This is however not
> allowed because of your heartbeat setting and gets lost.
> 
> What happens if you do update at time 1477434000, using value U  ?

Would that make a difference ?
I know that with a counter type, the second interval would still be unknown 
because the (b-a)/t calculation can't be done if a is unknown - but how does 
RRD handle the gauge type ? As "everything is a rate", does this limitation 
still apply with the gauge type ?
___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Unsubscribe

[Simon Hobson]

Jeff Letendre  wrote:

Absolutely nothing at all, but quoted an entire day's worth of list messages.
In common with virtually all mailing lists, there's a footer on every list 
message. It has a link to the mailing list website 
(https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users) where you can unsubscribe 
or otherwise manage your subscription.

In addition, if you look in the mail headers, which unfortunately most mail 
user agents hide, you'll find this :
> List-Id: Help and discussion among RRDtool users 
> List-Unsubscribe: , 
> 
> List-Archive: 
> List-Post: 
> List-Help: 
> List-Subscribe: , 
> 

Lastly, in common with almost all mailing lists - requests to a list should be 
sent to ${name_of_list}-request@${server}. Hence, to unsubscribe you send your 
REQUEST to the REQUEST address which is as in the headers above, 
rrd-users-requ...@lists.oetiker.ch?subject=unsubscribe

This is pretty well standard across mailing lists, AND it will have been 
included in the welcome mail sent when you subscribed in the first place.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Confusion about VDEF total and graph values

[Simon Hobson]

"Alex van den Bogaerdt"  wrote:

>> That would also work visually, but what I had in mind was not to reduce
>> the
>> width, but create the bars by separating them with vertical lines of the
>> correct height.
> 
> I have difficulty understanding what you describe here.

I interpret him as : think about drawing each column in the graph as a box with 
a thin (I guess 1 pixel is the minimum) outline round it in a contrasting 
colour - but don't draw the top or bottom lines. You've now got vertical lines 
separating the filled area.

It's slightly more subtle, you actually want to only draw a line on either the 
right, or the left, of each column - otherwise the side lines from two adjacent 
columns combine to form a 2 pixel wide line.

It also won't work if the columns are only 1 pixel wide.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Confusion about VDEF total and graph values

[Simon Hobson]

humke  wrote:

> I can see where you are going with this. I want the total surface of 24 bars
> to be the same as the surface for the 1 bar. So AVERAGE would be the correct
> CF choice here, because only that CF would give me the same surface.  I will
> try this.

Correct

> I could (or maybe even should) still use LAST for the highest resolution
> (one hour) in this case right? As it's one PDP or are all my 10 second tries
> to store the value also CF'ed?

What is your step size - that has a large impact on what you need t put in and 
what you get out.

If (for example) your step size is one hour (for the gas), and you ask for data 
with a resolution of one hour, **AND** you have put in data exactly every hour 
ON THE HOUR - then you'll get out what you put in (barring rounding errors and 
conversion from counter to rate).
If you remove the "on the hour" bit of that, then what you get out will be 
different to what you put in. So say your meter gives you data timestamped at 
20 minutes past the hour, each one hour slot within the RRD will comprise 1/3 
of one value plus 2/3 of the next value. Ie, to give you a value for the slot 
from 12 noon to 1pm, it'll use 1/3 of the value from 11:20 to 12:20 plus 2/3 of 
the value from 12:20 to 13:20 - and you will not see any data (it'll return as 
NaN) for that slot until the 13:20 data is entered.

Consolidation functions have no effect on this as this is the standard 
normalisation to map the data you put in to the primary data points (PDPs) 
defined by your step size.

Updating more often than the step size doesn't really change this - all the 
interim data is just accumulated up until you pass the next step time and the 
PDP is complete. So the above example is modified a bit. Assuming that the data 
actually only changes at 20 minutes past the hour, if you updated (using the 
same counter value) every minute, the 12:00-13:00 PDP could be completed as 
soon as 13:00 (if you update "on the minute") or at the latest by 13:01.

This would give you different values though.
Assuming usage was zero beforehand, 3600 units registered in the 12:20 update, 
and 7200 units registered in the 13:20 update.
Updating hourly with the data timestamp will give usage for 12:00-13:00 of 1.67 
units/s (1/3 x 3600 + 2/3 * 7200). If you update every minute on the minute, 
all the 3600 units would appear to have been used between 12:19 and 12:20 with 
zero for the other 59 minutes. At 13:00, RRD would "close" the PDP, average the 
3600 units across the hour, and arrive at a value of 1 unit/s.

In reality, neither value is right, and neither value is wrong ! You might have 
been (for example) running the shower (and hence burning plenty of gas) before 
12, or after 12, and using nothing at other times; or you might have had the 
heating on, and burned the same amount of gas at a steady rate during the whole 
hour - but the meter won't tell you. Just like the example Alex uses in his 
tutorial - you might have driven slowly for an hour, or driven like a lunatic 
for part of the time and stopped for a coffee, in either case the odometer will 
only tell you total distance travelled unless you read and store the values at 
frequent intervals.


Now, what does the CF do to this ? Actually nothing ! If "a" is a simple 
number, then min(a)=avg(a)=max(a)=last(a). However, if "a" were a set of 
numbers 1,2,3,40,4, then min(a)=1, avg(a)=10, max(a)=40, and last(a)=4
In my experience, LAST as a CF is only of use for printing in the legend to 
show the last value on the graph, or rather, the last value that went into the 
last CDP shown on the graph. So taking those numbers I just used, you could 
have a column on the graph (assuming you are using a consolidation of 5 PDPs 
into one CDP) that's 10 units high, and assuming no larger values, last would 
show 4.

So if you have a step size of 1, and graph hourly data, the consolidation 
function used is effectively a null operation - assuming you have a CF 
consolidating 1 PDP into a CDP !

> I do need to read the tutorials again, because I am getting all these 
> questions about stuff I thought I understood.

To be fair, there's come fairly complicated stuff going on there, it does take 
some getting your head around.
___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Confusion about VDEF total and graph values

[Simon Hobson]

Bram  wrote:

> You may want to consider splitting the databases.

That's a "six of one, half a dozen of the other" choice really. In this case, 
it's not too likely that you'd be wanting to add or abandon data sources, but 
I'd probably also use separate files.
The main reason for splitting them would be to allow complete independence of 
updates. As it is, you have different update periods for the two data sources, 
that alone says to use separate data files.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Confusion about VDEF total and graph values

[Simon Hobson]

humke  wrote:

> By the way, I am not sure why you don't see the code with the variables.
> It's there when I lookup the post via the forum solution on oetiker's
> rrdtool website:

Same here - I see it on the website, but not in the emails I got from the list. 
My guess is that it got stripped in the plain-text conversion by the mailing 
list manager.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] COMPUTE DS & altering existing rrd file

[Simon Hobson]

I wrote:

> Can I confirm I have this right ...
> 
> If I want to have a DS showing if the measured value was above some threshold 
> (in this case, network traffic above say 80%) I can use a compute DS like :
>  DS:pc80:COMPUTE:in,,GT
> 
> So if the "in" value is greater than the threshold value ( here) then 
> pc80 will be 1, and this will get consolidated so that later I'll be able to 
> see (for example) a consolidated value that might show (say) 0.2 meaning that 
> the "in" value was above the threshold for 20% of the time during that 
> consolidated period.
> 
> And I can add this with rrdtune with
>  rrdtool tune somefile.rrd DS:pc80:COMPUTE:in,,GT

I assume it's right since there's been no corrections. Also, I assume this is 
done after computing the value for each PDP - so if I have a step of 300 (5 
mins) it'll compute the rate for each 5 min step before applying the RPN 
expression to calculate the computed value ?

> What about if (or rather, when) I need to change the thresholds later ? 
> Specifically I'm thinking that if we currently have (say) 100Mbps service, 
> and we upgrade it to (say) 200Mbps, I'll need to change the threshold (and 
> thus the computation for new data) without affecting historical data. I don't 
> see an option for rrdtune to do this.
> 
> 
> Lastly, in what version does rrdtune get to add a DS ? I have 1.4.7 (Debian 
> Wheezy) and the man page doesn't mention adding or deleting DSs.

And a quick test shows that "rrdtool tune test.rrd 
DS:InPC:COMPUTE:In,1,/" just shows current values for the DSs without 
doing anything - the same as just "rrdtool tune test.rrd". Later versions of 
rrd in the Debian archives have dependencies not met on Wheezy - unless someone 
felt like packaging it up for backports.


___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

[rrd-users] COMPUTE DS & altering existing rrd file

[Simon Hobson]

Can I confirm I have this right ...

If I want to have a DS showing if the measured value was above some threshold 
(in this case, network traffic above say 80%) I can use a compute DS like :
  DS:pc80:COMPUTE:in,,GT

So if the "in" value is greater than the threshold value ( here) then pc80 
will be 1, and this will get consolidated so that later I'll be able to see 
(for example) a consolidated value that might show (say) 0.2 meaning that the 
"in" value was above the threshold for 20% of the time during that consolidated 
period.

And I can add this with rrdtune with
  rrdtool tune somefile.rrd DS:pc80:COMPUTE:in,,GT

What about if (or rather, when) I need to change the thresholds later ? 
Specifically I'm thinking that if we currently have (say) 100Mbps service, and 
we upgrade it to (say) 200Mbps, I'll need to change the threshold (and thus the 
computation for new data) without affecting historical data. I don't see an 
option for rrdtune to do this.


Lastly, in what version does rrdtune get to add a DS ? I have 1.4.7 (Debian 
Wheezy) and the man page doesn't mention adding or deleting DSs.
On this, could I suggest that when documenting (the online docs are great BTW) 
new features, adding a small "added in version x.y.z" would be helpful given 
how much development there's been.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] montor size of dir

[Simon Hobson]

On 19 Sep 2016, at 16:24, Martin Friberg  wrote:

> ok i use GAUGE for temp mayby wrong?

Sorry, my mistake - gauge is correct.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] montor size of dir

[Simon Hobson]

Martin Friberg  wrote:

> if i run `du -sm /path/to/dir// |sed -e s/". "//g|awk '{$1=$1 ;printf  $1}'
> 
> i get the size of that directory in mb all good
> but how do i go about and create a rrd file i know how to create one for 
> tempereature but i dont think that would work here :)
> 
> should i use COUNTER or ?

Same as for temperature - use absolute.


___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Counting the number of times a threshold has been reached and print that number, trouble with TOTAL

[Simon Hobson]

Alexander Topolanek  wrote:

> But the result is weird... the line for hotdays is working, showing 1's for 
> all days above 30, but the total is 1.296000 M, not 14...

Did you look to see what you get if you divide 1296000 by 14 ? You'll find it's 
quite close to 86400 which is the number of seconds in a day. What you've done 
is get a rate of "one per second" for days that are hot, and totalling this 
gets you 1296000 for 15 days.
All you need to do is divide by 86400

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] does rrdtool support https or ssh?

[Simon Hobson]

Yamei Feng  wrote:

> Does rrdtool have the option of enabling secure protocols (E.g. HTTPS, LDAPS, 
> SSH, etc) for sending data securely over the network? I go through the 
> documentation on http://oss.oetiker.ch/rrdtool/, but does not find any 
> information about that.

No it doesn't.
Rrd does one thing and does it well.
There are other tools - eg OpenVPN, SSH, both of which do security well but 
don't do data storage - that will allow you to wrap a layer of security over 
your rrd data.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Changing the start and end times for daily (step=86400) consolidation?

[Simon Hobson]

mmclure  wrote:

> Is there any way to specify that daily consolidation should occur between
> times that are not midnight UTC?

AFAIK it's a known issue with no easy solution. I think the only workarounds 
have involved fiddling with the timezone to fudge things.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] RRD Graph Data

[Simon Hobson]

Leandro Costa  wrote:

> Hi Alex, I'm not graphing 24h. I've made a test with a single point in time, 
> same start and and date/time ( 20160608 18:00 ). 
> The graph shows one value while the rrdfetch gives me other one.

And what resolutions do you have consolidations for, and what consolidations 
are the two commands giving you ?

rrdgraph will automatically select the highest resolution that covers the 
required timespan, I don't know what rrdfetch will select by default. It may be 
worth explicitly specifying the resolution to be used and see what difference 
that makes.
What I'm trying to say is that the graph may be showing the average for the 
previous 5 minute period, while fetch is showing you the average for the 
previous 30 minutes.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] only getting nan values

[Simon Hobson]

Jim Sher  wrote:

> RRD creating scripts here: (NOTE! I've been chopping and changing the scripts 
> since yesterday when I composed the original email I sent earlier to see if I 
> could jiggle something loose :-) - no luck so far)
> 
> #! /bin/bash
> 
> rrdtool create data.rrd \
> --step '60' \
> 'DS:datadown:COUNTER:120:0:U' \
> 'DS:dataup:COUNTER:120:0:U' \

So here you have a heartbeat of only 120 seconds - that means if two updates 
are 121 seconds apart then you'll get unknown values stored. The problem here 
is that we don't know if you've changed this - if you were using this value 
when collecting every 5 minutes then you'd be 100% guaranteed to get no data !

> rrdtool create load.rrd --start N --step 300 \
> DS:load:GAUGE:1200:0.0:4.0 \

OK, here you've got 20 minutes. As long as you've collected data for at least 
10 minutes then you should have something stored.




> As for rrdtool fetch, I can't seem to get the syntax to get any results right 
> :-(

>From one of my systems :
$ rrdtool fetch load.rrd AVERAGE -s 1464339000 -r 1800
  load_1min   load_5min  load_15min

1464339600: 2.33e-03 6.67e-04 0.00e+00
1464341400: 1.20e-02 6.005556e-03 3.33e-04
1464343200: -nan -nan -nan

$ rrdtool info load.rrd 
filename = "load.rrd"
rrd_version = "0003"
step = 300
last_update = 1464342015
header_size = 3632
ds[load_1min].index = 0
ds[load_1min].type = "GAUGE"
...
rra[0].cf = "AVERAGE"
rra[0].rows = 576
rra[0].cur_row = 227
rra[0].pdp_per_row = 1
...
rra[2].cf = "AVERAGE"
rra[2].rows = 672
rra[2].cur_row = 91
rra[2].pdp_per_row = 6
...

This fetches data from the AVERAGE cf for resolution 1800s (6x300), and only 
showing the last few values for brevity. Note that you can only fetch what is 
stored, you can't fetch arbitrary data consolidated on the fly like you can 
with rrdgraph.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] only getting nan values

[Simon Hobson]

Jim Sher  wrote:

> I'm using cron to run a script every 5 minutes to get some values and update 
> the RRD files and then draw the graphs.
> My script to populate the RRAs in the RRD:

> But when I view the graphs (all of them are like the one below), all the 
> values are not numbers (nan).

Two common problems :

1) How long did you let your collection script run for ? One update is *NOT* 
sufficient to create any known values in the database.

2) Post the commands used to create the RRD files. A common mistake is to make 
heartbeat only 300s and update "every 5 minutes". "Every 5 minutes" often isn't 
"every 300s", especially when using "N" as the timestamp. If two updates are 
301s apart, a heartbeat of 300s means "unknown" data will be stored.

Try "rrdtool fetch" to see what's stored in the database - it's easier than 
trying to work out if it's the data or the graphing that's causing the problem.

For reference, see http://rrdtool.vandenbogaerdt.nl
In particular, "Rates, normalizing and consolidating" should make things a lot 
clearer.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Adding data in the past

[Simon Hobson]

Olivier Nicole  wrote:

> I am stuck with aan annoying issue with rrdtool.
> 
> I want to add data in the past but cannot figure how to do so.

You can't That's a design feature, not a bug.

> So far, I have only added data now, but I want to reconstruct an history
> from data gathered from an older log file.

The *ONLY* way to do it is to start a new RRD file, with a start time before 
your first data point, and then feed the data in in increasing timestamp order. 
It gets tricky if some of the data you need to enter is actually in an RRD file 
and has already been consolidated - but that can (mostly) be worked around.


A bit more detail. RRD is not a regular database - it's a highly specialised 
one designed for one task (originally storing router traffic data). It doesn't 
have an arbitrary table that can have data inserted willy nilly - it has a 
circular buffer of predetermined size (set at database creation time) which 
only supports updated with time going forward (values actually stored depend on 
previous data).
In fact, other than (IIRC) the last value entered, not a single value you put 
in is stored ! All stored values are calculated on-the-fly and the data to 
update a previously entered value and recompute the stored value simply doesn't 
exist.

Alex has some great tutorials at http://rrdtool.vandenbogaerdt.nl
I suggest starting with "Rates, normalizing and consolidating" and "Describing 
the past, not predicting the future".

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] [Fwd: Suppress graph output on on empty data (e.g. min=0, max=0)]

[Simon Hobson]

Karsten Siegmund  wrote:

> As to 2), I found this post to the list from 2007
> https://lists.oetiker.ch/pipermail/rrd-users/2007-September/013179.

It would help if you didn't miss the html off the end ;-)

> I guess no further solution has been forthcoming for that problem since
> 2007?

Probably so

> Is the problem an unusual one that one has many graphs, but not all of
> them are of interest (e.g. contain data!=0) for a certain timespan?

I suspect it's not sufficiently widespread and/or the people needing the 
feature are already working around it. Eg, if you are building your own CGI 
applications then it's not really much extra work to handle it then.
Or put another way, it's a known problem, but it's easily worked around - so 
not really much incentive to build anything better.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] [FORGED] Re: Installing rrdtool on Linux machine

[Simon Hobson]

azeddine bezzir  wrote:

> The machine i am trying to install rrdtool on it, is not mine, it belong to 
> my company, so i cannot install another OS on it, or change any configuration.

> I want to manually install the tool

There's a conflict there - you aren't allowed to change anything, but you are 
allowed to install new software !


> - How to identify the dependencies needed for this installation?
> - Once identified, from where can i manually download them?

Your best bet is to use the OS repositories for the packages - making sure you 
get packages appropriate to the vintage of the system. Try to install the rpm 
and it will fail due to a missing dependency. Download the package (rpm file) 
that satisfies that, install it, and repeat. I'm not familiar with RH, but in 
Debian you can try to install multiple packages in one go with "dpkg --install 
package1.deb package2.deb ..." - which means you can keep trying without 
actually installing anything until you have a full list of required packages.


The downside is that you need to download multiple files one at a time, so it 
gets fairly tedious - which is why I suggested you start with the list of 
packages you got from yum when you first tried and install all of them in one 
go.

It's the "best" way to do it, but it's a pain to do - especially if you end up 
with a longer and longer list of missing dependencies at each iteration.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Installing rrdtool on Linux machine

[Simon Hobson]

I wrote:

> azeddine bezzir  wrote:
> 
>> In fact i downloaded the latest rrdtool package rrdtool-1.5.5, and tried to 
>> run the steps decribed in the below url:
>> 
>> http://www.cyberciti.biz/faq/howto-install-rrdtool-on-rhel-linux/
> 
> Did you look at the date on that ? JULY 28, 2008 which is over 7 years old. 
> You need to be aware that following random posts from many years ago is, in 
> general, likely to give problems as things will have changed in that time.

Just to add, since that post tells you to install a load of packages with "yum 
install ..." then that too will fail without an internet connection.

Unless already installed, then you are going to have to manually install all 
the development tools required to build the RRD package (and possibly build all 
the dependencies as well). That's a "significant undertaking" to say the least 
- and the fact that you tried to follow that post suggests that you probably 
don't have the skills to do that.

Is there any way you can connect this system to the internet (temporarily) to 
install packages ?
The alternative is to download the rpm for rrdtool on a system that is internet 
connected, transfer it to the system in question, try to install that and see 
what dependencies are needed - the install will fail with a message to the 
effect that "package foo is needed but isn't installed". Download those, 
transfer them, and repeat until you either give up with an ever expanding list 
of missing dependencies - or succeed. You could start by downloading the list 
of packages yum install said it needed.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Installing rrdtool on Linux machine

[Simon Hobson]

azeddine bezzir  wrote:

> In fact i downloaded the latest rrdtool package rrdtool-1.5.5, and tried to 
> run the steps decribed in the below url:
> 
> http://www.cyberciti.biz/faq/howto-install-rrdtool-on-rhel-linux/

Did you look at the date on that ? JULY 28, 2008 which is over 7 years old. You 
need to be aware that following random posts from many years ago is, in 
general, likely to give problems as things will have changed in that time.

> I need to identify the dependecies needed, and download them manually as the 
> machine is not directly connected to the internet.

OK, that changes things - and is one of those bits of useful information that 
should have been in your original message.

> I run the command you proposed 'yum install rrdtool' and got the below output:

Without an internet connection then it won't work - for one thing, it won't be 
able to get an up to date list of packages.

I'm not familiar enough either with manual install or RH to help further - 
hopefully one of the other regulars will be along soon.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Installing rrdtool on Linux machine

[Simon Hobson]

azeddine bezzir  wrote:

> I want to install rrdtool on a Linux machine, in order to store some 
> statistics files and generate graphs accordingly.
> 
> In fact, it's the first time i am using this tool, and i am unbale to install 
> it, could you support please?
> 
> The OS version is:
> 
> Red Hat Enterprise Linux Server release 6.4 (Santiago)

Your email is really really lacking in detail - how are you trying to install 
it ? What is going wrong ? What is happening ? What isn't happening ?
See http://www.catb.org/~esr/faqs/smart-questions.html

But in answer to the question I think you are asking ...
I would suggest you simply use the packaged version, I believe on RH systems 
that would be just "yum install rrdtool". Doing it that way means you'll get a 
version built for your system, and all the dependencies automatically installed.
If you want to install from source, then you'll need to install dependencies 
yourself, and you'll need to understand how to configure it for your system 
(should be a matter of "make configure", but not all packages are that simple).

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Help with: Input from a file in the RRD Graph

[Simon Hobson]

Mats Gustavsson  wrote:

> I have a Perl script that creates a .png file with the graph. So far so good.
> In the Graph I have the function "HRULE:22#00" that works just fine.
> 
> Now I want to make the value 22 flexible. I want it be be read from a 
> external file.
> So when I just change the value to 21, or 23 or what ever, its been read from 
> a file.
> 
> Because the value in the file ex 22 is being read from multiple scripts, both 
> a bash and a python script, I want it to pick the value from the file to the 
> HRULE in the RRDs::Graph, in the perl Script.
> 
> The file, that is today being read from bash and python, is just containing a 
> value between -30 to +50. So the help that I need with, it needs to handle a 
> value ex -5 and 22.
> 
> The reason I want it to read from the same file, is that I just needs to 
> adjust in two files.
> File 1 High value like -5
> File 2 Low value like 22.
> Instead in 20-24 different places, that is easily to miss a place to change.
> 
> I hope that some one can help me.

TBH I'm not sure what the question is ! It seems that you've already figured 
out that the way to do this is to put the values into files, and read those 
files in to set a variable that you can then use in building the RRD command. 
In bash I'd be thinking along the lines of :

MaxLineValFile="/some/path/to/file"
read $MaxlineVal < ${MaxLineValFile}
# Apply some sanity checking here !
rrdtool ... ${MaxLineVal} ...

I actually use rrdcgi in my scripts (cgi scripts served up by Apache), like 
this :

(
  echo some stuff
  echo more stuff
  some conditional stuff
  ...
) | /usr/bin/rrdcgi --filter - 2>&1 > /dev/null

This is because some of my RRD stuff is quite large - biggest is 254 values 
stacked, plus another 254 valued inverted and stacked, plus a legend for min 
and average for each. I did try building up the list of RRD commands in a text 
variable and then calling "rrdtool graph ${command_list}" but it got really 
slow building that up.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] RRDTOOL Update with script in string

[Simon Hobson]

Tony Mountifield  wrote:

>> # Get the file
>> Utmp=$( < /home/pi/net/devices.csv )
>> 
>> # Get everything after the 12th ','
>> Utmp=$(Utmp#*,*,*,*,*,*,*,*,*,*,*,*,)
> 
> Always dangerous to type suggestions on the fly without testing!
> (been there, done that)
> 
> I think you mean ${ ... } not $( ... ), and similarly on the next two.

Just testing to see if anyone is paying attention ;-)

Yes, some of those () should be {}. The first line is correct using the $(...) 
to run a command in a sub-shell. After that it should be ${...} for value 
substitution.

Every code example should have at least one error - it teaches more is someone 
has to work out what's wrong and fix it :D


As an aside, it's said that every Meccano instruction leaflet had at least one 
error for that reason. Ie, the kit could not be build by following the 
instructions to the letter (or image as they are pictorial), so the user has to 
apply some thinking for themselves.
Wikipedia says that those involves say they were accidental errors.
https://en.wikipedia.org/wiki/Meccano#Capabilities

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] RRDTOOL Update with script in string

[Simon Hobson]

maxmax20  wrote:

> rrdtool update /home/pi/net/net.rrd N:'cut -d, /home/pi/net/devices.csv -f13
> | cut -d: -f2': to get the first value out of the csv file, but it seems
> that it won't work.

As suggested, put echo in front of it - and possibly pipe the output through 
hexdump.

One thought, does the SCV file contain exactly one line - not zero lines, not 2 
lines (having one line and a newline counts as 2 lines). You may want to add 
"head -1"

If working in bash, you can cut things down somewhat by using Bash builtin 
string functions - one some of my data collection scripts, it's made a huge 
difference. When you use cut you are spawning two new processes (three is you 
add head as well).

Check the man page for parameter expansion, specifically the ${parameter#word}, 
${parameter##word}, ${parameter%word},  and ${parameter%%word} forms. I think 
this will get you your value :

# Get the file
Utmp=$( < /home/pi/net/devices.csv )

# Get everything after the 12th ','
Utmp=$(Utmp#*,*,*,*,*,*,*,*,*,*,*,*,)

# Get everything after the first ':'
Utmp=$(Utmp#*:)

# Optional - depends on your file, delete everything after the next , or :
Utmp=$(Utmp%%,*) or Utmp=$(Utmp%%:*)


Don't think it applies here (though I don't know if fiddling with IFS might 
help), but another trick you could try is using arrays. My specific action was 
getting traffic counts from an interface, so I have a script with :
  Utmp=$(< /proc/net/dev)
  Utmp=(${Utmp#*ethext:})
  UpdateVal=${Utmp[0]}:${Utmp[8]}
I'll leave you to figure it out.


PS - no I didn't come up with this myself, someone else pointed me in that 
direction.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] RRA of unaltered data

[Simon Hobson]

Thomas Nilsson  wrote:

> 1. Is there a way to get the values I add to show up unaltered in the RRA?
>Tried with LAST, AVERAGE, MIN and MAX in the RRA declaration.
>   
> 2. How are the values in my RRA calculated?

These together must be one of the most frequently asked questions, to which 
there is a simple answer :
http://rrdtool.vandenbogaerdt.nl

In particular, "Rates, normalizing and consolidating" 
http://rrdtool.vandenbogaerdt.nl/process.php

The latter will answer question 2. Once you understand that, then you'll 
understand the answer to question 1 which is: ensure that the normalisation 
process is a "null operation". That means you must supply data on *exactly* 
step boundaries, for every step, only once per step, and missing no steps out.

So in your example, you must ensure that the timestamp of every update is on 
*exactly* a multiple of 5 seconds since unix epoch - not "now", not "a 5 second 
sleep plus the execution time of a few statements after the previous one", but 
on exactly a multiple of 5 seconds.
Thus (in bash) you can do :
s=5
t=date +%s
t=$( ( $t / $s ) * $s )
rrdtool update ... $t:$v

This will ensure that the values stored will be exactly what you put in - but 
only until there is some consolidation.

But in general, if you want to store exact values, and especially if they are 
produced at irregular times, then RRD is probably not the right tool.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] graph question

[Simon Hobson]

Steve Shipway  wrote:

> OK, so your RRD file is correctly configured to allow from -20 to +50 as 
> valid values for the internal/external DS.

Well something doesn't tie up then, the original question shows -55 to +125 as 
the allowed range.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Graph NaN Values

[Simon Hobson]

"Manzke, Udo"  wrote:

> Result is an Error: “RPN Stack Underflow”. Do anyone know what’s wrong?


DEF:rtMills=/icmp.rrd:icmp:AVERAGE \
CDEF:rt=rtMills,1000,/ \
CDEF:unknownpos=rt,UN,INF,UNKN,IF,/ \

OK, lets break that down.

rt -> puts rt on the stack

UN -> take rt off the stack and leaves true or false depending on whether it's 
unknown or not

INF, UNKN -> puts INF and UNKN values on the stack. Stack now holds 
[true|false],INF,UNKN

IF -> takes 3 items off the stack, sees the true or false left by the UN 
operator, and selects either INF or UNKN which it puts on the stack. Stack now 
holds [INF|UNKN] - if rt was unknown then the stack holds INF, else it holds 
UNKN. I assume you actually wanted rt here so your line needs to have 
"INF,rt,IF"

/ -> takes two items off the stack. Oh dear, there was only one item there -> 
stack underflow. So either remove the "/" or add something to divide by. 
Perhaps you intended to combine the two lines, in which case you'd have had 
"rtMills,UN,INF,rtMills,IF,1000,/" or "rtMills,UN,INF,rtMills,1000,/,IF" - the 
two are equivalent as long as INF divided by 1000 is still INF (which I'd 
expect it to be).


I don't think INF is likely to be what you want, (according to the docs) it 
draws a line along the top of the graph which (depending on the data) may or 
may not be readily distinguishable from steady data - it wouldn't be how I'd 
want to draw it anyway. Personally I'd just accept the gaps in the graph which 
(IMO) is more meaningful - for ping, no response is indistinguishable from 
"response outside the timeout window".



___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] graph question

[Simon Hobson]

s pascal  wrote:

> But when i create the graph, the temperature curve shows only positive 
> temperature and not negative temperature.
> 
> When the temperature is going under 0 C, the curve is blank then become 
> visible when temperature is positive.

I can't see anything wrong with either command.
What data are you putting in ?
What data is stored - use rrdtool fetch to dump it ?

It might be worth creating a test file and putting known test data into it, 
then see what comes out.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] reduce graph generation times

[Simon Hobson]

Alex van den Bogaerdt  wrote: some very good advice.

The only thing I can add is to, if you have the storage, the --lazy option. 
That way, while each graph will take the same time to create, it only needs to 
be created once.
So if (for example) you have a page with several graphs of the same thing over 
different timescales (eg day, week, month, year) then you really only need to 
create the day graph the first time the page is shown in any day - and after 
that it won't change again until tomorrow.
Using --lazy means it'll keep that first image on disk and just serve it up 
rather than re-render it.

It really depends on your workload, but can significantly reduce the number of 
graphs you need to draw.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Incorrect MAX total from consolidated cacti graph

[Simon Hobson]

Steve Shipway  wrote:

> Unfortunately, your RRD file and the commandline are being generated
> internally by Cacti, ...

You can get Cacti to show you the command it used to draw a graph - useful for 
debugging.

Also be aware that I've had "issues" getting Cacti to use the correct def. IIRC 
(it was a while ago now) I ended up drawing an invisible line for the max value 
so I could get it to use the right set of data to generate the subsequent max 
value for a gprint.



___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Incorrect MAX total from consolidated cacti graph

[Simon Hobson]

jbhowe  wrote:

> I have a cacti graph which is a consolidation of 34 graphs of online user
> counts. A primary purpose of this graph is to identify peaks in user counts.
> My problem is that the SUM of the MAX values is incorrect unless I constrain
> the timeline of the graph to the minute containing the peak.

You can't add MAX values once data is consolidated - it will give you incorrect 
results. Eg, take the following sets of data :

100,0,0,0
0,100,0,0
0,0,100,0
0,0,0,100

If you add up the unconsolidated data, you'll get the correct answer that the 
max values are 100,100,100,100.

But if you consolidate them, you'll get the following max values :
100
100
100
100

and if you then add them up, you get 400.

OK, that's an extreme example, but it shows the problem.

The answer is that you need to do the adding up BEFORE consolidating the data. 
I wrote a script to do just this (for generating combined mail queue stats 
across several servers). It's harder than it looks at first glance, but 
basically I iterate round a loop, find the last update time of the combined 
stats file, and if all the source files have an update time which is later then 
I pull down the next set of numbers, add them up, and update the combined file 
- else I sleep for a while to wait for new data. That bit is fairly easy - but 
you also need to consider if one (or more) server is down and not updating it's 
stats - then you need to decide how long you'll wait (in case it's lost 
connectivity but is still collecting stats) before you ignore it's values and 
just combine what you do have. And if you get behind for whatever reason (eg 
the server doing the consolidation is down for maintenance), then you also need 
to restrict how far back you go so you can always work with unconsolida
 ted data.

If you have a variable number of sources then it's a bit more fiddly still.

Summary: Create a "total users" data file, have a script that fetches new data 
as it's available (before consolidation) and totals it into this summary file. 
Then consolidate the data in that summary file.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] rrdtool memory usage and the Raspberry Pi

[Simon Hobson]

Rafal Gwizdala  wrote:

> Maybe the file size growth is expected to be parabolic - i think every RRA is 
> 'global' - it covers all the data sources. So when you're adding DS and RRA 
> pair you're actually adding N RRAs where N is number of data sources. At 
> least this is how i understood the concept of RRAs.

Yes, that is correct - each RRA applied to every DS in the file. File size will 
scale (roughly) as D*R where D is number of data sources, and R is number of 
RRAs.



> [08:32:57 pe@hope] $ rrdtool create test.rrd \
> > --step 1s \
> > DS:progreset:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> > DS:tpm:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m \
> > DS:voltcontrol:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m \

Duplicate RRA

> > DS:freqcontrol:GAUGE:1000s:0:1 RRA:LAST:0.1:5s:5000m \

Another duplicate

> > DS:fault:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> > DS:uff:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> > DS:off:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> > DS:uvf:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \
> > DS:ovf:GAUGE:1000s:0:1 RRA:MAX:0.1:5s:5000m \

5 more duplicates

> > DS:svopen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:svclose:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:bvopen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:bvclose:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:syncen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:busconnen:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:wmfull:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:wmmed:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:wmlow:GAUGE:1000s:0:1 RRA:AVERAGE:0.1:5s:5000m \
> > DS:vred:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:exciter:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:ired:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:iwhite:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:iblue:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:f:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:isp:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:dlred:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:dlwhite:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:dlblue:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m \
> > DS:svpos:GAUGE:1000s:0:32000 RRA:AVERAGE:0.1:5s:5000m

Another 19 duplicates


As a matter of style, I always put my RRAs at the end - so I have some "header" 
stuff, then a list of DSs, then a list of RRAs. Makes it much easier to read.
As an aside, does it matter what order they are declared in ? Ie, if an RRA is 
declared before all the DSs, does it still apply to all DSs ? I would hope so.



___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Local maximum and minimum

[Simon Hobson]

Philip Gwyn  wrote:

> Hello,
> 
> I have a 1-wire setup in my attic.  I log temperature and humidity and 
> generate daily/weekly/monthly graphs.  I use a modified temps.pl from Thierry 
> Hugue.
> 
> What I would very much like is to graph the local maximum and minimum 
> temperature.  This would be roughly the maximum and minimum over a 24 hour 
> period.  The MAXIMUM and MINIMUM operators don't do what I want - they find 
> the max for the entire period.

If you have a 24 hours consolidation for MAX and another for MIN, then that 
will contain values for max and min per day*. You can then graph that and get a 
graph of min/max by day. You may have to specify the consolidation when 
graphing if you have other min & max consolidations.

* Where "day" is a 24 hour period starting at midnight UTC. It will not be 
midnight to midnight if your local timezone isn't UTC.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Temperature diff colors

[Simon Hobson]

Martin  wrote:

> I´m wondering if anyone has a code example for showing diffrent colors in a 
> graph depending on temperatue like this, if it above +0 and up the color is 
> red and if its below -0 its blue ?

Here's one I did that does "light red" when things are close to the limit, and 
"full on red" when above the limit (it's part of a Bash script) :

>   echo "${RRDHead}
> 
> DEF:op_pc_ave=${RRDPath}/${RRDName}:op_pc:AVERAGE
> DEF:op_pc_min=${RRDPath}/${RRDName}:op_pc:MIN
> DEF:op_pc_max=${RRDPath}/${RRDName}:op_pc:MAX
> CDEF:op_pc_spread=op_pc_max,op_pc_min,-
> VDEF:v_op_pc_ave=op_pc_ave,AVERAGE
> VDEF:v_op_pc_max=op_pc_max,MAXIMUM
> VDEF:v_op_pc_min=op_pc_min,MINIMUM
> VDEF:v_op_pc_lst=op_pc_ave,LAST
> 
> CDEF:alarm1=op_pc_max,${Plimit},100,LIMIT
> AREA:alarm1#FFBFBF:
> CDEF:alarm2=op_pc_max,99,1,LIMIT
> AREA:alarm2#FF:
> 
> COMMENT:\"  min  ave  max  last\n\"
> HRULE:${Plimit}#7F
> AREA:op_pc_min#FF00:
> AREA:op_pc_spread#BFBFFF::STACK
> LINE1:op_pc_ave#FF:\"%age Power\"
> GPRINT:v_op_pc_min:\"%6.0lf \"
> GPRINT:v_op_pc_ave:\"%6.0lf \"
> GPRINT:v_op_pc_max:\"%6.0lf \"
> GPRINT:v_op_pc_lst:\"%6.0lf\"
> 
> 
> COMMENT:\"  Data to ${PrintedTime}\n\"
> >

For yours' what I'd suggest is two CDEFs, one which is X when X < 0 and UNKN 
when X >= 0, the other which is X when X is >= 0 and UNKN when X < 0. Then plot 
the first CDEF in blue, and the second one in red.

So something like :
CDEF:xb=x,unkn,x,0,lt,if
CDEF:xr=x,unkn,x,0,ge,if
should (if I've got my RPN right from memory) give you the two series to draw.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Stacking

[Simon Hobson]

Devante Vargas devan...@gmail.com wrote:

 I am working on creating a graph for my ping script. It collects, min, max 
 and average response times. I have added the ::STACK for the AREA after the 
 first AREA is defined. Problem is it adds the next two on top and moves it up 
 a notch, e.g. Min response time is 0.42 (this is measured accurately and on 
 the Y axis it shows it hits 0.42ms) but the next 2, e.g max response times, 
 0.64ms, it places it on top of the min area and when aligned with the Y axis 
 it shows at 1.08ms, and then average, 0.60ms on top of that and shows aligned 
 at the value of 1.68ms.
 
 Is there anyway to get the max and average to start at zero instead of from 
 where the last area ended up at?

Several ways to do this, depending on what you want.

One would be to plot an area for max, then an area for avg, then an area in 
white for min - no stack involved. So you end up with a colour band from min to 
avg, and a different colour band from avg to max.

Another one I've used is : use a CDEF to get the difference between max and 
min, plot min in white (or transparent), then stack the CDEF on top of that, 
then draw a line for avg. I tend to use a solid colour for the line, and the 
same colour (but with a lot of transparency) for the min-max spread - that way 
the min-max spread area doesn't obliterate what's been plotted before.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] updating rrd file with ping min, max and avg times in milliseconds

[Simon Hobson]

Devante Vargas devan...@gmail.com wrote:

 I created the rrd file using the following parameters:
 
 rrdtool.exe create latency.rrd 
 --start now 
 DS:response:GAUGE:600:U:U 
 RRA:MIN:0.5:12:8760 
 RRA:MAX:0.5:12:8760 
 RRA:AVERAGE:0.5:12:8760
 
 I am trying to update it but having problems getting the following error:
 
 ERROR: latency.rrd: found extra data on update argument: 0.40:0.88:0.73

That's because you are offering 3 values to an RRD that's configured to take 
only one.

 I am typing the following to update the rrd file:
 
 rrdtool update latency.rrd 08/10/2015_11:33:0.40:0.88:0.73
 
 0.40 = Minimum
 0.88 = Maximum
 0.73 =  Average

It doesn't work like that. If you want to record the min/avg/max values from a 
ping then you need to define THREE DSs, eg response_min, response_avg, 
response_max
You can then define your RRAs as required. You cannot be selective about this, 
so you will end up with all 9 permutations of min of min, min of avg, min of 
max, avg of min, ... Only three of these (min of min, avg of avg, max of max) 
are actually useful.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] scheduling RRD graph upates on a windows 2008 R2 platform

[Simon Hobson]

Brent Barr b.b...@f5.com wrote:

 I've gone to a system that uses a python file to generate the graphs on the 
 fly.  Request the HTML page and it takes 5+ seconds to go get the data and 
 generate the charts for that page.  Means I only generate charts a few times 
 a week, and that the data is always the freshest.

That's how I do it (though I'm on GNU/Linux).
All the web pages (or more accurately, the images) are CGI scripts that 
generate an RRD script on the fly. Most of them take parameters, for example 
setting the period to day, week, etc or selecting an IP. I have the --lazy 
option set, so RRD will only actually generate a new graph if the data is newer 
than the cached image file.
With the number of potential images, it would be a huge waste of resources (not 
to mention needing a much much more powerful system) generating the thousands 
of potential graphs - most of which will be rarely (if ever) be looked at.

An example :
We have a /24 subnet at work, and I log the traffic by IP address at the border 
gateway. Excluding the network and broadcast addresses, that's 254 IPs, with in 
and out for each. I can draw a graph for any IP for a day, week, month, or year 
- and with or without a line to show max (significantly different from average 
once you start aggregating data). 254x4x2 = 2032 graphs, and that's only part 
of the logging I do. Most of them will not be looked at other than once in a 
while so it would be a complete waste of resources to regenerate them 
periodically.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] graph is empty, not sure what I am doing wrong

[Simon Hobson]

Devante Vargas devan...@gmail.com wrote:

 I then connect to XenApp and launch a session and runt he update command 
 using the following:
 
 c:\Perl\site\lib\auto\RRDs\rrdtool.exe update 
 f:\PSscripts\Common\Get-FarmData\DASHBOARD\DEV_Accounts\SessionState.rrd 
 N:2:1:1:0:0:0
 
 Once its updated

How many times/over what period did you run the updates ? You have a step size 
of 300s (5 minutes), and your most granular consolidated data for MIN and MAX 
is 12 times that (ie one hour). You'll need to run updates less than 600s (10 
minutes) apart until both of the following are true :
- You go past the hour, since your consolidated data points will all start 
end on the hour (ie 1am, 2am, and so on)
- You have more than 50% of the preceding primary data points filled.
This means you must be regularly updating for *AT LEAST* half an hour, and up 
to almost 1 1/2 hours depending on when you start, before you'll see data.

Use rrdtool fetch ... to see what data is available.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] How to find a grafics error?

[Simon Hobson]

Peter A. Menzel ad...@famenzel.at wrote:

 I actually found some missing update values - that means not every second a 
 new set of values.
 
 Now I intent to tune the heartbeat value to 5 instead of 2,
 and step size of 1 remain constant...
 So missing 1, 2 or 3 measures will not generate an *unknown* value in the RRD.
 
 Do you think that is the right way?

It's probably the right thing to do and my guess is that it will cure your 
gaps - but really it's your data, you have to decide whether it's right for 
your needs. Some people want to see the unknowns, others want to ride over 
them.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] How to find a grafics error?

[Simon Hobson]

Peter A. Menzel prot_from_s...@famenzel.at wrote:

 Here is the setup script:
 
 rrdtool create AgroFire.rrd --step 1 \
  DS:KesselTemp:GAUGE:2:U:U \
  ...
  RRA:AVERAGE:0.5:1:864000 \

Can you be certain that you provide an update *every* second ?
You have a step size of 1, and a heartbeat value of 2. That means that you can 
only have data 2 seconds old before it is considered unknown. What isn't clear 
is which side of the point the decision is taken as the manual says :
 heartbeat defines the maximum number of seconds that may pass between two 
 updates of this data source before the value of the data source is assumed to 
 be *UNKNOWN*.

Eg, if you miss an update at time t=1 having done one at time t=0, does the 
value become unknown at time t=2 or *after* time t=2. The difference is that if 
you miss just one update, then in the former condition, when you come to do the 
update at time t=2, the data from time t=0 to time t=2 will be marked as 
unknown; but if it's the latter then the data wouldn't become unknown unless 
you also miss an update at time t=2.

So firstly I would be checking to see that you are reliably doing an update 
every second and not creating a few unknown values. You might consider 
increasing the heartbeat value.

You have specified an xff value of 0.5, so you would regard a consolidation as 
unknown if 50% of the source values are unknown. Since missing one update could 
be marking 2 seconds worth (ie 2 primary steps) as unknown, then missing an 
average of one update every 4 seconds would result in missing consolidated data.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] How to find a grafics error?

[Simon Hobson]

Peter A. Menzel prot_from_s...@famenzel.at wrote:

 my cgi output for the monthly periode seems to have errors (in detail: see 
 attached image).
 As you can see on http://hk1.famenzel.at/cgi/hk1.cgi the output for the last 
 week (letzte Woche) is ok (continued graphs for the temperatures), but 
 shown interrupted in the monthly graph (letzter Monat).
 Also the pump indicators (HK1Pumpe, HK3Pumpe aso) show some activity in 
 this graph and the yearly one (letztes Jahr). This seems to be wrong too.
 
 How can I find the source of the problem?

You'll need to provide a lot more information - at the moment it's I create a 
graph using an unknown script/command, using unknown data, and 'it doesn't 
work'.

A first step would be to show what commands you used to create the RRD files 
and the graphs, plus a sample of the data going in.

Your graph in the email (the link doesn't work for me) has breaks in it - that 
can often be as simple as having a heartbeat interval that's too short so you 
get a lot of unknown data.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] OpenWrt + Collectd + RRDTool

[Simon Hobson]

Aizad Fauzi red...@gmail.com wrote:

 I've installed these three components and my graph is working perfectly.
 I create a symlink of the PNG to the root of my www folder so that I can 
 easily view the graph directly.
 However, in order for the graph to be updated, I need to log-on to the router 
 and view from there in order to refresh the graph.
 
 Are there any tricks that can make the graph automatically update by running 
 some command to setup as crontab?

How are you generating the graph ? How are you viewing them when you log in to 
the router ?

There are two ways to do it - create graphs periodically in case you want to 
view them, or create them on demand.

The first of these means using some tool (eg cron) to run commands to generate 
the graphs periodically. The downside of this is that you expend resources 
generating graphs that you will most likely never look at.

The alternative is to trigger graph creation when you want to view it - 
typically by CGI scripts on the server.

I suspect that when you log onto the router (presumably via it's web UI) and 
view the graph, that's triggering a CGI script that generates the graph there 
and then - if you right click on the graph image and open in new window/tab, 
what URL do you get ?. This normally also creates an image file which hangs 
around until it gets over-written by the next version. If you statically link 
or view this image file, then it won't update on it's own.
The fix for this is probably to link to the CGI script rather than the static 
image file - that way, when you view it you'll always get an up to date version.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Graphing inconsistencies graphing different time slices

[Simon Hobson]

Munroe Sollog m...@lehigh.edu wrote:

 I have an inconsistency when I graph the same event over 10days versus 48hrs. 
  I would expect
 because I am using MAX that the peak on the 48hr graph and the 10day graph 
 for the *same* event
 would still read the same number.  However, the peak for Sunday noon on the 
 48hr graph is at 600,
 and the peak for that same period on the 10day graph is *higher* at 1.1k.  I 
 believe I have
 included all the necessary files below.  I didn't want to paste the raw info 
 here as it would be
 very large.  Any insight would be appreciated.

You are adding up different max values. Consider this set of data :
a1 = 1 5 3 7 4
b1 = 2 6 9 1 4

If you do a graph of max (max a + max b) without any consolidation then you get 
the values :
m1 = 2 11 12 8 8

If you consolidate 5 values into one, you then get :
a2 ave = 4
a2 max = 7
b2 ave = 4.4
b2 max = 9

m2 (= max (max a2 + max b2)) = 16

So when you draw a graph over a longer time period, depending on the 
consolidation, you may see your maximum value go up (with these made up 
figures, by 1/3) when you add max values together.



___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Graphing inconsistencies graphing different time slices

[Simon Hobson]

I wrote:

 So when you draw a graph over a longer time period, depending on the 
 consolidation, you may see your maximum value go up (with these made up 
 figures, by 1/3) when you add max values together.


To add ...

I had something similar for some of my servers - I wanted a maximum mail queue 
size across several servers. Simply adding the max queue sizes doesn't work for 
the reasons I gave before, eg if 3 servers all have a queue size of 1 but at 
different times, the maximum cumulative queue size is 1, not 3.
I wrote a script* that would periodically query the stats for each server, get 
the latest figures, add them, and put them into another RRD file for the 
totals. So as well as RRD files for each server, I have one for the combined 
values. As long as the script keeps the combined file up to date enough, it can 
use unconsolidated values from the individual servers.

This combined RRD can then be consolidated and return the correct max values.

* Which turns out to be more complicated than you might think !
It needs to find out the last complete bucket of data for each of the source 
files and see if the oldest of these is newer than that for the combined file - 
and if so, then update the combined file. You need to allow for delayed data 
from one or more servers (I'm using RRD cached which can delay updates unless 
you keep flushing it which defeats the object of the cacheing).
But then, you need to allow for one (or more) servers being offline (as far as 
stats collection goes), so you need to decide how long to wait before taking an 
individual value as NaN.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Graphing inconsistencies graphing different time slices

[Simon Hobson]

Munroe Sollog m...@lehigh.edu wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 
 Very interesting.  Thank you for clarifying it.
 
 My problem is, as you can see from the graphing function, I am arbitrarily 
 taking numerous rrds
 and adding them together to get a cumulative graph.  While some of these 
 summation graphs are
 predictable, many are not; thus I can't store a cumulative rrd in all 
 situations.

 My goal is to minimize this disparity that is caused by taking the MAX of a 
 consolidated function.
 I am going to see if using the AVG instead of the MAX of the consolidated 
 function will help, but
 if I really want to fix this, my only recourse is to not consolidate? I'd be 
 storing a 1years
 worth of 1min slices for ~5000 devices.

That would be a lot of data to store, and process each time you draw a graph. 
Quick finger in the air guestimate suggests a few gigs at least. It also 
needs a lot of memory to make graphs.

AFAIK there is no way to consolidate a consolidation and still get an accurate 
MAX function, so you either have to store and work with unconsolidated data, or 
aggregate the data before consolidation.

It sounds like you may need to sit down and decide what's important and whether 
there are any simplifications you can make. Eg, if the devices naturally fall 
into groups, then consider if aggregating data by group would do what you want 
- even if the groups are dynamic in membership it's still possible to aggregate 
the data at the cost of more complexity in the scripts that do it.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Getting an overview of many statistics...

[Simon Hobson]

Peter Valdemar Mørch pe...@morch.com wrote:

 Looking at average and standard deviation is a possibility, but most of my 
 users (and I) really have no good intuitive feeling for what standard 
 deviation really means.

+1, I don't either

 I've seen that smokeping does something interesting, see e.g.
 
 http://oss.oetiker.ch/smokeping-demo/img/Customers/OP/james~octopus_last_10800.png
 
 The historgram approach where darker grey implies more datapoints in this 
 region could be cool. This gives the overview. Have no idea how this is 
 accomplished, though.

I'm not sure dark = more in the way you are expecting. I suspect it's more a 
case of shading ranges - so the central range (say the range that contains from 
40% to 60% of the results when sorted by time) is drawn dark, the ranges either 
side of that are drawn lighter, and so on until the outmost ranges (eg from 
smallest to say 10% and 90% to largest) are drawn in light grey. Finally the 
median is drawn as a line - who's colour indicates packet loss.

The areas can be drawn three ways. Lets assume we have 11 values, representing 
the ping times for the fastest (t0), the 10th decile (t1), through to the 
slowest (t10).

We can draw t0 to t10 in very light grey, then overlay t1 to t9 in less light 
grey, t2 - t8, t3 - t7, and finally overlay t4 to t6 in dark grey/black.
Or we can draw t0 to t1, stack t1 to t2, stack t2 to t3, t4 - t4, t4- t6, 
t6-t7, t7-t8, t8-t9, and finally t9-t10.
Or we can draw 0-t10 in light grey, then draw 0-t9 in darker grey, and so on 
until you've drawn 0-t1 in light grey. Then draw 0-t0 in white to erase the 
bit between axis and lowest value.

Neither is right or wrong - personally I'd do it the first way, which would be 
(from memory) something like :
DEF:t010=t10,t0,-
DEF:t19=t9,t1,-
...
AREA:t0#FF00  - *
AREA:t010#202020:STACK
AREA:t1#FF00  - *
AREA:t19#404040:STACK
...
* Note that I've used full transparency to draw nothing from the axis up to 
the bottom of each range.


Then you need to draw the line, and again you need to generate bands and then 
draw several overlays. Again there is more than one way :
You can draw the line, in each colour, only where that colour is needed; or you 
can draw the line in each colour, overlaying each colour on top of the previous 
one.
Eg, you could draw a red line all the way, then draw the light blue line only 
where packet loss 19/20, and so on until you draw the green line only where 
loss=0. Or you can draw the red line only where loss =19/20, the light blue 
line only where loss =10/20 and 19/20, and so on. The line itself is drawn at 
the median value (t5 - bet you were wondering where that had gone !)
Something like this :
LINE:t5#FF
DEF:l10=loss,19,lt,t5,unkn,if
LINE:t5#FF
...
Which means, draw t5 in red, then calculate l10 which equals t5 where loss 19 
otherwise it's set to unknown, then draw that in blue. Where L10 is unknown, 
then the line is not drawn and the red line shows through.
Repeat for the other steps.

So it's not actually all that hard to draw.



Alex van den Bogaerdt a...@vandenbogaerdt.nl wrote:

 I would generate a number of squares, showing either green, amber or red.

Just be aware that those colours are the ones that are affected by the most 
common form of colour blindness (red-green deficiency), which affects about 1 
in 7 males ! Many red/amber/green graphs are almost invisible to me (it 
depends on the area, the specific colours used, the display, ambient light, 
etc) and with some of them I really cannot see a change between the colour 
sections without blowing the screen up to make the areas larger.
A good example is the Unify software from Ubiquiti, where on one page, it shows 
a health bar for each access point showing red where there is a lot of 
competing traffic and packet loss and green for good. The bars are thin, and 
it was a while before I even realised that there was a red section on some of 
them !

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Concatenate 2 RRD without overlap or sum

[Simon Hobson]

topgunx1 topgu...@hotmail.com wrote:

 Sorry but I'm not so expert in rrd for implementing what you have suggested
 me.

Then treat it as an opportunity to learn !

 Is it possible for you correct my script (I know you are very busy!)?

No, but I'll get you started. See 
http://oss.oetiker.ch/rrdtool/doc/rrdgraph_rpn.en.html

Assuming you have a DEF of a which is the first value you want up until (and 
including) time t, then this should give you a value up until time t and 
unknown after that :
CDEF:aa=a,UNKN,TIME,t,LE,IF

Breaking that down:
TIME,t,LE means :
Push the time the currently processed value was taken at onto the stack, push 
the value t onto the stack, if the first was less than or equal to the second 
then push true onto the stack, else push false.
a,UNKN,something,IF means :
push a onto the stack, then push unknown onto the stack, then push something 
(the result of the If statement) onto the stack. If that something is true 
then take the a, else take the unknown value.

You need to replace t with the numerical value. I tend to use Bash and 
dynamically generate the RRD scripts, so the above would become 
CDEF:aa=a,UNKN,TIME,${t},LE,IF where t is previously defined/calculated in 
the script. If you are writing a static script, then use the numerical value.

Further hints.
Don't try to write the whole script and then debug it. Just do one DEF and CDEF 
and graph that (you should have an area that stops abruptly at time t). When 
you have that working then you can start adding the other parts.
Don't forget that you need to use LE and GT or LT and GE depending on 
whether you want time t to be the older data set or the newer one. If you use 
LT and GT then there's be a very small gap, if you use LE and GE then both sets 
will be included at time t and there's be a blip in the data.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Concatenate 2 RRD without overlap or sum

[Simon Hobson]

On 20 Mar 2015, at 15:45, topgunx1 topgu...@hotmail.com wrote:

 I should draw a graphic with the values (IN/OUT) of 2 NICs.
 I wouldn't like that graphs overlap or sum 
 I need to placed RX1 above the RX2 (starting from Y-0),
 and TX1 above TX2 (newly starting from Y-0).

You mean, at some point T you switched things round, and you want the graph to 
switch around as well ?

Without going and looking stuff up (really busy right now), it's similar to the 
I started a new RRD at time T and want to use the old one up to that point.

I used something like :
CDEF:aa=a,unkn,timestamp,t,lt,if
CDEF:bb=b,unkn,timestamp,t,lt,if
CDEF:cc=b,unkn,timestamp,t,ge,if
CDEF:dd=a,unkn,timestamp,t,ge,if

Then plot aa, bb:stack, cc, dd:stack
For where I've put timestamp, you'll need to check the docs, there's a 
function which returns the timestamp of the value being processed.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Found a old .rrd

[Simon Hobson]

Martin mar...@swetech.se wrote:

 So i found a old .rrd db  is there a way to find out the settings used to 
 create it, so i can graph it again ?

rrdtool info file.rrd
All the information should be in there.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Min Max Values

[Simon Hobson]

jr1 jr4...@hotmail.de wrote:

 I use 
  DS:i1:GAUGE:60:-10:100 \
 RRA:AVERAGE:0.5:1:86400

 I need 3 Number for each DS avverage, min, max   for a periode of 60 sec.
 
 How to create the Database?

 DS:i1:GAUGE:60:-10:100 \
RRA:AVERAGE:0.5:1:86400
RRA:MINIMUM:0.5:1:86400
RRA:MAXIMUM:0.5:1:86400

(or could be MIN and MAX - check the docs)

When accessing the data (eg to draw a graph) you need to specify which RRA to 
use.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Drawing daily Min/Max graph from a source with UN values

[Simon Hobson]

Alexander Topolanek at...@ocv.org wrote:

 I'm fetching data from my radio weather station and put them into graphs. A 
 graph that shows the temperature over the day, and follow up graphs that show 
 min, max and average temps from the previous days.
 
 Unfortunately some times the transmission from the weather station fails, and 
 I have unknown values for a few minutes. A short gap in the daily temps gauge 
 is not a problem, but for any day with a UN Value I do not get 
 min/max/average calculations...
 
 
 That is the command for calculation the weekly values:
 
 rrdtool graph temps-week-2103.png   -s 'now - 1 week' -e 'now'  \
 --title Temperatur in LE  \
 DEF:temps9=/var/rrd/weather.rrd:temps9:AVERAGE  \
 VDEF:temps9min=temps9,MINIMUM \
 VDEF:temps9max=temps9,MAXIMUM \
 VDEF:temps9last=temps9,LAST \
 COMMENT:\t\t\tMin   \
 COMMENT:Max   \
 COMMENT:Aktuell \
 COMMENT:\n \
 LINE4:temps9#00C000:KS-300  \
 GPRINT:temps9min:%6.1lf  \
 GPRINT:temps9max:%6.1lf %S \
 GPRINT:temps9last:%6.1lf %S \n

You would probably be better having an RRA for min and max in your database 
rather than just one for average. Two reasons - firstly you can specify the XFF 
value (eg allow 50% to be unknown before the RRA becomes unknown), and also 
because min and max will become more and more inaccurate the further the data 
is consolidated.
On the latter, you could have a day that swings between say -10˚ at night and 
+10˚ during the day - but after consolidation it could end up with 
min=max=average=0 !

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Updates once a day

[Simon Hobson]

Simone Morandini simone.morand...@mix-it.net wrote:

 With this setup, when I issue an rrdtool fetch command, shouldn't I see one 
 row per day? I have the update script running since a week now, but I still 
 see one sigle row of output.

Yes, but did you specify a period and resolution ? It's possible that it's 
picking the wrong resolution and you don't have enough data to get multiple 
lines.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Inserted value becomes NaN

[Simon Hobson]

Jon August jonaug...@gmail.com wrote:

 I do an update like this:
 
 rrdtool updatev filename.rrd N:1102
 
 What am I doing wrong?

One update does not create a value in the database - it merely sets a known 
point, and for the very first update, all data up to that point is unknown. 
You need to make repeated updates over a period of time. Once you pass the 
*END* of a data point, then a value will be computed for the previous interval 
- but only if enough data has been entered for the interval to have a valid 
value, that's what the fudge factor (0.5 in your example) is for.

So in your case, you need multiple updates, not more than 48 hours apart (or 
the data becomes unknown again), and after the end of a day* you will get a 
value if you have enough data to fill the previous 12 hours (0.5 x 1day) with 
valid data.

* Note that a day will start/end at midnight UTC. All intervals end at an 
integer multiple of step after midnight Jan 1 1970 (unix epoch).

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] perfdata not correctly parsed ?

[Simon Hobson]

Nicolas - rrdtool rrdt...@chirgui.eu wrote:

 I then modified the plugin so that it outputs raw values, the new perfdata is 
 the following :
 
 '/mnt/disk1'=4354426724352B;5110741642076;5220650064486;0;5495421120512
 
 But now, the graph doesn't recognize the values (too big ?), as it says 
 -nan'.

I'd be tempted to have the perfdata output as MB, but with no units (ie just 
the number). Thee days, it's hard to find a hard disk who's size (in bytes) 
isn't a very large number !

Also, remember that you need to allow up to a couple of step periods before you 
have valid data. If you'd just increased the numbers by 2^20, that's a big step 
and may well have exceeded the upper bounds on rate - or if there had been a 
gap there may well have been unknowns in the database. So allow at least 2xstep 
value (assuming you have a consolidation @ 1step resolution) before expecting 
valid data.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Measuring rain from weatherstation

[Simon Hobson]

Alex van den Bogaerdt a...@vandenbogaerdt.nl wrote:

 As before: don't write to RRDtool unless the value tipped over to the next 
 number.  There won't be spikes. The amount of rain will be divided by the 
 amount of seconds since the last update.

So use derive data type, write 0.5 (or any larger value that's accumulated) 
followed by 0 whenever the counter tips over ?
Guess we need to write 0 periodically to avoid getting long periods of unknown.

So we end up with :

forever
  read value
  if value  min
reset counter
reset timer
write T:value to rrd
writeT+1:0 to rrd
  else
if timer  some_period
write T:0 to rrd
reset timer
  sleep for a bit
go back and do it all again

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Measuring rain from weatherstation

[Simon Hobson]

Alex van den Bogaerdt a...@vandenbogaerdt.nl wrote:

 After getting above mentioned stuff right, you could try to increase the 
 accuracy of your rainfall meter.  It involves remembering values read from 
 it.  An example works best:
 
 At time 09:55 you get 1.
 At time 10:00 you get 2.
 Upto and including time 13:55 you get 2.
 At time 14:00 you get 5.
 Upto and including time 17:55 you get 5.
 At time 18:00 you get 10.
 So far you have been logging this as 1, 2, 5 and 10, but with some 
 programming you could achieve more accurate results.
 
 If you remember the read values and timestamps, then you can do something 
 like this:
 * do not update the database for rainfall, unless the value changes
 * do update the database every midnight, even if there was no change
 * as soon as the read value changes, you record the previous value in the 
 database.
 
 The result: at 10:00 you record 1. Then you wait. At time 14:00 you record 
 2. RRDtool will automatically fill the time intervals between 10:00 and 
 14:00 with the new computed rate of 1mm in 4 hours = 1/14400 mm/s.
 At time 18:00 you feed RRDtool the number 5. The computed rate will be 4mm in 
 4 hours = 1/3600 mm/s.
 
 If the value does not change anymore, then at midnight you have a choice. 
 Either log 5 again, or log 10, or log 7.5 being the average of those two. You 
 really do not know how much rain has fallen, you only know it is less than 10 
 but not less than 5.

If you have the option of resetting the counter after an arbitrary reading, 
then I could see a way of getting it more accurate.

Read the value quite often. While it is returning 0.5 then assume 0. When 
it switched to 1 then use 0.5 and reset the counter. You will potentially 
lose a tiny amount of rainfall, but you'd have to be resetting the counter just 
at the point where the bucket tips (assuming it's a tipping bucket system) - 
but unless your reading points are a long way apart, that's unlikely.

You now have a value that's normally 0 but pulses to 0.5 each time 0.5 units 
of rain are measured. My mind isn't working well enough right now to work out 
the best way to put that into the RRD database.
One thought is to accumulate units of rain into a file which is read by your 
script that updates the rest of the weather parameters.
Another is just to insert 0 or 0.5 at frequent intervals - you'll get a spike 
each time the rainfall reaches 0.5mm, which can be averaged out by RRD. 
Probably need to use derive and insert 0.5 followed by 0 one second later.

You'd need to allow for larger values - in case your script gets delayed for 
some reason.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] How to plot clustered column graph ?

[Simon Hobson]

Joel HATSCH j...@joel-hatsch.net wrote:

 Everybody knows that kind of representation, but now question is how to
 draw this with RRDgraph ? I'm not aware of a command to make columns
 narrower and group graphs side by side.

It can't be done, or at least, not without a lot of fudging around.
You'd probably be better exporting the values and calling something like (IIRC) 
GNUPlot to draw the graph.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Start graph at 1st of current month?

[Simon Hobson]

 but what if i want my graph to start at the 1st of the current month?
 all i could come up with was something like this:
 start=midnight 01.11
 which will indeed start the graph at the 1st of November, but not very handy
 as i have to modify the RPN for every concurrent month.

Use start=${StartTime} and set StartTime in the script you use to call RRD.

RRD does not have any concept of week, month, year - and only limited 
concept of day given that midnight is only midnight in UTC timezone. That 
means you need to do some work external to RRD if you need to work with 
calendar periods.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] be more selective with datasources to plot

[Simon Hobson]

Alceu R. de Freitas Jr. glasswal...@yahoo.com.br wrote:

 In most of cases, only part of those datasources will have interesting 
 results (~8% of datasources will have values near zero). I would like to 
 filter only the datasources that have a value greater than a threshold.

Fetch isn't a useful tool for this. Use graph, and PRINT (not GPRINT) to get 
the values you want (eg data source and max value). You can then use these in a 
script to dynamically generate a graph which only displays the data sources of 
interest.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] not being able to get exactly updates made

[Simon Hobson]

Alceu R. de Freitas Jr. glasswal...@yahoo.com.br wrote:

 Am I not following the concepts of RRDTool

Yes

 or there is possibly a bug with RRDTool::OO?

No bug - it's working as designed.

See Rates, normalizing and consolidating on this page
http://www.vandenbogaerdt.nl/rrdtool/

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] find top 10 in 4000 rrd files

[Simon Hobson]

Rob Hassing rob.hass...@deltics.nl wrote:

 I am measuring the bandwidth usage of over 4000 ports in a network using
 sFlow. 
 
 The sFlow daemon I use generates a rrd file for each port. 
 
 So I have over 4000 rrd files named: x.x.x.x-Y.rrd 
 Where x.x.x.x is the ip address of the host and y is the portindex number. 
 
 Now I would like to find the top 10 of bandwidth usage in these 4000 files. 

There's nothing in RRD to do this. I think what you'll need to do is use 
rrdtool fetch or graph* to get the data you want from each file, stuffing it 
into a normal database - and then do your query on the database you've 
created.

* rrdtool fetch will only give you actual values stored in the RRD file - 
multiple values if you ask for a period other than a single CDP. If you use 
rrdtool graph, you can use all the RPN stuff to munge data and then use PRINT 
(not GPRINT) to output the (I assume) single value you are after.

Alternatively, you might run a periodic task to fetch certain data from your 
RRDs and insert/update a normal database. You could then run queries against 
that database.
For example, suppose you have data consolidated to 24 hour resolution in your 
RRDs, and shortly after midnight you pull this and update your (eg) SQL 
database. If you want top 10 for the previous 7 days then you do a query, 
group by x.x.x.x-Y, and with a total(d) in the select clause - sort by total(d) 
desc and you've got your top 10 in the first 10 results out.


Neither way is right or wrong, they just have different tradeoffs. The first 
method involves a lot of work (and modest storage) when you want to run the 
query, but nothing at other times. The second method involves periodic work and 
more storage even if you don't make any queries - but when you do run queries 
the results will come out quicker. So a lot depends on how often you will want 
to be running this.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Bar graph by hour, day, month

[Simon Hobson]

Bruno Suarez bsua...@free.fr wrote:

Some clarification is needed, it could be that you are using terms that mean 
one thing to you and something else to us.

 I want to make graphs of water consumption.
 I have a counter that is incremented for every liter of water consumed,
 this counter will be stored in the database and reset every minute.

Can you clarify that.
I take you mean you'll be reading the counter and updating the RRD file every 
minute, but does the act of reading the counter reset it to zero or not ?

 In the end I would make graphs (bar) to view the water consumption for
 each period (eg 5 min, 1 hour, 1 day, 1 month ...)

Do you want the bars to match specific time periods - eg have a bar that covers 
1st Sept to 30th Sept and so on ?
5 minutes is fairly easy - if your RRD consolidates data to a minimum 
resolution of 5 minutes then that's what you'll get. But once you get up to 
days and weeks it gets harder - depending on what you want.
Showing a graph of the last 7 days or the last 31 days is easy. Doing it 
for the last calendar week (Sun -Sat or Mon-Sun) or last calendar month 
(1st to 28th/30th/31st) is somewhat harder.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Bar graph by hour, day, month

[Simon Hobson]

bsua...@free.fr wrote:

 My water meter delivers one pulse per liter consumed. I have a script that 
 counts the pulses (raspberry and piface).
 
 Periodically I want to store in the RRD database  the number of pulses.
 The first solution is to periodically store the number of impusions since the 
 last backup.
 The second solution is to periodically store the total impusions since the 
 start of counting

Counter or Derive will work. See 
http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html where they are explained 
and see the NOTE on COUNTER vs DERIVE.
Alternatively, you could use Absolute and reset the counter each time - then 
you don't need to worry about storing the total.

 For graphs, displaying data on last 5 minutes, last hour, last day, last 
 month should be enough.

Assuming by last week you mean previous 7 days ending now rather than 
previous calendar week then it shouldn't be too hard. Decide what resolution 
you want, and how much data to store at each resolution, then RRD will just do 
it. However, to get one bar per day you may have to force RRD tool to use a 
specific consolidation while graphing (that's not hard) if there is higher 
resolution data available.
Eg, supposing you had 2 hour consolidations for the last 14 days, by default 
if you plot a graph over the last week then RRD will use that 2 hourly data and 
draw you a wiggly line. If you tell graph to use another consolidation which 
has only one sample/day then it will oblige by drawing a wiggly line which 
looks very much like a bar chart.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Can I stuff an RRD with data after-the-fact?

[Simon Hobson]

On 16 Sep 2014, at 19:17, Alan McKay alan.mc...@gmail.com wrote:

 On Tue, Sep 16, 2014 at 11:00 AM, Simon Hobson li...@thehobsons.co.uk wrote:
 That will work just fine *IF* none of the data in your file is older than 
 data in the RRD file AND all the timestamps in the file are in increasing 
 order. With each update, the RRD will update just the same as if you'd fed 
 them in in real time as the data was collected.
 
 Well I only wrote this because it did not seem to work at all.  It was
 a fresh RRD - newly created.  And timestamps were increasing.

As already mentioned, you can specify the start time for an RRD when it's 
created - the default is now.

 Lastly, have you considered using rrdcached ? Collect the data on one 
 machine, and do rrdtool updates from there specifying the cached address - 
 the data is then transferred to the other machine and the RRD updated in 
 real time*. It works really well for distributed data collection like this - 
 I use it on many of my systems.
 
 I'll have a look at it, but in our environment any kind of extraneous
 agent or daemon can only be introduced after considerable scrutiny.
 Which is why I did not want to run rrdtool locally.  THough I'm going
 through the process to get it introduced.
 It is technically not a daemon like that, so should be good.
 Also, I can't really allow data to be pushed from a host to a collector.
 I could only pull from the collector.

Do you run Nagios (or anything similar) ? For some of my stats, I twigged that 
Nagios already collected some performance stats (but not all I wanted) - it 
just needs a bit of scripting to drop that data into a file and process it.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Can I stuff an RRD with data after-the-fact?

[Simon Hobson]

 Do you run Nagios (or anything similar) ? For some of my stats, I twigged 
 that Nagios already collected some performance stats (but not all I wanted) - 
 it just needs a bit of scripting to drop that data into a file and process it.

http://nagios.sourceforge.net/docs/3_0/perfdata.html

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Can I stuff an RRD with data after-the-fact?

[Simon Hobson]

Alan McKay alan.mc...@gmail.com wrote:

 For various reasons I want to collect data in my own logfile formatted thus :
 
 EPOCHTIME:data1:data2
 
 And then when I want to see a graph, copy that over to another host and feed 
 it
 into an RRD file, then graph it from there.

That's OK.

 But of course in doing so I get the
 error :
 
 illegal attempt to update using time when last update time is

There's no of course about it.

 Is there a way to do this?   Basically do the following but tell it to
 force timestamps
 
 #!/bin/bash
 
 while read line
 do
 rrdtool update foobar-1yr-5sec.rrd -t diskiops:diskutil $line
 done  diskperf-1yr-5sec.log

That will work just fine *IF* none of the data in your file is older than data 
in the RRD file AND all the timestamps in the file are in increasing order. 
With each update, the RRD will update just the same as if you'd fed them in in 
real time as the data was collected.

Now, if you have (say) a big data file that you keep adding to, and you are 
trying to update an existing RRD file that's already had some of the data 
inserted then you'll get the error. In that case, you'd need to modify the 
script a bit. I see two ways of dealing with it :

1) Just throw away STDERR so you don't see the errors. RRD will barf on each 
update it's already seen, but then add the new stuff.

2) Have your script use rrdtool last to get the timestamp of (IIRC) the last 
complete bucket, and discard all the entries older than this - then insert 
the new values. You might still get one or two errors - IIRC rrdtool last gives 
the timestamp of the last complete bucket (aka Primary Data Point) which is 
likely to be earlier than the timestamp of the last value inserted.

Lastly, have you considered using rrdcached ? Collect the data on one machine, 
and do rrdtool updates from there specifying the cached address - the data is 
then transferred to the other machine and the RRD updated in real time*. It 
works really well for distributed data collection like this - I use it on many 
of my systems.

* Subject to flushing the cache.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] How to plan RRD creation from previously acquired data?

[Simon Hobson]

gschanuel gabriel.schan...@gmail.com wrote:

 I guess there is something to do with archives or step.. but I really have
 no idea!

The problem is the heartbeat (plus the format already mentioned).

Your new RRD is expecting data spaced by (at most) 600 seconds - while your 
data is spaced a LOT wider than that. You'll need to set your heartbeat to 
longer than the period between updates while you import the data. WHat will 
happen then is that when an update comes in, all the buckets from the previous 
update to the new one will get filled in - instead of what happens now which is 
that they get filled with NaN as the interval is too long for the heartbeat.
After import, you should be able to tune the heartbeats back to a normal 
setting.

Alternatively, you'll need to write a program that fills in the blank and 
does multiple updates spaced not more than heartbeat apart.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Empty sections in AREA on graph, but values in database

[Simon Hobson]

Out of interest, what happens if you simply add a cdef for stufe*100 
(CDEF:stufe2=stufe,100,*) and plot a line for that ? I'd expect to see an 
almost straight line at 100, 200, or 300. Remove the plot commands for the 
other stuff and the autoscaling should show (say) 99 to 101 or something like 
that which would accentuate any wobble.

My guess is that internal maths and rounding means that you don't get exactly 
1, 2, 3 - could be 0.99 or 1.01 for example.

I'd be tempted to re-write the code like this :
CDEF:stufe0_flag=stufe,0.5,LT,INF,UNKN,IF
CDEF:stufe1_flag=stufe,0.5,GE,INF,UNKN,IF
CDEF:stufe2_flag=stufe,1.5,GE,INF,UNKN,IF
CDEF:stufe3_flag=stufe,2.5,GE,INF,UNKN,IF

AREA:stufe0_flag#7c6767:Stufe 0 AUS \\t
AREA:stufe1_flag#98FB98:Stufe 1 \\t
AREA:stufe2_flag#fafa78:Stufe 2 \\t
AREA:stufe3_flag#fd6969:Stufe 3 \\t

This would also mean that your graph would work after consolidation of data. 
Eg, at present, if you graph over (say) a week instead of a day, then there 
will be periods where the values is changing between (say) 1  2 and the result 
will be 1.something for that consolidated period. Using equality comparisons, 
these values won't math anything, but using comparative statements, it'll show 
as 1 if the consolidated value is less than 1.5 and 2 if it's over that.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] max / min values not precise? Bug?

[Simon Hobson]

spock collec...@sappers.de wrote:

 I have difficulties to interpret the graph created with the parameters below
 (I omitted some lines, but relevant data should be there).
 
 http://rrd-mailinglists.937164.n2.nabble.com/file/n7582374/tempmo1.png 
 
 The graph shows
 - one line with high resolution data
 - one line with minimum values per day (step = 86400)
 - one line with maximum values per day  (step = 86400)
 
 On the graph I do not understand, why the maximum value of the high
 resolution data does not correspond with the minimum / maximum values of the
 compressed data.

Because the line is not showing all the data.

 On the first and second day e.g. the max values are higher than any data
 point of the actual high resolution data.

 # rrdtool create temp_pool.rrd --step 900 --start 20140101 00:00 \

 graph_defs='-w 900 -h 525 -D -a PNG -T 15 --slope-mode \

First off, you haven't provided all the information - like what timescale the 
graph is plotted over (it looks like a bit over 2 weeks) or the actual data.
At a step of 900s, that's 96 periods per day. Over 16 days, that's over 1500 
periods. You are plotting a graph that's around a third of that in pixels, so 
around 3 points will be averaged per pixel.

So you'd need to look carefully at the actual data to see what is missing 
from the graph you are plotting.

What I've done for a few of my graphs is plot a shaded area between min and 
max. Do a CDEF x_spread=x_max,x_min,- (where x_min is x:MINIMUM and x_max is 
x:MAXIMUM) and plot an area with full transparency of x_min, then stack and 
area of (say) 50% transparency of x_spread on top of it. Then draw your line of 
x. I suspect that with your parameters you'll find that the shaded area touches 
the min and max lines.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] How to calculate desired value?

[Simon Hobson]

spock collec...@sappers.de wrote:

 I tried:
   CDEF:heattime2=heat_flag,900,*,60,/ \
   VDEF:totalheattime=heattime2,AVERAGE'
   GPRINT:totalheattime:Total\: %2.2lf h \n'
 but I get wrong results. I probably have to do the calculation completly
 different.
 
 Can anyone help me out here?
 
 I should get perfectly aligned values as a result: 0.25 h; 0.50h; etc.
 How can I achieve this?

Give us a clue to go on, what values do you get ?
Is it that you expected (say) 15 but got 14.99, or you got something completely 
different ?
Your approach is right, I suspect rounding errors come into it.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Issue: only the first value gets into the database

[Simon Hobson]

spookyx p...@infocon.ro wrote:

 rrdtool update wazers_ro.rrd
 N:${USERS[1]}:${USERS[2]}:${USERS[3]}:${USERS[4]}:${USERS[5]}:${USERS[6]}:${USERS[7]}:${USERS[8]}:${USERS[9]}

Create a short test script that does updates with fixed values - both for time 
and inputs. We have absolutely no idea when you ran the command, nor what the 
contents of your array were. If we don't know what went in, we cannot even 
start to guess at what should come out.
Also, as already pointed out, you need enough updates to have values for a 
complete period of time before you'll get data out.

Go to http://www.vandenbogaerdt.nl/rrdtool/ and in particular see the tutorial 
on Rates, normalizing and consolidating - it'll help a lot.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Can't graph anything from populated rrd

[Simon Hobson]

Victor Mercado victor.merc...@ahold.com wrote:

 rrdtool.create('%s' % rrd_input_file, '--start', 'now', '--step', '120',
'DS:Jitter:GAUGE:5000:U:U',
'DS:PacketLossPerc:GAUGE:5000:U:U',
'RRA:AVERAGE:0.5:1:1440',
'RRA:AVERAGE:0.5:1:3600',
'RRA:AVERAGE:0.5:1:1440',
'RRA:AVERAGE:0.5:1:3600')

You have 3 redundant RRAs there - 1440 is a subset of 3600, and they all 
consolidate the same way (ie nothing as it's one PDP (Primary Data point) per 
CDP (Consolidated Data Point).

 Unfortunately, I'm unable to graph the data.  I'm trying to create a simple 
 graph without any optional parameters just to see something.  The syntax is a 
 follows:
 
 $ rrdtool graph -j rrd filename  graph.png
 
 Unfortunately, I don't get any output.

You won't because that doesn't tell RRD to produce anything.
See :
http://oss.oetiker.ch/rrdtool/doc/rrdgraph_graph.en.html
http://www.vandenbogaerdt.nl/rrdtool/

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] api for iterating over single rows in a rrd file

[Simon Hobson]

Daniel Pocock dan...@pocock.pro wrote:


 From a technical perspective, if rrdtool provided such an API:
 - it would make programming easier for people accessing data in this way
 - it could be optimized for efficiency (e.g. the IO subsystem would be
 reading 4KiB of data from each file at a time, the iterator would then
 return values from that buffer and the IO subsystem would only do more
 reads when the iterator advances over a page boundary)
 
 From a user perspective, it would enable people to look across all their
 RRDs, e.g. to answer questions like which CPUs were over 90% utilized
 at any time between 09:00 and 10:00

I can see the point of being able to iterate through the data (open, seek, get 
row, get row, ...) - I'm sure there are probably a few applications where that 
would be useful.

I don't see where having RRD handle multiple files would be any better than the 
program itself opening multiple files. I suspect it would be a massively 
complicated task to build generic open and merge multiple files function 
given the variety of ways they can be configured.
Better, IMO, for your program to handle that - you are in a position to know 
what RRDs you've set up, and overall it's probably no harder than finding a 
reliable way to tell RRD what you want.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Fetch the time of the first entry

[Simon Hobson]

Steve Shipway s.ship...@auckland.ac.nz wrote:

 You seem to be misunderstanding how RRDTool works.
 
 Remember that, after every update of an RRA, the oldest value is thrown away 
 to make room for the new one.  So, the 'first' time (the timestamp of the 
 oldest bucket in the RRA) will increase every time you update, as the last 
 bucket is thrown away.
 
 Also, the 'first' value is the oldest bucket, *whether or not it has been 
 filled*.

As I pointed out, it looks like the documentation is a little misleading on 
this. Where teh database has been filled and old data thrown away then the 
documentation is correct - but in the OPs case, it's misleading by talking 
about the first data sample entered into the RRA.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Fetch the time of the first entry

[Simon Hobson]

Alex van den Bogaerdt a...@vandenbogaerdt.nl wrote:

 The other option is to modify the documentation to match the source.

How about changing it to first value in the RRA, plus a note along the lines 
of :
The value returned will always be the timestamp of the last completed entry in 
the RRA minus the timespan covered by the RRA*. For RRAs which cover a timespan 
longer than the periods covered by updates, the returned value will predate the 
first update.

There are probably better words to express it, but they aren't flowing through 
my mind at the moment :(
* Is this the correct description ?

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Fetch the time of the first entry

[Simon Hobson]

Steven Sim unixan...@outlook.com wrote:

 I'm being asked to write a scrip to append data to an rrd and plot daily, 
 weekly and monthly graphs of said data.
 
 I already have a script to create an rrd and write data to it.
 
 To append, I'll simply bypass the 'creation' logic of my script and use a 
 user specified rrd file.

It may be just the way you write, but the wording makes me wonder if you 
properly understand how RRD works. Because in RRD, there is no such function as 
appending data - only updating. You specify the number of consolidated data 
points to keep at the point of creation, after that the size never changes and 
you only update values.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Fetch the time of the first entry

[Simon Hobson]

Steven Sim unixan...@outlook.com wrote:

 I intend to process a data file each day but create the rrd database on the 
 FIRST day of the month.
 
 The creation shall ensure sufficient data points in the first day to contain 
 the entire month but plot daily graphs until the end of the month, whereby it 
 will plot the entire month.
 
 Say for example,
 
 If i had a step interval of 15 minutes (900 seconds), 
 
 rrdtool create $RRDB --step 900 --start $STARTIME \
 DS:..
 DS:..
 .
 .
 .
 DS:..
 DS:..
 RRA:AVERAGE:0.5:1:2880
 
 2880 = 24 hours x 4 data points per hour x 30 (thirty days in a month)
 
 4 data points since one hour has 4 data points (15 minutes step).
 
 Would the above be correct?

Yes, your numbers are correct, but I wonder about your methodology. It might 
help if you said what you are trying to achieve, because it's a rather unusual 
way of using RRD. Normally, you simply create one RRD that collects, stores, 
and consolidates the data you want. Typically this means keeping high 
resolution data for a shortish time, and keeping progressively lower resolution 
for progressively longer times - eg most applications don't need to keep (say) 
5 minute resolution traffic data for 2 years ago. That's not to say you can't 
keep several years worth of high-resolution data if you want to.

By keeping separate RRD files for each 'month', you'll find that it's hard work 
if you later want to do a graph for a year ! You'd also need to lock your 
filled data files otherwise one update to the wrong file could wipe out all 
the data.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Fetch the time of the first entry

[Simon Hobson]

Steven Sim unixan...@outlook.com wrote:

 my rrdtool first rrdfile command keeps returning different timing after 
 each rrdtool update.
 
 Why is that so?
 
 Shouldn't it ALWAYS return the Unix time stamp for the first entry?

It's not a function I've used ...

The docs say it should return the first value entered. Once you've filled a 
data set, then it's going to give you the timestamp of the oldest available 
value - I'm not sure what it gives you for an RRD file that hasn't been 
filled.
Here I'm using the term filled to mean you done enough updates that the whose 
database (or at least data series) has been filled with data and you are now 
losing older data to make way for the new data.

Have you looked at how the value is changing ? Does it by any chance advance at 
the same rate as the timestamp of your updates ?

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Fetch the time of the first entry

[Simon Hobson]

Steven Sim unixan...@outlook.com wrote:

 BUT when I use rrdtool first rrd database in an attempt to get the time of 
 the first entry, I get a Unix time stamp which is one entire month EARLIER 
 than the first entry in the file.
 
 And each time I do update, rrdtool first rrd_database returns a DIFFERENT 
 number.

With each update, does the FIRST value update to be one month (or 30 days) 
before the timestamp of the update ?

If so then I think I know what's happening. When you create teh RRD file, it is 
created in it's entirety - in your case with buckets for 2880 consolidated 
values. These exist regardless of what updates you do or do not do, and the 
timespan of them is determined by your step and consolidation values.
What I suspect is happening, and what I alluded to earlier, is that even though 
you haven't done any updates for those historical buckets, they are still there 
- and FIRST is merely reporting the timestamp of the oldest bucket. Since you 
are setting the start time of the RRD when you create it, then the timestamp of 
the oldest bucket will be 30 days prior to that. As you perform updates, you 
overwrite the oldest buckets and the value of FIRST will advance to be 30 days 
prior to the last update.

Internally to RRD, there is no storage for whether a bucket actually had any 
updates - only for it's value after applying the consolidation rules specified. 
Thus there is no way to know if the oldest bucket ever had an update if it's 
value is NaN - you don't know if it was created with NaN and has never been 
overwritten, or if it was updated but the calculated value was NaN. You would 
have to search back through the database to find the oldest bucket with an 
actual value and infer that this bucket was *probably* the one with the oldest 
data.

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Fetch the time of the first entry

[Simon Hobson]

Steven Sim unixan...@outlook.com wrote:

 When I create the RRD, I am forced to give the --start TIME as one step 
 BEHIND my actual update.
 
 So let's say my data starts at midnight 1st July, I am forced to create a RRD 
 with --start time at 11:45 30th of June.
 
 data starts - 1st July 00:00:00 -- first data entry.
 Step = 15 minutes (900 seconds)
 RRD has to be created with --start 11:45 30th of June (one step behind).
 
 if I do not create it 1 step behind, I get an error with my updates.

This came up recently. In effect, the START parameter sets the last update time 
for the RRD, as you can't have two updates with the same time, you need to set 
the start parameter to before the timestamp of your first update.
In principal the actual value doesn't matter - it could even be 0. However, if 
it's a long time ago then on your first update, RRD will have to roll forward 
and process all the non-existant updates to get up to date. How this came up 
was someone found that if they had no updates for a few days (machine down) 
then they found the first update took a lot longer than normal.
___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Fetch the time of the first entry

[Simon Hobson]

Alex van den Bogaerdt a...@vandenbogaerdt.nl wrote:

 As already explained, rrdtool first gives you the first available slot.

Looks like the documentation is misleading then :

http://oss.oetiker.ch/rrdtool/doc/rrdfirst.en.html says :
 The first function returns the UNIX timestamp of the first data sample 
 entered into the specified RRA of the RRD file.

Can't just come up with the right words though. It's the timestamp of the 
oldest bucket, or is CDP slot a better term ?

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] wrong rra used

[Simon Hobson]

spock collec...@sappers.de wrote:

 I am using this database to collect temperature data for a swimming pool.
 I want to keep the detailed data for 36 months (103680 * 900sec) but also
 want to have a daily average temp (96 values = 24h).
 
 # rrdtool create temp_pool.rrd --step 900 --start 20140101 00:00 \
 # DS:air:GAUGE:1200:-40:50 \
 # DS:flow:GAUGE:1200:-40:50 \
 # DS:return:GAUGE:1200:-40:50 \
 # DS:delta:COMPUTE:return,flow,- \
 # DS:pump_flag:GAUGE:1200:0:1 \
 # DS:heat_flag:GAUGE:1200:0:1 \
 # DS:device:GAUGE:1200:0:90
 # RRA:AVERAGE:0.5:1:103680 \
 # RRA:MIN:0.5:96:3650 \
 # RRA:MAX:0.5:96:3650 \
 # RRA:AVERAGE:0.5:96:3650
 
 When I create a graph with 
   rrdtool graph temp7d.png \
-t Pooltemp (Status $(date +%d.%m.%Y %H:%M))  \
-s $graph_start_7d -e $timestamp_rounded_unix \
-w 640 -h 400 -D -a PNG -T 15\
DEF:airmax2=temp_pool.rrd:air:MAX \
DEF:airmin2=temp_pool.rrd:air:MIN \
DEF:airavg2=temp_pool.rrd:air:AVERAGE \
CDEF:airrange=airmax2,airmin2,- \
LINE2:airmin2#00FF00:Luft Min \
AREA:airrange#8dadf588::STACK \
LINE2:airmax2#00FF00:Luft Min \
LINE2:airavg2#00FF00:Aussen
 
 I get for airavg2 the datailed values - but not the daily average.
 Airmin2 and airmax2 work as  expected - there is no other RRA Archive to
 choose from.
 
 Question (1)
 How do I convince the tool to use the consolidated data instead of the
 detailed?

It does the best it can to use the most detailed data. Looking at the docs, I 
don't see any option to override that.


 Question (2)
 When drawing the graph, the line for max/min values are not precisely
 aligned above a day. It looks like it starts about 900 secs after 0:00 each
 day.

Looks more like an hour to me, are you on +1 timezone by any chance ?

 How does rrdtool determine, which 96 values to consolidate? My data point
 are stored with rounded timestamps (e.g. 13.06.2014 0:00, 0:15, 0:30, etc.)

*ALL* periods are aligned to an integer multiple of step (or step * steps/CDP) 
since unix epoch (midnight 1-1-1970 UTC). So if you consolidate to a day then a 
day will always start at 00:00:00 UTC. If your timezone isn't UTC, then the 
data will appear to be off by your timezone offset.


 Question (3)
 The ssf factor is 0.5 for the RRAs. However, there is no min/max value for
 certain days with missing data, although there are only a few values
 missing. I am sure, is is not more than 50%. How can I check this.
 
 http://rrd-mailinglists.937164.n2.nabble.com/file/n7582214/temp7d.png 

Are you certain you are updating frequently enough ? I can see some gaps in the 
data, if you have more than 1200s between updates then you'll have gaps.


For more on how consolidation is done, see this tutorial
http://www.vandenbogaerdt.nl/rrdtool/process.php

___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users