[rrd-users] Re: Data granularity
Serge Maandag wrote: > > > >I still would use the log file solution. >It's way cleaner. > > I took mailgraph as an example for my scripts, and I build this with it: http://haas.oezie.org/rrd/httpd/ (and the script here http://haas.oezie.org/rrd/httpd.pl) it works like a tail -f logfile, and counts the new lines. strips them and count the apache errorcodes 200,404,502 etc. I will write som documentation about it like in http://haas.oezie.org/rrd/cpu.htm (sorry dutch only, maybe later in englisch). maybe you could take it also as an example and adjust i to your needs. mvg Erik de Mare >Serge. > >- >Op de inhoud van dit e-mailbericht en de daaraan gehechte bijlagen is de >inhoud van de volgende disclaimer van toepassing: >http://www.zeelandnet.nl/disclaimer.php > >-- >Unsubscribe mailto:[EMAIL PROTECTED] >Helpmailto:[EMAIL PROTECTED] >Archive http://www.ee.ethz.ch/~slist/rrd-users >WebAdminhttp://www.ee.ethz.ch/~slist/lsg2.cgi > > > -- Unsubscribe mailto:[EMAIL PROTECTED] Helpmailto:[EMAIL PROTECTED] Archive http://www.ee.ethz.ch/~slist/rrd-users WebAdminhttp://www.ee.ethz.ch/~slist/lsg2.cgi
[rrd-users] Re: Data granularity
> rrdtool create resp.rrd --step 5 \ > DS:resp:GAUGE:10:0:U \ > RRA:AVERAGE:0.999:1:1000 > > If I do single data point update using: > > rrdtool update resp.rrd 1106249083:45 > > and *not* post any update for a while ( > 10 seconds ), the > RRD fetch command shows that the AVERAGE CF has the following value: > 1106249083: nan > > Why do I have a NaN? Shouldn't I have "4.50e+01" > instead? No, rrdtool normalizes your input. Therefore it really only is useful if you enter data in a regular interval. Say you updated like this: rrdtool update resp.rrd 1106249073:15 rrdtool update resp.rrd 1106249083:45 Then after the second update, rrdtool is able to determine the value At timestamp 1106249080. It would be: (45-15) * (1106249080-1106249073)/(1106249083-1106249073) = 30 * 7/10 = 21 It is interpolated because you did not update on a N x 10 seconds timestamp In your case the value is unknown, because you have no previous value to use for interpolation. You need multiple consecutive values for that. > Is there a way to retain the value entered for that step? One solution is to set the heartbeat very high, so multiple rows will be filled by a single update, but that is not what you want. What you could try is: - update only on N x 10 timestamps. So 1106249083 would become: 10 x int(1106249083/10) = 1106249080 or - before updating, do a "rrdtool last" to get the last time the database was updated. Then for ech step size between now and that last update time, update the database with 0. I still would use the log file solution. It's way cleaner. Serge. - Op de inhoud van dit e-mailbericht en de daaraan gehechte bijlagen is de inhoud van de volgende disclaimer van toepassing: http://www.zeelandnet.nl/disclaimer.php -- Unsubscribe mailto:[EMAIL PROTECTED] Helpmailto:[EMAIL PROTECTED] Archive http://www.ee.ethz.ch/~slist/rrd-users WebAdminhttp://www.ee.ethz.ch/~slist/lsg2.cgi
[rrd-users] Re: Data granularity
Serge, Thanks for your response and detailed explanation. I really appreciate that. I have been experimenting with different settings. Below is an example: rrdtool create resp.rrd --step 5 \ DS:resp:GAUGE:10:0:U \ RRA:AVERAGE:0.999:1:1000 If I do single data point update using: rrdtool update resp.rrd 1106249083:45 and *not* post any update for a while ( > 10 seconds ), the RRD fetch command shows that the AVERAGE CF has the following value: 1106249083: nan Why do I have a NaN? Shouldn't I have "4.50e+01" instead? Is there a way to retain the value entered for that step? Thanks, Tolga. Serge Maandag wrote: >>I have a monitor scripts that gets invoked every time a user >>interacts with a Web service. There are several Web services >>that we are collecting data from. Each one goes to a >>different RRD. Some services are very busy (several hits per >>second) and some are not (a few hits per >>week). There is no way for me to identify which is which... > > > I cannot see how this works. Rrdtool collects rates, while you use > It as a counting mechanism. But as long as it works for you.. > > >>I would like to use 5sec STEP so that there is enough detail >>for busy services. A few questions to RRD gurus: >> >>(1) What is best value for HEARTBEAT? Is 10 seconds too low? > > > That depends on what you want to see in your database. > If you set it to 10: > - updates that are longer apart than 10 seconds, will cause NaNs > to appear in your database > - updates that are between 5 and 10 seconds apart, will cause > 2 consecutive rows to be influenced, it will be averaged out. > > If you set heartnbeat to N x step, N rows will be affected by > An update, provided your high res RRA has a row time that is equal > To the step size. > > >>(2) What is the best value for XFF? > > > That also totally depends. > > Say you have these RRA's: > > RRA:AVERAGE:0.5:1:1000 > RRA:AVERAGE:0.5:10:300 > > Every step tim, one value will be put in the first RRA. > Every 10 step times, 10 values from the first RRA will be averaged > And the result will be put in the second RRA. Unless more than 50% > Of the 10 values are NaNs. If that is the case NaN will be put in the > Second RRA. > > If the second RRA would read: > > RRA:AVERAGE:0.9:10:300 > > Then the average of the 10 values would still be put in the second > RRA if 9 out of 10 values would be NaN. > > So it all depends on what you want to see. > > In your case I'd probably just dump the hits with a timestamp in a > log file and setup a cron job that would do every 5 minutes: > - read back all lines and count the hits that were within the last > 5 minutes > - update the RRD database with the sum of the counted hits as a gauge > value. > > Serge. > > - > Op de inhoud van dit e-mailbericht en de daaraan gehechte bijlagen is de > inhoud van de volgende disclaimer van toepassing: > http://www.zeelandnet.nl/disclaimer.php > > -- > Unsubscribe mailto:[EMAIL PROTECTED] > Helpmailto:[EMAIL PROTECTED] > Archive http://www.ee.ethz.ch/~slist/rrd-users > WebAdminhttp://www.ee.ethz.ch/~slist/lsg2.cgi > > > -- Tolga Yalcinkaya _ WSABI: A fiery condiment for Apache Axis! http://www.wsabi.org -- Unsubscribe mailto:[EMAIL PROTECTED] Helpmailto:[EMAIL PROTECTED] Archive http://www.ee.ethz.ch/~slist/rrd-users WebAdminhttp://www.ee.ethz.ch/~slist/lsg2.cgi
[rrd-users] Re: Converting rrd files from one system to another
Thanks everyone, this is just what I needed.
Regards,
Jim
Jim Horwath
GCUX
Andreas Maus <[EMAIL PROTECTED]>
Sent by: [EMAIL PROTECTED]
01/20/2005 08:33 AM
To: James B Horwath <[EMAIL PROTECTED]>
cc: rrd-users@list.ee.ethz.ch
Subject:[rrd-users] Re: Converting rrd files from one system to
another
On Thu, Jan 20, 2005 at 08:04:56AM -0500, James B Horwath wrote:
Hi James.
> for x in `ls *.rrd`
> do
> rrdtool xport ${x} > ${x}_xport
---^
xport ??? Thats wrong.
The name (and description in the man page) is a little bit misleading
("rrdxport - Export data in XML format based on data from one or several
RRD")
You should use rrdtool dump and rrdtool restore to convert databases.
rrddump - dump the contents of an RRD to XML format
rrdrestore - restore the contents of an RRD from its XML dump format
So you should run:
rrdtool dump rrd_on_old_host.rrd > rrd_on_old_host_dumped.xml
and restore them on the new host via:
rrdtool restore rrd_on_old_host_dumped.xml rrd_on_new_host.rrd
Easy, isn't it? ;)
Hope that helps,
Andreas.
--
Dipl.-Ing. Andreas Maus science+computing ag
System Administration Hagellocher Weg 71-75
mail: [EMAIL PROTECTED] 72070 Tuebingen, Germany
tel.: +49 7071 9457 456 www.science-computing.de
-- Binary/unsupported file stripped by Ecartis --
-- Err : No filename to use for decode, file stripped.
-- Type: application/pgp-signature
--
Unsubscribe mailto:[EMAIL PROTECTED]
Helpmailto:[EMAIL PROTECTED]
Archive http://www.ee.ethz.ch/~slist/rrd-users
WebAdminhttp://www.ee.ethz.ch/~slist/lsg2.cgi
-
This message, and any attachments to it, may contain information that is
privileged, confidential, and exempt from disclosure under applicable law.
If the reader of this message is not the intended recipient, you are
notified that any use, dissemination, distribution, copying, or
communication of this message is strictly prohibited. If you have received
this message in error, please notify the sender immediately by return
e-mail and delete the message and any attachments. Thank you.
--
Unsubscribe mailto:[EMAIL PROTECTED]
Helpmailto:[EMAIL PROTECTED]
Archive http://www.ee.ethz.ch/~slist/rrd-users
WebAdminhttp://www.ee.ethz.ch/~slist/lsg2.cgi
[rrd-users] Re: Converting rrd files from one system to another
On Thu, Jan 20, 2005 at 08:04:56AM -0500, James B Horwath wrote:
Hi James.
> for x in `ls *.rrd`
> do
> rrdtool xport ${x} > ${x}_xport
---^
xport ??? Thats wrong.
The name (and description in the man page) is a little bit misleading
("rrdxport - Export data in XML format based on data from one or several RRD")
You should use rrdtool dump and rrdtool restore to convert databases.
rrddump - dump the contents of an RRD to XML format
rrdrestore - restore the contents of an RRD from its XML dump format
So you should run:
rrdtool dump rrd_on_old_host.rrd > rrd_on_old_host_dumped.xml
and restore them on the new host via:
rrdtool restore rrd_on_old_host_dumped.xml rrd_on_new_host.rrd
Easy, isn't it? ;)
Hope that helps,
Andreas.
--
Dipl.-Ing. Andreas Maus science+computing ag
System Administration Hagellocher Weg 71-75
mail: [EMAIL PROTECTED] 72070 Tuebingen, Germany
tel.: +49 7071 9457 456 www.science-computing.de
-- Binary/unsupported file stripped by Ecartis --
-- Err : No filename to use for decode, file stripped.
-- Type: application/pgp-signature
--
Unsubscribe mailto:[EMAIL PROTECTED]
Helpmailto:[EMAIL PROTECTED]
Archive http://www.ee.ethz.ch/~slist/rrd-users
WebAdminhttp://www.ee.ethz.ch/~slist/lsg2.cgi
[rrd-users] Re: Converting rrd files from one system to another
> in it. I was hoping I could write a simple shell loop to dump
> everything:
> for x in `ls *.rrd`
> do
> rrdtool xport ${x} > ${x}_xport
> done
Wouldn't it be much easier to use "rrdtool dump" and "rrdtool restore"?
Serge.
-
Op de inhoud van dit e-mailbericht en de daaraan gehechte bijlagen is de inhoud
van de volgende disclaimer van toepassing:
http://www.zeelandnet.nl/disclaimer.php
--
Unsubscribe mailto:[EMAIL PROTECTED]
Helpmailto:[EMAIL PROTECTED]
Archive http://www.ee.ethz.ch/~slist/rrd-users
WebAdminhttp://www.ee.ethz.ch/~slist/lsg2.cgi
[rrd-users] Converting rrd files from one system to another
I am using rrtool 1.0.45 on Solaris 9 for mrtg and CPU stats via
routers2.cgi. I need to convert the files to an AIX system 5.2 ML4
running rrdtool 1.0.49. I searched the archives for hints and looked at
the documentation for using xport but I cna't get it right. For every
.rrd file I have to I have to specify start and stop times for everything
in it. I was hoping I could write a simple shell loop to dump
everything:
for x in `ls *.rrd`
do
rrdtool xport ${x} > ${x}_xport
done
However trying to dump a file without including command line arguments
gets me an error. I'm not sure what command line arguments I need. Can
someone point me to an example on the line of "rrdtool conversion for
dummies." :)
Thanks in advance,
Jim
Jim Horwath
GCUX
Unix Admin
[EMAIL PROTECTED]
-
This message, and any attachments to it, may contain information that is
privileged, confidential, and exempt from disclosure under applicable law.
If the reader of this message is not the intended recipient, you are
notified that any use, dissemination, distribution, copying, or
communication of this message is strictly prohibited. If you have received
this message in error, please notify the sender immediately by return
e-mail and delete the message and any attachments. Thank you.
--
Unsubscribe mailto:[EMAIL PROTECTED]
Helpmailto:[EMAIL PROTECTED]
Archive http://www.ee.ethz.ch/~slist/rrd-users
WebAdminhttp://www.ee.ethz.ch/~slist/lsg2.cgi
[rrd-users] Re: Data granularity
> I have a monitor scripts that gets invoked every time a user > interacts with a Web service. There are several Web services > that we are collecting data from. Each one goes to a > different RRD. Some services are very busy (several hits per > second) and some are not (a few hits per > week). There is no way for me to identify which is which... I cannot see how this works. Rrdtool collects rates, while you use It as a counting mechanism. But as long as it works for you.. > I would like to use 5sec STEP so that there is enough detail > for busy services. A few questions to RRD gurus: > > (1) What is best value for HEARTBEAT? Is 10 seconds too low? That depends on what you want to see in your database. If you set it to 10: - updates that are longer apart than 10 seconds, will cause NaNs to appear in your database - updates that are between 5 and 10 seconds apart, will cause 2 consecutive rows to be influenced, it will be averaged out. If you set heartnbeat to N x step, N rows will be affected by An update, provided your high res RRA has a row time that is equal To the step size. > (2) What is the best value for XFF? That also totally depends. Say you have these RRA's: RRA:AVERAGE:0.5:1:1000 RRA:AVERAGE:0.5:10:300 Every step tim, one value will be put in the first RRA. Every 10 step times, 10 values from the first RRA will be averaged And the result will be put in the second RRA. Unless more than 50% Of the 10 values are NaNs. If that is the case NaN will be put in the Second RRA. If the second RRA would read: RRA:AVERAGE:0.9:10:300 Then the average of the 10 values would still be put in the second RRA if 9 out of 10 values would be NaN. So it all depends on what you want to see. In your case I'd probably just dump the hits with a timestamp in a log file and setup a cron job that would do every 5 minutes: - read back all lines and count the hits that were within the last 5 minutes - update the RRD database with the sum of the counted hits as a gauge value. Serge. - Op de inhoud van dit e-mailbericht en de daaraan gehechte bijlagen is de inhoud van de volgende disclaimer van toepassing: http://www.zeelandnet.nl/disclaimer.php -- Unsubscribe mailto:[EMAIL PROTECTED] Helpmailto:[EMAIL PROTECTED] Archive http://www.ee.ethz.ch/~slist/rrd-users WebAdminhttp://www.ee.ethz.ch/~slist/lsg2.cgi
[rrd-users] Re: DS:var:ABSOLUTE:... simulation
Resend, because the mailer broke some line. Below a script that demonstrates an "ABSOLUTE" counter for counting e-mail messages (spam in this case). I posted it, because I couldn't find any satisfactory examples of ABSOLUTE and as a chance to get feedback from others and possibly help others. It seems to work fine, but I'm not an experienced rrdtool user, so if other users have tips or find bugs please respond. Thx. JM === #!/bin/bash ## ## Start this script in am empty dir (just in case because it creates ## some files) and then open simulate.html in your favorite browser. DAY=10 GENERATIONS=60 PERIOD=$(($DAY * $GENERATIONS)) ## Create self refreshing html-page cat > simulation.html << EOF Sim The graph above shows a simulation of the number of spam messages (red bars) I receive each day. The short blue stacked bars are false negatives, meaning messages that were not filtered out by the spam filter, but manually identified as spam later on*. This simulation uses a 10 second "day" for the obvious reason that a real time simulation would take too long. The script as a whole is not useable in this form**. You'll have to split it up in a create, update and a graph piece. Set DAY to 86400 (seconds). The current number of generations would result in about 2 Months of data. Make sure you manually expand the shell variables or put them in a separate script and include that script in all the pieces. (Too include a script use this statement: ". /scriptpath/scriptname" without the quotes.) Btw: bash is needed, because of the calculations in the script. HTH for those people that are too lazy, like me, to read the excellent documentation that comes with rrdtool. You really should read it, but I posted this because I couldn't find any satisfactory examples of "ABSOLUTE" counters. --- * These are moved to a specific folder and then automatically collected by a script, which feeds them to the spam filter to learn from. The same script updates rrdb. ** This was first developed as a simulation to find out if every was working and put to use in the real world later. 24 Hour debug cycles aren't a lot of fun! EOF ## Create rrdb rrdtool create sim.rrd --start 'N' --step $DAY \ DS:sim:ABSOLUTE:$DAY:0:U \ DS:unk:ABSOLUTE:$DAY:0:U \ RRA:AVERAGE:0.1:1:$GENERATIONS ## Can update only once per second. ## Avoid writing too soon. sleep 1 while [ 1 ] ; do ## Feed simulated data to the rrdb MSG=$((RANDOM % 4)) if [ $MSG == 0 ] ; then UNK=$((RANDOM % 2)) else UNK=0 fi ## Allow pausing updates for a while, e.g. do: ## touch stop; sleep 10; rm -f stop if [ -f stop ] ; then sleep 10; fi rrdtool update sim.rrd N:$MSG:$UNK rrdtool graph tmpgraph.gif --start -$PERIOD \ DEF:vsim=sim.rrd:sim:AVERAGE \ DEF:vunk=sim.rrd:unk:AVERAGE \ CDEF:real=vsim,$DAY,* \ CDEF:unrl=vunk,$DAY,* \ AREA:real#FF:"Number of spam messages" \ STACK:unrl#FF:"Number of false negatives" ## To prevent reading the gif while it's being written: mv -f tmpgraph.gif simgraph.gif date ## Sleep one more second. Note that the whole loop ## will take more, because it's relatively resource ## intensive. sleep 1 done -- Unsubscribe mailto:[EMAIL PROTECTED] Helpmailto:[EMAIL PROTECTED] Archive http://www.ee.ethz.ch/~slist/rrd-users WebAdminhttp://www.ee.ethz.ch/~slist/lsg2.cgi
