Re: [rrd-users] api for iterating over single rows in a rrd file
Hi Plamen, Yesterday Plamen Dimitrov wrote: Hi rrdtool users! As part of my google summer of code project with Ganglia I'm developing an R package that imports the values from an RRD file into vectors in R (without exporting to csv, xml or other intermediate format first). I'm using the rrdfetch api to do this. Here is a working prototype: https://github.com/pldimitrov/Rrd Wnile this seems to work fairly well, it struck me that in a scenario when I'm only interested in reading one row at a time (e.g. to compare values from many RRDs simultaneously) , the rrdfetch code would need to go through all error/sanity checks, find the RRA we want and seek to the desired location in the file at each iteration. I know the use of the internal rrd_read, rrd_seek, rrd_open, etc... functions is not encouraged so I'm wondering what might be a good solution. Ideally, it would be useful to have something that iterates one row at a time, reads and caches the data. Does anything like this already exist? Would you agree it makes sense to have this in addition to rrdfetch? no there is no iterator ... if you do see a memory problem with reading the whole file, you may want to split your reading into of say 1 rows cheers tobi Regards, Plamen -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland www.oetiker.ch t...@oetiker.ch +41 62 775 9902 ___ rrd-users mailing list rrd-users@lists.oetiker.ch https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Re: [rrd-users] api for iterating over single rows in a rrd file
On 04/07/14 09:57, Tobias Oetiker wrote: Hi Plamen, Yesterday Plamen Dimitrov wrote: Hi rrdtool users! As part of my google summer of code project with Ganglia I'm developing an R package that imports the values from an RRD file into vectors in R (without exporting to csv, xml or other intermediate format first). I'm using the rrdfetch api to do this. Here is a working prototype: https://github.com/pldimitrov/Rrd Wnile this seems to work fairly well, it struck me that in a scenario when I'm only interested in reading one row at a time (e.g. to compare values from many RRDs simultaneously) , the rrdfetch code would need to go through all error/sanity checks, find the RRA we want and seek to the desired location in the file at each iteration. I know the use of the internal rrd_read, rrd_seek, rrd_open, etc... functions is not encouraged so I'm wondering what might be a good solution. Ideally, it would be useful to have something that iterates one row at a time, reads and caches the data. Does anything like this already exist? Would you agree it makes sense to have this in addition to rrdfetch? no there is no iterator ... if you do see a memory problem with reading the whole file, you may want to split your reading into of say 1 rows Hi Tobi, What do you think of the other half of the question, simultaneously reading a row from all the RRDs? Has anything like this been discussed before, has anybody else expressed interest in that? Regards, Daniel ___ rrd-users mailing list rrd-users@lists.oetiker.ch https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Re: [rrd-users] api for iterating over single rows in a rrd file
Hi Daniel, Today Daniel Pocock wrote: On 04/07/14 09:57, Tobias Oetiker wrote: Hi Plamen, Yesterday Plamen Dimitrov wrote: Hi rrdtool users! As part of my google summer of code project with Ganglia I'm developing an R package that imports the values from an RRD file into vectors in R (without exporting to csv, xml or other intermediate format first). I'm using the rrdfetch api to do this. Here is a working prototype: https://github.com/pldimitrov/Rrd Wnile this seems to work fairly well, it struck me that in a scenario when I'm only interested in reading one row at a time (e.g. to compare values from many RRDs simultaneously) , the rrdfetch code would need to go through all error/sanity checks, find the RRA we want and seek to the desired location in the file at each iteration. I know the use of the internal rrd_read, rrd_seek, rrd_open, etc... functions is not encouraged so I'm wondering what might be a good solution. Ideally, it would be useful to have something that iterates one row at a time, reads and caches the data. Does anything like this already exist? Would you agree it makes sense to have this in addition to rrdfetch? no there is no iterator ... if you do see a memory problem with reading the whole file, you may want to split your reading into of say 1 rows Hi Tobi, What do you think of the other half of the question, simultaneously reading a row from all the RRDs? Has anything like this been discussed before, has anybody else expressed interest in that? Neither ... my question is, what is the purpose of such an endevour on the API level. cheer tobi Regards, Daniel -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland www.oetiker.ch t...@oetiker.ch +41 62 775 9902 ___ rrd-users mailing list rrd-users@lists.oetiker.ch https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Re: [rrd-users] api for iterating over single rows in a rrd file
On 04/07/14 11:09, Tobias Oetiker wrote: Hi Daniel, Today Daniel Pocock wrote: On 04/07/14 09:57, Tobias Oetiker wrote: Hi Plamen, Yesterday Plamen Dimitrov wrote: Hi rrdtool users! As part of my google summer of code project with Ganglia I'm developing an R package that imports the values from an RRD file into vectors in R (without exporting to csv, xml or other intermediate format first). I'm using the rrdfetch api to do this. Here is a working prototype: https://github.com/pldimitrov/Rrd Wnile this seems to work fairly well, it struck me that in a scenario when I'm only interested in reading one row at a time (e.g. to compare values from many RRDs simultaneously) , the rrdfetch code would need to go through all error/sanity checks, find the RRA we want and seek to the desired location in the file at each iteration. I know the use of the internal rrd_read, rrd_seek, rrd_open, etc... functions is not encouraged so I'm wondering what might be a good solution. Ideally, it would be useful to have something that iterates one row at a time, reads and caches the data. Does anything like this already exist? Would you agree it makes sense to have this in addition to rrdfetch? no there is no iterator ... if you do see a memory problem with reading the whole file, you may want to split your reading into of say 1 rows Hi Tobi, What do you think of the other half of the question, simultaneously reading a row from all the RRDs? Has anything like this been discussed before, has anybody else expressed interest in that? Neither ... my question is, what is the purpose of such an endevour on the API level. From a technical perspective, if rrdtool provided such an API: - it would make programming easier for people accessing data in this way - it could be optimized for efficiency (e.g. the IO subsystem would be reading 4KiB of data from each file at a time, the iterator would then return values from that buffer and the IO subsystem would only do more reads when the iterator advances over a page boundary) From a user perspective, it would enable people to look across all their RRDs, e.g. to answer questions like which CPUs were over 90% utilized at any time between 09:00 and 10:00 ___ rrd-users mailing list rrd-users@lists.oetiker.ch https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Re: [rrd-users] api for iterating over single rows in a rrd file
Today Daniel Pocock wrote: Neither ... my question is, what is the purpose of such an endevour on the API level. From a technical perspective, if rrdtool provided such an API: - it would make programming easier for people accessing data in this way - it could be optimized for efficiency (e.g. the IO subsystem would be reading 4KiB of data from each file at a time, the iterator would then return values from that buffer and the IO subsystem would only do more reads when the iterator advances over a page boundary) From a user perspective, it would enable people to look across all their RRDs, e.g. to answer questions like which CPUs were over 90% utilized at any time between 09:00 and 10:00 rrdtool 2.x will provide cooler apis ;) cheers tobi -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland www.oetiker.ch t...@oetiker.ch +41 62 775 9902 ___ rrd-users mailing list rrd-users@lists.oetiker.ch https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Re: [rrd-users] api for iterating over single rows in a rrd file
Daniel Pocock dan...@pocock.pro wrote: From a technical perspective, if rrdtool provided such an API: - it would make programming easier for people accessing data in this way - it could be optimized for efficiency (e.g. the IO subsystem would be reading 4KiB of data from each file at a time, the iterator would then return values from that buffer and the IO subsystem would only do more reads when the iterator advances over a page boundary) From a user perspective, it would enable people to look across all their RRDs, e.g. to answer questions like which CPUs were over 90% utilized at any time between 09:00 and 10:00 I can see the point of being able to iterate through the data (open, seek, get row, get row, ...) - I'm sure there are probably a few applications where that would be useful. I don't see where having RRD handle multiple files would be any better than the program itself opening multiple files. I suspect it would be a massively complicated task to build generic open and merge multiple files function given the variety of ways they can be configured. Better, IMO, for your program to handle that - you are in a position to know what RRDs you've set up, and overall it's probably no harder than finding a reliable way to tell RRD what you want. ___ rrd-users mailing list rrd-users@lists.oetiker.ch https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Re: [rrd-users] api for iterating over single rows in a rrd file
From a user perspective, it would enable people to look across all their RRDs, e.g. to answer questions like which CPUs were over 90% utilized at any time between 09:00 and 10:00 how is your solution better than for rrdfile in set of relevant files do cpumax=`rrdtool graph dummy.png --start starttime --end endtime DEF:cpuutil= VDEF:cpumax=cpuutil,MAXIMUM PRINT:cpumax:%lf` if cpumax90 then print rrdfile done (i am not judging your solution, i am asking!) ___ rrd-users mailing list rrd-users@lists.oetiker.ch https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
[rrd-users] api for iterating over single rows in a rrd file
Hi rrdtool users! As part of my google summer of code project with Ganglia I'm developing an R package that imports the values from an RRD file into vectors in R (without exporting to csv, xml or other intermediate format first). I'm using the rrdfetch api to do this. Here is a working prototype: https://github.com/pldimitrov/Rrd Wnile this seems to work fairly well, it struck me that in a scenario when I'm only interested in reading one row at a time (e.g. to compare values from many RRDs simultaneously) , the rrdfetch code would need to go through all error/sanity checks, find the RRA we want and seek to the desired location in the file at each iteration. I know the use of the internal rrd_read, rrd_seek, rrd_open, etc... functions is not encouraged so I'm wondering what might be a good solution. Ideally, it would be useful to have something that iterates one row at a time, reads and caches the data. Does anything like this already exist? Would you agree it makes sense to have this in addition to rrdfetch? Regards, Plamen ___ rrd-users mailing list rrd-users@lists.oetiker.ch https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
