> -----Original Message-----
> From: Peter Stamfest [mailto:[EMAIL PROTECTED]
> Sent: eai u?e 03 ioo 2003 21:56
> To: Sasha Mikheev
> Cc: [EMAIL PROTECTED]
> Subject: Re: [rrd-developers] Improving RRD tool scalability
>
>
>
> Hi,
>
> Just a quick note: I am currently writing a tool to do
> high-volume data
> gathering using SNMP with the possibility to store data to
> RRDs. It is
> multithreaded and it can actually write to multiple RRDs in
> parallel (if
> this turns out to work for 17k RRDs remains to be tested -
> disk seek time
> and concurrency might become a problem here.
seek time is the _major_ issue here. You will get into the disk bottleneck
before there is a need to move to high performance SNMP.
I thought it would be the other way around and even started
working on high speed SNMP client. This is a bit off topic but are you
planning to use ucd-snmp
in that collector of yours ? My idea was to make single threaded snmp client
which has all snmp
packets prebuilt and just pumps it out then uses poll to get all responses
back. In theory i will
be limited by only by my network pipe and how fast device can process snmp
request.
>
> Does throwing in more RAM improve the situation? The OS might
> cache much
> more data then. Given 20K RRDs, each sized 100KB this would
> mean ~ 2GB of
> RAM to keep everything in RAM.
Our rrds are 4M sized (lots of high resolution data for drill downs) so
keeping everything in RAM is not an option.
>
> Isn't this on the TODO list (make the thin mmap-ed), but OTOH
> this might
> still not speed up things, as disk seeks are essentially the same...
It will make things much worse for update and i think it will not matter
that
much for rrd_fetch(). In linux msync is working by 4K pages while write
operation uses 512bytes
sector. So write() will have to transfer only 1/8 of data to disk.
--
Unsubscribe mailto:[EMAIL PROTECTED]
Help mailto:[EMAIL PROTECTED]
Archive http://www.ee.ethz.ch/~slist/rrd-developers
WebAdmin http://www.ee.ethz.ch/~slist/lsg2.cgi
