On Sep 30, 2013, at 12:37 PM, Les Mikesell <lesmikes...@gmail.com> wrote:
> On Mon, Sep 30, 2013 at 12:19 PM, Fabrice Bacchella
> <fbacche...@spamcop.net> wrote:
>>
>>>> I kept FILE as the backend, I agree that NIO is too heavy on memory.
>>>>
>>>>
>>>> What symptoms do you have of NIO weightiness? I've struggled with NIO,
>>>> too,
>>>> and just wonder what you are seeing. Perhaps I missed something in the
>>>> scroll back on this thread, sorry.
>>>>
>>>>
>>>> I think the problems is in the read ahead policy on linux, that load page
>>>> in
>>>> chunk of mega bytes, just for a simple update of a few bytes. When you have
>>>> a lot of rrd collected, it files the memory in a blink and you get too much
>>>> swap in/swap out for real data. Switching from NIO to FILE decreased
>>>> dramaticaly my IO load.
>>>
>>> Whether that is good or bad should depend on how much RAM you have
>>> available for buffering - and RAM is pretty cheap these days. How
>>> much does your system have besides what you give to the JVM?
>>
>> I have 42 GB of rrd in 12154 files, all of them are updated every 5 minutes.
>> I think there is better usage of so much data than caching useless data.
>>
>
> Making the system use swap to emulate RAM for disk buffering is
> clearly a bad idea. But, on a 2nd thought, I don't see how using
> FILE can avoid any overhead added by linux filesystem access - NIO
> must be (over)doing something to hold it in memory.
I've got 162GB of rrd on a host with 72GB of ram, and update the files every 5
minutes also. The system has a lot of I/O going on, and its able to keep up
writing the data out. The point is that every time you want to update a single
file, there is too much read overhead associated with opening the file just to
write out a single update and close the file.
I've wondered if you were to implement something like storeByGroup, but for all
metrics under the node (storeByNode?) and have it pre-allocate a specific size
whether you could take advantage then of being able to keep the RRDs open (and
thus limit the read hits). So you would see something like:
share/rrd/1/
files.properties
rrd-types.properties
base1-chunk1.rrd
base2-chunk1.rrd
rrd-types.properties would map the <rrd><rra>…</rrd> blocks to the baseX
prefixes.
files.properties would map the the path & datasources to the specific chunk
files. The chunk files would pre-allocate 50-100 datasources each.
If you were able to store more datasources into these files, then the rrd
strategy wouldn't need to open/close the files for each update. It could take
advantage of the RrdDbPool and keep the files open.
Ron
------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ
opennms-devel mailing list
To *unsubscribe* or change your subscription options, see the bottom of this
page:
https://lists.sourceforge.net/lists/listinfo/opennms-devel
