I'm not familiar with the JKFlow/flowviewer config, but I am familiar with maintaining tens of thousands of RRD files. The system I/O burden can be tremendous. In a high-end SAN environment, I've seen RRD updates spike to 5,000+ IOPS. Since the normal hard disk that most of us use can only handle about 100-200 IOPS, the result is blocking updates and a stuttering kernel.
RRDtool 1.4 to the rescue -- it includes rrdcached, a caching daemon that batches RRD updates and commits them later on. The daemon is supported with all existing code and perl modules by simply setting an environment variable. It does a very nice job of freeing your code from RRD blocks and spreading the RRD load to reduce the IOPS. One other note -- as you think about automation, remember that netflow v5 from routers includes subnet information. You might be able to use that to build configuration files. -Craig On Sat, 15 Oct 2011, David Faught wrote: > I am currently using flow-tools 0.68, FlowScan 1.06 with JKFlow 3.5.2, > and FlowViewer 3.3.1 and all the NetFlow data is version 5. The > server processing this is currently receiving between 300 and 900 > flows/second, or between 150 and 450 kilobits/second of flow data, > although this may as much as double over the next couple of years. > > I am thinking of trying to use either FlowTracker or the JKFlow > FlowScan reporting module to set up about 7000 separate individual > tracking entries which are in pairs and then combining the pairs in > 3500 group tracking entries. I will have to come up with some > automated way to build these entries based on a spreadsheet, but that > part doesn't seem too hard to me. > > So the question is, is this reasonable? Do you have any idea what > kind of resources the server would need to process and store this > data? And maybe have some room left to actually browse them? > > Thanks for any guidance you can provide. > > Regards, > Dave Faught > _______________________________________________ > Flow-tools mailing list > [email protected] > http://mailman.splintered.net/mailman/listinfo/flow-tools > _______________________________________________ Flow-tools mailing list [email protected] http://mailman.splintered.net/mailman/listinfo/flow-tools
