On Thu, Sep 09, 2004 at 11:41:26PM +0300, Vlad GALU wrote: V> This made me raise my eyebrow. I wrote a small tool that we use in V> production at RDS: http://freshmeat.net/projects/glflow. The way I V> designed it, it is supposed to clean up the flow tree once in a while V> and remove 'old' flows (that haven't had any packet matching them in the V> last X seconds). The problem is that I currently have about 400-500k V> active flows on a 700Mbps link. Every 10 seconds the software removes V> about 100-200k of them in no more than 0.2-0.3 seconds. Of course, I V> couldn't possibly send them over a socket somewhere else at that speed,, V> and chose to open a tempfile, mmap() it, write the expired flows to the V> buffer. When the buffer exceeds a programatically chosen number of V> packets, it is msync()-ed, munmap()-ed and a new file is open.
If you remove 100-200k of flows every 10 seconds, this means you have 10 - 20 kpps of worm traffic or more! Impressive. 200k flows expired in 10 seconds is 666 export dgrams per second, 7 times more than I usually have in my testbed. Not sure, that ng_netflow + ng_ksocket can do this. Do you collect all this traffic on a single router? V> Do you accidentally have a better storage model ? I've been trying to V> dump these binary files to SQL but for a 42 meg binary log the necessary V> SQL storage went to about 150 megs, which is a bit over reasonable, V> considering the fact that the software dumps a binary file every 5 to 10 V> seconds. Dumping to SQL is a bad idea. I have tried it, too :) V> P.S. I haven't yet tried to aggregate the flows between reading them V> from the binary file and inserting the data into SQL. I thought it would V> take too much time to be able to keep up with the newly created dumps. This is how many people do. However I don't know details. -- Totus tuus, Glebius. GLEBIUS-RIPN GLEB-RIPE _______________________________________________ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"