----- Original Message ---- From: Ed Ravin <[EMAIL PROTECTED]> To: jay alvarez <[EMAIL PROTECTED]> Cc: [email protected] Sent: Saturday, December 23, 2006 2:33:26 AM Subject: Re: [Flow-tools] Trouble processing too many flows (8640 5-minute flow in a month)
> As the other poster suggested, you need to switch to a date-based > directory format. See the flow-capture man page for the details - > it will do it automatically. Then you can do things like > > flow-cat /var/netflow/ft/2006/2006-12/ > > To get all of December 2006 - flow-cat is smart enough to follow > the directory tree recursively. I got the script below from: http://www.dynamicnetworks.us/netflow/5-configure-flow-tools.html #!/usr/bin/perl $base = "/var/netflow/ft"; if ($ARGV[0] =~ /.*[\/]*(ft-v05[^\/]*$)/) { $fileName = $1; } else { print "Must specify file\n"; exit 1; } unless ( symlink("$base/all/$fileName","$base/$fileName") ) { print "Unable to create symbolic link: $base/$fileName\n"; exit 1; } As far as I can understand, this is used to avoid having flowscan to delete the each flows after processing it. It instead deletes the symbolic link. Right now all my flows are in a single directory, and I want to implement nesting level as you have suggested. My problem is i'm guessing that the script above will no longer work. Are you using the a linkme script for nesting storage configuratoin? Can you show it to me? :-) The man page for nesting flow-captured flows looks like this: -3 YYYY/YYYY-MM/YYYY-MM-DD/flow-file -2 YYYY-MM/YYYY-MM-DD/flow-file -1 YYYY-MM-DD/flow-file 0 flow-file 1 YYYY/flow-file 2 YYYY/YYYY-MM/flow-file 3 YYYY/YYYY-MM/YYYY-MM-DD/flow-file I guess I need to use "2" or "3" perhaps, and I need to create 2 linkme scripts to support both, only that I'm not good in this. Btw, what's the difference between -3 and 3 ? > Blech. You can do this in a much simpler fashion using find and xargs: > find /var/netflow/ft/all/ -type f -name "vt-v05.2006-12-*" -print | > xargs flow-cat > dec2006.flow My problem is that after running my looping script or xargs which you have suggested and writing the output to a single flow file, I got an error when running flow-cat: flow-cat: fstat(dec0): Value too large for defined data type It seems like the file is too big (20G) for it to process at once.. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
_______________________________________________ Flow-tools mailing list [EMAIL PROTECTED] http://mailman.splintered.net/mailman/listinfo/flow-tools
