I'm using nested directories. Here is what I use (I edit the files to remove sensitive information, I may have introduced some syntax errors):
####### Flow capture (I'm saving 16 days worth or data ):
flow-capture -w /u01/data/netflow/ft/router-name \
0/192.168.0.1/2055 \
-S 5 \
-V 5 \
-e 4608 \
-n 287 \
-N 3 \
-R /u01/bin/netflow/linkme \
-p /var/run/flow-capture.pid
####### Linkme script (based on the one from dynamicnetworks):
#
# Script to create links of Flow Files so that flowscan, CUFlow, and
# ResNetFlow don't delete the actual file.
#
# Called by flow-capture (flow-tools)
#
use strict;
## Constants
# Location of NetFlow Files
my($ROUTER_BASE) = "/u01/data/netflow/ft/router-name";
# Base location to put links for flowscan
my($FLOW_SCAN_BASE) = "/u01/data/netflow"; # data
## Variables
my($fullPathToFile);
my($fileName);
# Get full path to filename from argument
$fullPathToFile = $ARGV[0];
# Strip off path to just get file name
if ($fullPathToFile =~ /.*[\/]*(ft-v05[^\/]*$)/) {
$fileName = $1;
} else {
print "Must specify file\n";
exit 1;
}
# Create the symbolic link from netflow file to flow-scan directory
unless ( symlink("$ROUTER_BASE/$fullPathToFile","$FLOW_SCAN_BASE/$fileName")
) {
print "Unable to create symbolic link: $FLOW_SCAN_BASE/$fileName\n";
exit 1;
}
###### And the script to run a report:
#!/bin/sh
#
# run_adhoc_report.sh - Run flow-reports on flow files
#
# Constants
FLOW_TOOLS_PATH="/usr/bin/"
# Begin
echo "Starting Adhoc Report Run"
START_TIME="October 25, 2006 00:00:00"
echo -n "Processing data from time: ${START_TIME}"
END_TIME="October 26, 2006 23:59:59"
echo " to: ${END_TIME}"
echo "Starting report"
$FLOW_TOOLS_PATH/flow-cat \
-p \
-t "${START_TIME}" \
-T "${END_TIME}" \
/u01/data/netflow/ft/router-name
| $FLOW_TOOLS_PATH/flow-report \
-s /u01/bin/netflow/reports/resnet.rpt \
-S resnet
RC=$?
if [ "$RC" -eq 0 ]; then
echo "Report Completed."
else
echo "Report Failed."
fi
I hope this helps.
- Neil
--
Neil Johnson
Telecommunications and Network Services
The University of Iowa
319 384-0938 (Work)
319 330-2235 (Cell)
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of jay alvarez
Sent: Friday, December 22, 2006 7:35 PM
To: Ed Ravin
Cc: [email protected]
Subject: [Flow-tools] linkme script modification for nested date-
basedstorage (Re: Trouble processing too many flows)
----- Original Message ----
From: Ed Ravin <[EMAIL PROTECTED]>
To: jay alvarez <[EMAIL PROTECTED]>
Cc: [email protected]
Sent: Saturday, December 23, 2006 2:33:26 AM
Subject: Re: [Flow-tools] Trouble processing too many flows (8640 5-minute
flow in a month)
> As the other poster suggested, you need to switch to a date-based
> directory format. See the flow-capture man page for the details -
> it will do it automatically. Then you can do things like
>
> flow-cat /var/netflow/ft/2006/2006-12/
>
> To get all of December 2006 - flow-cat is smart enough to follow
> the directory tree recursively.
I got the script below from:
http://www.dynamicnetworks.us/netflow/5-configure-flow-tools.html
#!/usr/bin/perl
$base = "/var/netflow/ft";
if ($ARGV[0] =~ /.*[\/]*(ft-v05[^\/]*$)/) { $fileName = $1; } else {
print "Must specify file\n"; exit 1; } unless (
symlink("$base/all/$fileName","$base/$fileName") ) { print "Unable to
create symbolic link: $base/$fileName\n"; exit 1; }
As far as I can understand, this is used to avoid having flowscan to delete
the each flows after processing it. It instead deletes the symbolic link.
Right now all my flows are in a single directory, and I want to implement
nesting level as you have suggested. My problem is i'm guessing that the
script above will no longer work. Are you using the a linkme script for
nesting storage configuratoin? Can you show it to me? :-)
The man page for nesting flow-captured flows looks like this:
-3 YYYY/YYYY-MM/YYYY-MM-DD/flow-file
-2 YYYY-MM/YYYY-MM-DD/flow-file
-1 YYYY-MM-DD/flow-file
0 flow-file
1 YYYY/flow-file
2 YYYY/YYYY-MM/flow-file
3 YYYY/YYYY-MM/YYYY-MM-DD/flow-file
I guess I need to use "2" or "3" perhaps, and I need to create 2 linkme
scripts to support both, only that I'm not good in this. Btw, what's the
difference between -3 and 3 ?
> Blech. You can do this in a much simpler fashion using find and xargs:
> find /var/netflow/ft/all/ -type f -name "vt-v05.2006-12-*" -print |
> xargs flow-cat > dec2006.flow
My problem is that after running my looping script or xargs which you have
suggested and writing the output to a single flow file, I got an error when
running flow-cat:
flow-cat: fstat(dec0): Value too large for defined data type
It seems like the file is too big (20G) for it to process at once..
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ Flow-tools mailing list [EMAIL PROTECTED] http://mailman.splintered.net/mailman/listinfo/flow-tools
