Until direct html output is supported with flow-report I imagine a
number of people have written their own parsers to convert the output
to HTML. I have hacked up one that creates familiar top 10 tables in
HTML format that appears to work for me. Others off-list indicated
that it might be useful for them to use the methods I did so I'll post
the details here. It seems there could be more examples of how others
are using flow-tools so hopefully this will help someone.
It's not very generic and may even have a couple of minor problems.
A pointer to something better or enhancements are of course welcome.
I hereby place any code below into the public domain.
Using the -R option to flow-capture, I run a script that, among other
things, creates a top TCP/UDP port usage report that is run like this
(where $1 is the recently rotated flow file passed to the script):
flow-cat $1 | flow-report -s report.conf -S top-ports
To create top 10 reports (change the records option to whatever is
appropriate for you) a tcp-egress.report looks like this:
include-filter filter.conf
stat-report tcp-dstport-egress-octets
type ip-destination-port
filter tcp-egress
output
format ascii
records 10
# scale xxx if you're getting sampled flows
path tcp-dstport-egress-octets.txt
fields -duration
sort +octets
options +totals
#
stat-report tcp-srcport-egress-flows
type ip-source-port
filter tcp-egress
output
format ascii
records 10
# scale xxx if you're getting sampled flows
path tcp-srcport-egress-octets.txt
fields -duration
sort +flows
options +totals
#
# ...add other tcp/udp definitions as necessary
#
stat-definition top-ports
report tcp-dstport-egress-octets
report tcp-srcport-egress-flows
#
# ...
Create the appropriate filter.conf file that sets up the needed
primitives and definitions for the filters in report.conf. For
example:
# ...
#
filter-primitive my-netblocks
type ip-address-prefix
permit 192.0.2.0/24
default deny
#
filter-primitive tcp
type ip-protocol
permit tcp
default deny
#
filter-definition tcp-egress
match ip-source-address my-netblocks
match ip-protocol tcp
#
# ...
For TCP/UDP reports you can create a bunch of reports by mixing the
combination of protocol (e.g. TCP or UDP), direction (e.g. ingress
or egress), port (e.g. source or destination) and rank type (e.g. flows,
octets, packets), which should keep network analysts busy and asking
for more.
Now that those definitions are out of the way, it is time for the text
to HTML conversion. I do this with perl and with the code below I make
some assumptions about what the output looks like. The code may need
to be tweaked if your reports are going to output something that the
script below doesn't expect (e.g. bps or other columns).
I ended up making a command line options necessary:
[ --type stat-report-type ] [ --rows max-rows ] [ --key sortfield ]
[ --title report-title ] [ --headfoot ]
--type type from stat-report-type (e.g. ip-source-port or just srcPort
for short)
--rows the number of rows to create in the HTML table
--key the sort field (e.g. flows, octets or packets)
--title <caption> text for the table
and --headfoot is a boolean option if present will add the HTML header
and footer tags. You might not enable this if you are taking the output
from this script and appending it to aggregate top ports file with other
tables. I'll show an example of this shortly.
In the meantime, here is the script:
#!/usr/bin/perl -wT
use strict;
$|=1;
### init variables to defaults
#
my $rows = 10; # maximum number of rows
my $title = "Flow Report"; # report title
my $key = "octets"; # sort on flows/octets/packets
my $type = "n/a"; # report type (e.g. srcPort, dstPort)
my $headfoot = 0; # do HTML header/footer tags?
### retrieve options
#
use Getopt::Long;
my $result = 0;
$result = GetOptions( "title=s" => \$title, # report title
"rows=i" => \$rows, # max no. of rows
"key=s" => \$key, # key (flows, packets, octets)
"type=s" => \$type, # type (e.g. srcPort, dstPort)
"headfoot" => \$headfoot # do HTML header/footer tags?
);
my $rank = 1; # current row/ranking in report
# doHeader
# HTML header
# if HTML header not disabled, so scripts can incorpriate into another page
if( $headfoot) {
print "<html>\n<body bgcolor=\"#ffffff\">\n<center>\n";
}
# table header
print "<table border=\"1\" cellspacing=\"0\" cellpadding=\"3\">\n";
print "<caption align=\"top\">Top $rows <b>$title</b></caption>\n";
### use variable to determine where to put highlighted column
my %highlight = ( "flows" => "",
"octets" => "",
"packets" => ""
);
### total flow, octet and packet counts
my %totals = ();
# highlight key table cell in green
$highlight{$key} = ' bgcolor="#90ee90"';
# printer column heading
print "<tr bgcolor=\"#ffffcc\"
align=\"center\"><td><b>rank</b></td><td><b>$type</b><td$highlight{'flows'}><b>flows<b></td><td$highlight{'octets'}><b>octets</b></td><td$highlight{'packets'}><b>packets</b></td></tr>\n";
# highlight key column in light blue
$highlight{$key} = ' bgcolor="#add8e6"';
while( defined(my $input = <>) ) {
chomp $input;
# found a totals line, use for % output
if( $input =~ /^# rec1: ignores,flows,octets,packets/ ) {
$input = <>; # get totals
( $totals{'flows'}, $totals{'octets'}, $totals{'packets'} ) =
$input =~ /^\d+,(\d+),(\d+),(\d+)(?:,\w+)*$/;
next;
}
if( $input =~ /^\s*(#.*)?$/) { next; } # skip comments or blank lines
my( $port, $flows, $octets, $packets) = ( "", "", "", "" );
( $port, $flows, $octets, $packets ) = split ( /\,/, $input, 4 );
# print a row: rank, port, flows (%), octets (%), packets (%)
printf( "<tr align=\"right\"><td>%d</td><td>%s</td>", $rank++, $port );
printf( "<td$highlight{'flows'}>%s \(%3.2f%%\)</td>", commify($flows),
($flows/$totals{'flows'})*100 );
printf( "<td$highlight{'octets'}>%s \(%3.2f%%\)</td>", commify($octets),
($octets/$totals{'octets'})*100 );
printf( "<td$highlight{'packets'}>%s \(%3.2f%%\)</td></tr>\n",
commify($packets), ($packets/$totals{'packets'})*100 );
}
# table footer
print "</table>\n";
# html footer
if ($headfoot) {
print "</center>\n</body>\n</html>\n";
}
exit(0); ### EXIT SUCCESS
### SUB ROUTINES
#
# from Perl Cookbook
sub commify {
my $text = reverse $_[0];
$text =~ s/(\d\d\d)(?=\d)(?!\d*\.)/$1,/g;
return scalar reverse $text;
}
### EOF
What I do is run this script against multiple port report text files
without the --headfoot option and concatenate each output together into
one top ports report page. Doing something like this:
cat > ${HTMLREPORT} << EOF
<html>
<body>
`cat ${HTMLHEADER}`
<center>
<table border="0" cellspacing="0" cellpadding="1">
<caption align="top"For previous 5 minutes starting at
<b>${FLOWDATE}.${FLOWTIME}</b></caption>
<tr>
<td>
`${TXT2HTML} --title="TCP source ports egress by octets" --type=srcPort --rows=10
--key=octets tcp-srcport-egress-octets.txt`
</td>
<td>
<! ...>
<! also by flows and packets>
</td>
</tr>
<! ...>
<! other reports and HTML tags here as appropriate>
</table>
</center>
`cat ${HTMLFOOTER}`
EOF
${HTMLREPORT} is the resulting output HTML file containing all the
tables concatenated together. ${HTMLHEADER} points to a file that
includes any introdoctury text/html that I want to appear at the top
of the page. ${HTMLFOOTER} is for anything I want to appear at the
end of the page. ${TXT2HTML} should point to the Perl script I
included above.
One last tip. You may have noticed the ${FLOWDATE}.${FLOWTIME}
variables that get put into the outer page table. I just pull
out that info from the rotated flow file name. This will vary
depending on how flow-capture is storing your files (-N nesting
level parameter), but if we assume the default, the parameter
passed to the script will look something take this format if you
are using NetFlow version 5 format:
ft-v05.YYYY-MM-DD.HHMMSS-ZONE
where
YYYY is the year
MM is the month
DD is the day
HH is the hour in 24-hour format
MM is the minutes past the hour
SS is the seconds past the minute
ZONE is the timezone offset to UTC in 24-hour format
Grabbing the flow date and time the would be accomplished like this,
where $1 is the flow filename passed to the script:
FLOWTIMESTAMP=`expr substr $1 8 17`
Using all those tips you can store the top ports HTML reports off in
their own directories nested by date and hour if necessary. The end
of the script can update a link to the most recent. for example:
ln -sf ${HTMLREP} /path/to/web/directory/top-ports.html
When all is said and done, if I typed in all of the above correctly and
your setup is similar to mine, you might end up with something that looks
a little like this:
<http://aharp.ittns.northwestern.edu/tmp/top-ports-sample.html>
John
_______________________________________________
Flow-tools mailing list
[EMAIL PROTECTED]
http://mailman.splintered.net/mailman/listinfo/flow-tools
