Continuous packet capture

[Jonathan S. Polacheck]

Perhaps this is a case of "premature optimization".  I am working toward a
continuous packet capture application (as in Infinistream, Gigastor,
NetVCR, etc).  So far, I have two perl scripts.  One takes Net::Pcap output
and Hexdumps it to a fifo "file".  The other reads the fifo and inserts the
hexdump into a database table.

This is all well and good, but when I tried to dump the pcap output
directly into the fifo and convert to hex as the fifo was read things got
ugly.  So now I am considering rewriting the whole thing in POE.

I want some kind of buffer between pcap and the database inserts to reduce
the chance of dropped packets.  It seems to me that the "process" that has
the pcap session should have a higher priority than the db insert
"process".

Is POE the way to go?  Are there existing examples of POE code that does
something similar?

Thanks for your interest,

Jon Polacheck

Used mkfifo to create the named pipe.  Perl sees it as a disk file
(that I called qtfifo).  ENQUE.pl dumps packet hexdumps to the fifo.
DEQUE.pl reads lines from the fifo.  /^0000 / acts as the delimiter. mysql
compression worked with the standard OpenSuSE install, no recompiling or
other mucking about necessary.

Lines used for debugging marked as such.

ENQUE.pl

use Net::Pcap;
use Data::Hexdumper qw(hexdump);

$dev = "eth0";

# used a 50 packet cap file to make sure what came out matched what went in
#$dump = "ip.cap";
#$pcap = Net::Pcap::open_offline($dump, \$err) or die "Can't read
'$dump': $err\n";

# live, real-time feed
$pcap = Net::Pcap::open_live($dev, 1514, 1, 0, \$err);

Net::Pcap::loop($pcap, -1, \&process_pkt, ""); # <- subroutine call

sub process_pkt {
   open(QT, "> qtfifo");
# $_[2] is the third element of the default array "@_" which was created
# by the subroutine call "&process_pkt"
   my $pkt=$_[2];
   $results = hexdump( data => $pkt
       , number_format => 'C',
   );
   print QT $results;
   close(QT);
    $i++; # debug
    &stop_run if $i > 100; #debug
}

# all debug below
sub stop_run {
 print "stop_run\n";
 open(QT, "> qtfifo");
 print QT "\nx\n";
 close(QT);
 print "enque ended\n";
 exit;
}


DEQUE.pl

use Time::HiRes ( nanosleep );
use DBI;

$hostname="127.0.0.1";
$database="cpc";
$port="3306";

$dsn = "DBI:mysql:database=$database;host=$hostname;port=$port";

$dbh = DBI->connect($dsn,
   "root",
   "",
   {'RaiseError' => 1});

# call the Net::Packet collector script
system(q{perl ENQUE.pl&});

open(EQT, " < qtfifo");

$i = 0; # debug
$pc = 0; # debug

while(1) {
   $i++; # debug
   $line = readline(EQT);
   if ($line =~ /^0000 / ) {
       $dbh->do(qq{INSERT INTO cpc VALUES ( compress("$pkt"))});
       $pc++ if defined($pkt); # debug
       print "packet $pc:\n$pkt\n" if defined($pkt); # debug
       undef($pkt);
       $pkt .= $line;
   } else {
       $pkt .= $line;
       &theend if $pkt =~ /x/; # debug
   }
   nanosleep(1); # would not work without this!
}

# all debug below
sub theend {
   close(EQT);
   print "$i loops\ndeque ended\n";
   exit;
}

This generated a cap file that looks just fine in Wireshark.

mysql -Br -D cpc -e "select uncompress(packet) from cpc;" | text2pcap -
m_cap.cap