Rajeev,
Thanks for your reply. What you have published is very close to what I was
looking for.
I still may implement this as a database so I can build some custom qeries against the
data using
SQL stmts. The only thing I am concerned about is the potential size of the database.
We are
currently on break, so our students are gone. MRTG reports that we are using 1/3 of
the bandwidth
we normally use. We are still generating around to 1 million lines per day in our
exported log
file (this is internet access only, no VPN, which is coming). So, I think we are
better off
splitting this into DBM files than having a database with 100 million rows per month.
Thanks for your help,
Brad
Rajeev Kumar wrote:
> Brad,
> you can deny DNS lookup, by providing -n option to fw logexport. But most of
>the time you
> need Names instead of IP addresses in your final report so you need DNS lookups.
>This will
> become very slow for some lookups and overall increase the time to process logs.
>One to
> implement this use some sort of caching. (like DNS server (named) also does, but
>that is
> more complicated. and I won't go there I sometimes back wrote the log analyzer in
>perl and
> there I use the DBM files to store DNS resolved names:
>
> So everyday you parse your log files and resolve IP addresses and store the resolved
>IP
> address in DBM hash. Next time you encounter the same IP address you simply lookup
>locally
> stored DBM database, instead of DNS server and wait for long time. So theoritically
>you
> keep these resolved names forever in your disk as DBM hash, but what if somebody
>change
> the name for IP addresses. So what you can do specify kind of TTL(Time to live
>value) for
> the DBM hash itself and after TTL expired, a new DBM hash would be created and old
>one
> deleted. So let's say if you keep these TTL (1 month, assuming not many IP addresses
> (Names) changed during this period,) you won't be resolving same IP addresses each
>time
> you parse log files. This greatly enhances the conversion speed.
>
> If you are intrested in this, just look through perl code and see how I implemented
>this.
> http://www.geocities.com/rxknh/pub/fwlogstat/index.html
>
> Rajeev
>
> Brad Grant wrote:
> >
> > Is there a more efficient way to export logs than with fw logexport.
> > Has anyone developed a way to use dbi and a database to perform lookups
> > against a database. This is how I parse my apache log files for
> > archiving . We are running VPN-1 4.1 on an Ultra 10, and have a full T3
> >
> > to the internet. I would like to export my logfiles to resolved text
> > and compress them each day for permanant archive, but it does not seem
> > logistically possible due to the time it takes to export the log files.
> > Does fw logexport really perform an nslookup for each entry as it
> > appears? Is the data structure of the binary format logfile published?
> > Thanks in advance,
> >
> > Brad Grant
> > Network Manager
> > Savannah College of Art and Design
> > Savannah, GA
> > (912) 525-6147
> >
> > ================================================================================
> > To unsubscribe from this mailing list, please see the instructions at
> > http://www.checkpoint.com/services/mailing.html
> > ================================================================================
>
> --
> #########################################################################
> (Titanic creators used Linux to simulate the sinking of the great ship)
> #########################################################################
> Rajeev Kumar ([EMAIL PROTECTED])
> Fluent Inc. 10, Cavendish Court, Lebanon NH-03766
> -------------------------------------------------------------------------
> Phone :: (603)-643-2600 x 349 Fax :: (603)-643-3967
> Web:: http://www.fluent.com
> #########################################################################
================================================================================
To unsubscribe from this mailing list, please see the instructions at
http://www.checkpoint.com/services/mailing.html
================================================================================
