I realized the code was not suitable for public release, so I went along
and updated it. Hopefully the current revision is usable. If not, let me
know.
The nfdump wrapper, I call flows, is now on github:
https://github.com/toringe/flows
- Tor
On Mon, May 20, 2013 at 9:55 PM, Ryan Harden <[email protected]> wrote:
> Thanks Tor,
>
> Do you mind sending me your bash wrapper?
>
> /Ryan
>
> Ryan Harden
> Senior Network Engineer
> University of Chicago - AS160
> P: 773-834-5441
>
>
>
>
> On May 20, 2013, at 2:50 PM, Tor Inge Skaar <[email protected]>
> wrote:
>
> > Ryan, I use a wrapper written i bash that allows for cluster processing
> through GNU Parallel. Nfdump together with parallel are installed on
> several machines which all have read access to the same repository of
> nfcapd files. The wrapper splits the query into multiple jobs, e.g. split
> on source router or timeslots, before instructing the cluster to process
> the individual jobs on the different cluster nodes. By using this simple
> program I've gained a significant performance increase.
> >
> > - Tor
> >
> >
> > On Mon, May 20, 2013 at 5:48 PM, Ryan Harden <[email protected]>
> wrote:
> > Has anyone written a method for distributing nfdump query processing to
> a compute cluster?
> > We have a compute cluster (condor based) that I'd like to utilize when
> processing nfdump data.
> >
> > >From what I understand, some sort of wrapper would be required that
> would instruct N number of compute nodes to map certain data files, process
> them, return values, then another process would aggregate the values based
> on the original request and generate the output the user was looking for.
> >
> > Perhaps I'm thinking about this all wrong?
> >
> > Thoughts? Comments?
> >
> > /Ryan
> >
> > Ryan Harden
> > Senior Network Engineer
> > University of Chicago - AS160
> > P: 773-834-5441
> >
> >
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > AlienVault Unified Security Management (USM) platform delivers complete
> > security visibility with the essential security capabilities. Easily and
> > efficiently configure, manage, and operate all of your security controls
> > from a single console and one unified framework. Download a free trial.
> > http://p.sf.net/sfu/alienvault_d2d
> > _______________________________________________
> > Nfdump-discuss mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/nfdump-discuss
> >
>
>
------------------------------------------------------------------------------
Try New Relic Now & We'll Send You this Cool Shirt
New Relic is the only SaaS-based application performance monitoring service
that delivers powerful full stack analytics. Optimize and monitor your
browser, app, & servers with just a few lines of code. Try New Relic
and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_may
_______________________________________________
Nfdump-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nfdump-discuss
