On Apr 7, 2010, at 1:14 PM, Peter Haag wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>
> On 7/4/10 6:58 PM, ckotil wrote:
>> Yes, but my script performs a `mv nfcapd.YYYYMMDDhhmm nfcapd.hhmm`. How
>> would nfsen know to look for nfcapd.hhmm ?
>
> ?? why do you move files?? Do not touch/remove/move the original file. You
> may process the flows, but do not alter the
> file in any way.
>
The reason for this thread. The evolution of the thread lead me to rename the
files.
Because I want to get statistics from multiple hosts AND times. Nfdump does not
handle wildcards gracefully enough to do this currently. With the files renamed
to nfcapd.hhmm it became possible for nfdump to process multiple hosts and
times.
I have opted to write a script that will accept a set of hosts and times, and
then use nfdump -R to process the files.
> - Peter
>
>> Im running nfcapd-1.6.1 now.
>>
>> --Chad
>>
>> On Apr 7, 2010, at 12:53 PM, Peter Haag wrote:
>>
>>
>>
>> On 6/4/10 8:47 PM, ckotil wrote:
>>>>> I am using -x to rename the files to nfcapd.hhmm. Nfsen looks for files
>>>>> named nfcapd.YYYYMMDDhhmm. I think that's why Nfsen is no longer
>>>>> populating the RRDs, cannot process netflow via the webinterface, and
>>>>> nfexpire isn't purging the old flows..
>>>>> If I remove the optional argument to move the flows, everything starts
>>>>> working again.
>>>>>
>>
>> This _does_ work. As already mentioned, NfSen has no clue about this
>> additional argument. Expiring the profile is fully
>> independant from optarg.
>>
>>
>>>>> --Chad
>>>>>
>>>>>
>>>>> On Apr 6, 2010, at 4:23 AM, Peter Haag wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 4/2/10 22:06, ckotil wrote:
>>>>>>>> I was able to rename the nfcapd files by using a perl script.
>>>>>>>>
>>>>>>>> 'optarg' => '-x "perl /usr/local/bin/nfcapd-rewrite.pl %d/%f %d
>>>>>>>> %f'"},
>>
>> There is btw a mistake with quotes:
>>
>> 'optarg' => "-x 'perl /usr/local/bin/nfcapd-rewrite.pl %d/%f %d %f'"}
>>
>>
>> - Peter
>>
>>>>>>>>
>>>>>>>> However, This makes Nfsen sad. Nfsen loses track of the files. The rrd
>>>>>>>> graphs have holes in them. Nfexpire does not purge flows, and the
>>>>>>>> Nfsen frontend is unable to process stats on the flows.
>>>>>
>>>>> There is no reason, why NfSen shouldn't like this additional -x. It's
>>>>> executed in the additional nfcapd/launcher
>>>>> process, which NfSen has no knowledge about it's existance anyway.
>>>>> So there must be another reason for that. What do the logfiles say??
>>>>>
>>>>> - Peter
>>>>>
>>>>>>>>
>>>>>>>> It looks like we need a better way to rename nfcapd files. Feature
>>>>>>>> request?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> --Chad
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Apr 1, 2010, at 2:02 PM, ckotil wrote:
>>>>>>>>
>>>>>>>>> As long as NfSen can find the renamed nfcapd files, then that will be
>>>>>>>>> OK.
>>>>>>>>> However, So far I am unable to get optarg -x to move the nfcapd file
>>>>>>>>> from nfcapd.YYYYmmddhhmm to nfcapd.hhmm. It seems to be a problem
>>>>>>>>> with the -x variables; %d %f. Whenever you try to combine any -x
>>>>>>>>> variable such as %d or %f with any string, they stop working.
>>>>>>>>>
>>>>>>>>> None of the examples below work.
>>>>>>>>> 'cr-ul' => { 'port' => '10151', 'col' => '#00ff00',
>>>>>>>>> 'type' => 'netflow', 'optarg' => '-t 5 -x "mv -f %d/%f
>>>>>>>>> %d/nfcapd.new"' },
>>>>>>>>> The next two examples , are how i envision renaming the nfcapd
>>>>>>>>> files. Stripping out YYYYmmdd from the filename and replacing it with
>>>>>>>>> hhmm:
>>>>>>>>> 'cr-ul' => { 'port' => '10151', 'col' => '#00ff00',
>>>>>>>>> 'type' => 'netflow', 'optarg' => '-t 5 -x "perl -e \"my ($suf) =
>>>>>>>>> $ARGV[0] =~ m/(....)$/; `mv %d/$ARGV[0] nfcapd.$suf`;\" %f"' },
>>>>>>>>> 'cr-ul' => { 'port' => '10151', 'col' => '#00ff00',
>>>>>>>>> 'type' => 'netflow', 'optarg' => '-t 5 -x "suf=`expr substr %f 16
>>>>>>>>> 4`;mv -f %d/%f %d/nfcapd.$suf"' },
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The examples below actually work. But as soon as I combine the use of
>>>>>>>>> a -x variable, such as %d. They no longer work, as seen above.
>>>>>>>>> 'cr-ul' => { 'port' => '10151', 'col' => '#00ff00',
>>>>>>>>> 'type' => 'netflow', 'optarg' => '-t 5 -x "mv -f %d/%f
>>>>>>>>> /tmp/testflow"'
>>>>>>>>> 'cr-ul' => { 'port' => '10151', 'col' => '#00ff00',
>>>>>>>>> 'type' => 'netflow', 'optarg' => '-t 5 -x "mv -f %d/%f
>>>>>>>>> /var/data/nfsen/profiles-data/live/cr-ul/2010/04/01/nfcapd.new"' },
>>>>>>>>>
>>>>>>>>> Any ideas?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> --Chad
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Apr 1, 2010, at 1:41 AM, Peter Haag wrote:
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 3/30/10 15:11, ckotil wrote:
>>>>>>>>>>>> That's exactly what I am trying to do.
>>>>>>>>>>>> I did consider using the -x parameter after reading through the
>>>>>>>>>>>> man page for nfdump, but I wasn't exactly sure how to use it.
>>>>>>>>>>>> One problem I had with hacking up the source is that the nfsen
>>>>>>>>>>>> frontend then needed to be modified to look for filenames named
>>>>>>>>>>>> `nfcapd.hhmm`; the filenames with hour and minute.
>>>>>>>>>>>>
>>>>>>>>>>>> If -x is used with nfcapd, will nfsen still need to be modified or
>>>>>>>>>>>> is there a config bit we can set , instructing nfsen what
>>>>>>>>>>>> filenames to look for?
>>>>>>>>
>>>>>>>> No - you can use the 'optarg' argument in the %sources definition.
>>>>>>>> 'optarg' => '-x whatever ...'
>>>>>>>>
>>>>>>>> - Peter
>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> --Chad
>>>>>>>>>>>>
>>>>>>>>>>>> On Mar 30, 2010, at 12:53 AM, Manish Kumar wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi ckotil,
>>>>>>>>>>>>> If I get your problem, there is one way out. At the
>>>>>>>>>>>>> time of capturing itself you can rename your file like this.
>>>>>>>>>>>>>
>>>>>>>>>>>>> ./nfcapd -p port_no -t rotating_time -l location_of_files -I
>>>>>>>>>>>>> Binary_file_name -x 'mv file_location_dir/%f file_location_dir/%i'
>>>>>>>>>>>>>
>>>>>>>>>>>>> By this You will always have a single file in ur directory with
>>>>>>>>>>>>> the name of Binary_file_name, so that you don't have to use wild
>>>>>>>>>>>>> card while reading with nfdump -r, you can run the collector at
>>>>>>>>>>>>> the specified time only and stop it by controlling with a script.
>>>>>>>>>>>>>
>>>>>>>>>>>>> May be it work for you.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Mar 29, 2010 at 9:08 PM, ckotil <[email protected]>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> I would like to collect statistics on my netflow from multiple
>>>>>>>>>>>>> hosts , spanning multiple days and a specific time. For example
>>>>>>>>>>>>> from host1 , host2, and host3, on 03/26/2010, 03/27/2010, and
>>>>>>>>>>>>> 03/28/2010 at 0800. The problem I am having is that nfdump seems
>>>>>>>>>>>>> unable to use a wildcard.
>>>>>>>>>>>>> Here is the command:
>>>>>>>>>>>>>
>>>>>>>>>>>>> [u...@netflow]$ nfdump -M
>>>>>>>>>>>>> /var/data/nfsen/profiles-data/live/cr-ul/2010/03/26:27:28 -R
>>>>>>>>>>>>> nfcapd.*0800 'inet6 and not dst ip fec0:0:0:ffff::1' -S
>>>>>>>>>>>>> WARNING: -S depricated! use -s record/packets/bytes instead.
>>>>>>>>>>>>> Option will get removed.
>>>>>>>>>>>>> stat() error
>>>>>>>>>>>>> '/var/data/nfsen/profiles-data/live/cr-ul/2010/03/26/nfcapd.*0800':
>>>>>>>>>>>>> File not found!
>>>>>>>>>>>>>
>>>>>>>>>>>>> I was able to wrap this command in a script, and by using the -R
>>>>>>>>>>>>> command I could make this work.
>>>>>>>>>>>>> Another solution I found was to hack the source code so that
>>>>>>>>>>>>> filenames were written to disk without year, month, and day;
>>>>>>>>>>>>> nfcapd.0800 for example. Then I could use the command above
>>>>>>>>>>>>> without a wildcard.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is there another way to do this without additional scripting or
>>>>>>>>>>>>> hacking up the source?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> --Chad
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ------------------------------------------------------------------------------
>>>>>>>>>>>>> Download Intel® Parallel Studio Eval
>>>>>>>>>>>>> Try the new software tools for yourself. Speed compiling, find
>>>>>>>>>>>>> bugs
>>>>>>>>>>>>> proactively, and fine-tune applications for parallel performance.
>>>>>>>>>>>>> See why Intel Parallel Studio got high marks during beta.
>>>>>>>>>>>>> http://p.sf.net/sfu/intel-sw-dev
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Nfsen-discuss mailing list
>>>>>>>>>>>>> [email protected]
>>>>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>>> Manish Kumar,
>>>>>>>>>>>>> Project Associate,
>>>>>>>>>>>>> Computer Networks & Internet Engineering Division,
>>>>>>>>>>>>> Centre for Development of Advanced Computing R&D,
>>>>>>>>>>>>> #68,Electronics City,
>>>>>>>>>>>>> Bangalore 560100,
>>>>>>>>>>>>> Karnataka, India
>>>>>>>>>>>>> Mobile:9886739073
>>>>>>>>>>>>> Ph: 080 28523300 Extn: 2511
>>>>>>>>>>>>> Email: [email protected]
>>>>>>>>>>>>> http://cens.cdac.in/
>>>>>>>>>>>>
>>>>>>>>>>>> Chad E. Kotil
>>>>>>>>>>>> GRNOC Systems Engineer
>>>>>>>>>>>> 812-855-5288
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ------------------------------------------------------------------------------
>>>>>>>>>>>> Download Intel® Parallel Studio Eval
>>>>>>>>>>>> Try the new software tools for yourself. Speed compiling, find bugs
>>>>>>>>>>>> proactively, and fine-tune applications for parallel performance.
>>>>>>>>>>>> See why Intel Parallel Studio got high marks during beta.
>>>>>>>>>>>> http://p.sf.net/sfu/intel-sw-dev
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Nfsen-discuss mailing list
>>>>>>>>>>>> [email protected]
>>>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ------------------------------------------------------------------------------
>>>>>>>>> Download Intel® Parallel Studio Eval
>>>>>>>>> Try the new software tools for yourself. Speed compiling, find bugs
>>>>>>>>> proactively, and fine-tune applications for parallel performance.
>>>>>>>>> See why Intel Parallel Studio got high marks during beta.
>>>>>>>>> http://p.sf.net/sfu/intel-sw-dev
>>>>>>>>> _______________________________________________
>>>>>>>>> Nfsen-discuss mailing list
>>>>>>>>> [email protected]
>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
>>>>>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.5 (Darwin)
>
> iQCVAwUBS7y9fP5AbZRALNr/AQL4DgP/U2v0h7XBcxZXyiypVfRZHOONRorPm6SV
> YnT4FmLaq1e+i2cPB0frIfpXCKCwZxJ13r5jjhk/CVN1iowTaqMmOrIjEa2saOuf
> 1ppL6BQptzTZ70szY1HbBhoS/JIa59L28g2Db2g/1CDIGt1jgkx27pzzsGy5cLmZ
> hGgFw4vpsE8=
> =hzRZ
> -----END PGP SIGNATURE-----
Chad E. Kotil
GRNOC Systems Engineer
812-855-5288
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Nfsen-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
