Found a suitable solution.
http://www.mail-archive.com/[email protected]/msg00293.html
--Chad
On Apr 7, 2010, at 1:19 PM, ckotil wrote:
>
>
> On Apr 7, 2010, at 1:14 PM, Peter Haag wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>>
>>
>> On 7/4/10 6:58 PM, ckotil wrote:
>>> Yes, but my script performs a `mv nfcapd.YYYYMMDDhhmm nfcapd.hhmm`. How
>>> would nfsen know to look for nfcapd.hhmm ?
>>
>> ?? why do you move files?? Do not touch/remove/move the original file. You
>> may process the flows, but do not alter the
>> file in any way.
>>
>
> The reason for this thread. The evolution of the thread lead me to rename the
> files.
> Because I want to get statistics from multiple hosts AND times. Nfdump does
> not handle wildcards gracefully enough to do this currently. With the files
> renamed to nfcapd.hhmm it became possible for nfdump to process multiple
> hosts and times.
>
> I have opted to write a script that will accept a set of hosts and times, and
> then use nfdump -R to process the files.
>
>
>> - Peter
>>
>>> Im running nfcapd-1.6.1 now.
>>>
>>> --Chad
>>>
>>> On Apr 7, 2010, at 12:53 PM, Peter Haag wrote:
>>>
>>>
>>>
>>> On 6/4/10 8:47 PM, ckotil wrote:
>>>>>> I am using -x to rename the files to nfcapd.hhmm. Nfsen looks for files
>>>>>> named nfcapd.YYYYMMDDhhmm. I think that's why Nfsen is no longer
>>>>>> populating the RRDs, cannot process netflow via the webinterface, and
>>>>>> nfexpire isn't purging the old flows..
>>>>>> If I remove the optional argument to move the flows, everything starts
>>>>>> working again.
>>>>>>
>>>
>>> This _does_ work. As already mentioned, NfSen has no clue about this
>>> additional argument. Expiring the profile is fully
>>> independant from optarg.
>>>
>>>
>>>>>> --Chad
>>>>>>
>>>>>>
>>>>>> On Apr 6, 2010, at 4:23 AM, Peter Haag wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 4/2/10 22:06, ckotil wrote:
>>>>>>>>> I was able to rename the nfcapd files by using a perl script.
>>>>>>>>>
>>>>>>>>> 'optarg' => '-x "perl /usr/local/bin/nfcapd-rewrite.pl %d/%f %d
>>>>>>>>> %f'"},
>>>
>>> There is btw a mistake with quotes:
>>>
>>> 'optarg' => "-x 'perl /usr/local/bin/nfcapd-rewrite.pl %d/%f %d %f'"}
>>>
>>>
>>> - Peter
>>>
>>>>>>>>>
>>>>>>>>> However, This makes Nfsen sad. Nfsen loses track of the files. The
>>>>>>>>> rrd graphs have holes in them. Nfexpire does not purge flows, and the
>>>>>>>>> Nfsen frontend is unable to process stats on the flows.
>>>>>>
>>>>>> There is no reason, why NfSen shouldn't like this additional -x. It's
>>>>>> executed in the additional nfcapd/launcher
>>>>>> process, which NfSen has no knowledge about it's existance anyway.
>>>>>> So there must be another reason for that. What do the logfiles say??
>>>>>>
>>>>>> - Peter
>>>>>>
>>>>>>>>>
>>>>>>>>> It looks like we need a better way to rename nfcapd files. Feature
>>>>>>>>> request?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> --Chad
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Apr 1, 2010, at 2:02 PM, ckotil wrote:
>>>>>>>>>
>>>>>>>>>> As long as NfSen can find the renamed nfcapd files, then that will
>>>>>>>>>> be OK.
>>>>>>>>>> However, So far I am unable to get optarg -x to move the nfcapd file
>>>>>>>>>> from nfcapd.YYYYmmddhhmm to nfcapd.hhmm. It seems to be a problem
>>>>>>>>>> with the -x variables; %d %f. Whenever you try to combine any -x
>>>>>>>>>> variable such as %d or %f with any string, they stop working.
>>>>>>>>>>
>>>>>>>>>> None of the examples below work.
>>>>>>>>>> 'cr-ul' => { 'port' => '10151', 'col' => '#00ff00',
>>>>>>>>>> 'type' => 'netflow', 'optarg' => '-t 5 -x "mv -f %d/%f
>>>>>>>>>> %d/nfcapd.new"' },
>>>>>>>>>> The next two examples , are how i envision renaming the nfcapd
>>>>>>>>>> files. Stripping out YYYYmmdd from the filename and replacing it
>>>>>>>>>> with hhmm:
>>>>>>>>>> 'cr-ul' => { 'port' => '10151', 'col' => '#00ff00',
>>>>>>>>>> 'type' => 'netflow', 'optarg' => '-t 5 -x "perl -e \"my ($suf) =
>>>>>>>>>> $ARGV[0] =~ m/(....)$/; `mv %d/$ARGV[0] nfcapd.$suf`;\" %f"' },
>>>>>>>>>> 'cr-ul' => { 'port' => '10151', 'col' => '#00ff00',
>>>>>>>>>> 'type' => 'netflow', 'optarg' => '-t 5 -x "suf=`expr substr %f 16
>>>>>>>>>> 4`;mv -f %d/%f %d/nfcapd.$suf"' },
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The examples below actually work. But as soon as I combine the use
>>>>>>>>>> of a -x variable, such as %d. They no longer work, as seen above.
>>>>>>>>>> 'cr-ul' => { 'port' => '10151', 'col' => '#00ff00',
>>>>>>>>>> 'type' => 'netflow', 'optarg' => '-t 5 -x "mv -f %d/%f
>>>>>>>>>> /tmp/testflow"'
>>>>>>>>>> 'cr-ul' => { 'port' => '10151', 'col' => '#00ff00',
>>>>>>>>>> 'type' => 'netflow', 'optarg' => '-t 5 -x "mv -f %d/%f
>>>>>>>>>> /var/data/nfsen/profiles-data/live/cr-ul/2010/04/01/nfcapd.new"' },
>>>>>>>>>>
>>>>>>>>>> Any ideas?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> --Chad
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Apr 1, 2010, at 1:41 AM, Peter Haag wrote:
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 3/30/10 15:11, ckotil wrote:
>>>>>>>>>>>>> That's exactly what I am trying to do.
>>>>>>>>>>>>> I did consider using the -x parameter after reading through the
>>>>>>>>>>>>> man page for nfdump, but I wasn't exactly sure how to use it.
>>>>>>>>>>>>> One problem I had with hacking up the source is that the nfsen
>>>>>>>>>>>>> frontend then needed to be modified to look for filenames named
>>>>>>>>>>>>> `nfcapd.hhmm`; the filenames with hour and minute.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If -x is used with nfcapd, will nfsen still need to be modified
>>>>>>>>>>>>> or is there a config bit we can set , instructing nfsen what
>>>>>>>>>>>>> filenames to look for?
>>>>>>>>>
>>>>>>>>> No - you can use the 'optarg' argument in the %sources definition.
>>>>>>>>> 'optarg' => '-x whatever ...'
>>>>>>>>>
>>>>>>>>> - Peter
>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> --Chad
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mar 30, 2010, at 12:53 AM, Manish Kumar wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi ckotil,
>>>>>>>>>>>>>> If I get your problem, there is one way out. At the
>>>>>>>>>>>>>> time of capturing itself you can rename your file like this.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ./nfcapd -p port_no -t rotating_time -l location_of_files -I
>>>>>>>>>>>>>> Binary_file_name -x 'mv file_location_dir/%f
>>>>>>>>>>>>>> file_location_dir/%i'
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> By this You will always have a single file in ur directory with
>>>>>>>>>>>>>> the name of Binary_file_name, so that you don't have to use wild
>>>>>>>>>>>>>> card while reading with nfdump -r, you can run the collector at
>>>>>>>>>>>>>> the specified time only and stop it by controlling with a script.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> May be it work for you.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Mar 29, 2010 at 9:08 PM, ckotil <[email protected]>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>> I would like to collect statistics on my netflow from multiple
>>>>>>>>>>>>>> hosts , spanning multiple days and a specific time. For example
>>>>>>>>>>>>>> from host1 , host2, and host3, on 03/26/2010, 03/27/2010, and
>>>>>>>>>>>>>> 03/28/2010 at 0800. The problem I am having is that nfdump seems
>>>>>>>>>>>>>> unable to use a wildcard.
>>>>>>>>>>>>>> Here is the command:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [u...@netflow]$ nfdump -M
>>>>>>>>>>>>>> /var/data/nfsen/profiles-data/live/cr-ul/2010/03/26:27:28 -R
>>>>>>>>>>>>>> nfcapd.*0800 'inet6 and not dst ip fec0:0:0:ffff::1' -S
>>>>>>>>>>>>>> WARNING: -S depricated! use -s record/packets/bytes instead.
>>>>>>>>>>>>>> Option will get removed.
>>>>>>>>>>>>>> stat() error
>>>>>>>>>>>>>> '/var/data/nfsen/profiles-data/live/cr-ul/2010/03/26/nfcapd.*0800':
>>>>>>>>>>>>>> File not found!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I was able to wrap this command in a script, and by using the -R
>>>>>>>>>>>>>> command I could make this work.
>>>>>>>>>>>>>> Another solution I found was to hack the source code so that
>>>>>>>>>>>>>> filenames were written to disk without year, month, and day;
>>>>>>>>>>>>>> nfcapd.0800 for example. Then I could use the command above
>>>>>>>>>>>>>> without a wildcard.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is there another way to do this without additional scripting or
>>>>>>>>>>>>>> hacking up the source?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --Chad
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ------------------------------------------------------------------------------
>>>>>>>>>>>>>> Download Intel® Parallel Studio Eval
>>>>>>>>>>>>>> Try the new software tools for yourself. Speed compiling, find
>>>>>>>>>>>>>> bugs
>>>>>>>>>>>>>> proactively, and fine-tune applications for parallel performance.
>>>>>>>>>>>>>> See why Intel Parallel Studio got high marks during beta.
>>>>>>>>>>>>>> http://p.sf.net/sfu/intel-sw-dev
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Nfsen-discuss mailing list
>>>>>>>>>>>>>> [email protected]
>>>>>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>>>> Manish Kumar,
>>>>>>>>>>>>>> Project Associate,
>>>>>>>>>>>>>> Computer Networks & Internet Engineering Division,
>>>>>>>>>>>>>> Centre for Development of Advanced Computing R&D,
>>>>>>>>>>>>>> #68,Electronics City,
>>>>>>>>>>>>>> Bangalore 560100,
>>>>>>>>>>>>>> Karnataka, India
>>>>>>>>>>>>>> Mobile:9886739073
>>>>>>>>>>>>>> Ph: 080 28523300 Extn: 2511
>>>>>>>>>>>>>> Email: [email protected]
>>>>>>>>>>>>>> http://cens.cdac.in/
>>>>>>>>>>>>>
>>>>>>>>>>>>> Chad E. Kotil
>>>>>>>>>>>>> GRNOC Systems Engineer
>>>>>>>>>>>>> 812-855-5288
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ------------------------------------------------------------------------------
>>>>>>>>>>>>> Download Intel® Parallel Studio Eval
>>>>>>>>>>>>> Try the new software tools for yourself. Speed compiling, find
>>>>>>>>>>>>> bugs
>>>>>>>>>>>>> proactively, and fine-tune applications for parallel performance.
>>>>>>>>>>>>> See why Intel Parallel Studio got high marks during beta.
>>>>>>>>>>>>> http://p.sf.net/sfu/intel-sw-dev
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Nfsen-discuss mailing list
>>>>>>>>>>>>> [email protected]
>>>>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ------------------------------------------------------------------------------
>>>>>>>>>> Download Intel® Parallel Studio Eval
>>>>>>>>>> Try the new software tools for yourself. Speed compiling, find bugs
>>>>>>>>>> proactively, and fine-tune applications for parallel performance.
>>>>>>>>>> See why Intel Parallel Studio got high marks during beta.
>>>>>>>>>> http://p.sf.net/sfu/intel-sw-dev
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Nfsen-discuss mailing list
>>>>>>>>>> [email protected]
>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
>>>>>>
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.5 (Darwin)
>>
>> iQCVAwUBS7y9fP5AbZRALNr/AQL4DgP/U2v0h7XBcxZXyiypVfRZHOONRorPm6SV
>> YnT4FmLaq1e+i2cPB0frIfpXCKCwZxJ13r5jjhk/CVN1iowTaqMmOrIjEa2saOuf
>> 1ppL6BQptzTZ70szY1HbBhoS/JIa59L28g2Db2g/1CDIGt1jgkx27pzzsGy5cLmZ
>> hGgFw4vpsE8=
>> =hzRZ
>> -----END PGP SIGNATURE-----
>
> Chad E. Kotil
> GRNOC Systems Engineer
> 812-855-5288
>
>
> ------------------------------------------------------------------------------
> Download Intel® Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Nfsen-discuss mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Nfsen-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
