-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Peter,
- --On October 15, 2007 11:27:09 +0100 Peter Wood <[EMAIL PROTECTED]> wrote:
| Good Morning All,
|
| I sit writing this while I have a "large" query (for us ;) running on
| our NetFlow data. Here at Lancaster we record NF from both our border
| and our ResNet border (for the purpose of recording NAT translations).
|
| Currently running nfdump-snapshot-20070312 on one of our flows servers
| which is a Dell 2950 with 2 x 2.4Ghz Xeons (w/ Dual Cores), 4 x 500Gb (1
| System, 3 Data in FreeBSD gvinum striped). When performing a simple
| query, I see the three data disks at about 11Mb/s each, and one of the
| CPU cores at 7%.
|
| The result of said query is:
|
| Total flows processed: 345346192, skipped: 0, Bytes read: 17958283900
| Sys: 67.092s flows/second: 5147290.2 Wall: 827.917s flows/second: 417126.3
This makes up 17958283900/827 = 21.7 MB/s
However, this it the complete throughput incl. flow processing.
You now have an overhead of the filter processing, which you apply to each
of the 345 Mio flows. In worst case, you need the max. filter processing
time for each record. This processing time is worst case linear with the filter
elements of the entire tree, in case of the flow does not match the filter,
except for lists - rbtrees - which have O(log n). The filter is optimized
in that shorter sub trees are calculated first.
|
| Running a few rudimentary tests (which I understand are misrepresentive).
|
| # dd if=/dev/zero of=test bs=128k count=30000
| 3932160000 bytes transferred in 38.009780 secs (103451270 bytes/sec)
|
| systat -vmstat shows all three data disks writing data at about 35Mb/s each.
|
| # dd of=/dev/null if=test bs=128k count=30000
| 3932160000 bytes transferred in 25.982248 secs (151340253 bytes/sec)
|
| systat -vmstat shows all three data disks reading at about 40Mb/s each.
So looking at your nfdump throughput of 20MB/s incl. filter processing seems
reasonable for me comparing to 40MB/s raw disk throughput. Again - it depends
on your query type. The fastest you can get is:
./nfdump -r file ( or -M .. -R .. ) -w /dev/null
The tests I did:
dd if=/dev/zero of=test bs=128k count=30000
30000+0 records in
30000+0 records out
3932160000 bytes transferred in 37.757089 seconds (104143622 bytes/sec)
ll flows1
- -rw-r--r-- 1 peter staff 142197512 Dec 8 2006 flows1
time nfdump -r flows1 -w /dev/null
0.744u 0.216s 0:00.96 98.9% 0+0k 0+0io 0pf+0w
This is a linux with XFS
So nfdump shows not too much overhead with simply copy blocks to /dev/null
But take all with a grain of salt, as the FS cache also plays an important role.
|
| Obviously nfdump is reading from smaller files (in this case about 2.5Mb
| on average). Does anyone have any suggestions for optimisation of
| either FreeBSD or nfSen?
|
| Peter, I've had a quick look at nfdump's source code, I think that
| you're reading a record in at a time in small chunks, thus making many
| calls to fread/read making many calls per block. I can't guarantee any
| time due to other work commitments, but maybe looking at reading in
| larger blocks or maybe something like mmap might help?
nfdump files are splitted into record blocks of about 800-900k. I did many
tests when
searching for adequate block size, while designing the file format, as speed
was
always important to me. It turns out, you don't gain much above certain block
size.
Referring to nfdump-1.5.6 you can set the BUFFSIZE in nf_common.h. You may
change it,
to experiment. Just note - only new files are affected.
|
| Now that being said more recent versions might be different.
Up to and including all 1.5.x have the same architecture.
|
| Of course as always thanks for a great application, nfSen has made such
| a different to how and when we use flows, it really is excellent.
|
| Any information would be appreciated. Kind regards,
Hope, these insights help a bit more.
- Peter
|
| Peter.
| --
| Peter A. Wood e: [EMAIL PROTECTED]
| Network Security Specialist t: +44 1524 510153
| Technical Services Group
| Lancaster University
|
| PGP Fingerprint:
| C3B5 376B 16C5 F8D1 E3AE 617F 5718 C338 1D06 0689
|
|
- --
_______ SWITCH - The Swiss Education and Research Network ______
Peter Haag, Security Engineer, Member of SWITCH CERT
PGP fingerprint: D9 31 D5 83 03 95 68 BA FB 84 CA 94 AB FC 5D D7
SWITCH, Werdstrasse 2, P.O. Box, CH-8021 Zurich, Switzerland
E-mail: [EMAIL PROTECTED] Web: http://www.switch.ch/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)
iQCVAwUBRxNlqP5AbZRALNr/AQKKBwP7BcWb8JilRh3NMVfpIIjgNdTNQPGGfBe6
xOLaa5qIobtXJFrq2nNMWrcfN7tqB5a9aGWChM98mFaxNDEU3LUU5w+uVGGlFlPn
U/FQB6ypLrbWWBdoE59aWFLWVMNDi4NQt2IjWdrorBqnwruXAbAhinLZ/Q+Eql8J
j2Gx53qflL8=
=9wfk
-----END PGP SIGNATURE-----
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Nfsen-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
