Hi Alfredo,
Thanks a lot, it works with RSS=1,1 !
Does it mean that I don't need to use MQ to capture 10g at wire rate ?
If I understand this then when I had threads to the capture with -R, it
add reader threads to pull packets faster from the unique queue instead
of adding thread to read from multiple queues ?
Regards,
Olivier.
Le 19/06/2015 09:49, Alfredo Cardigliano a écrit :
Hi Olivier
please load the driver with RSS=1,1 when running n2disk on top of ZC (it does
capture from a single interface/queue unless you explicitly pass all the queues)
Alfredo
On 18 Jun 2015, at 18:21, Olivier Prin <[email protected]> wrote:
Hi everyone,
I am in the process of doing benchmarks for capturing 10Gbps network traffic on
a freshly installed Linux Ubuntu 12.04 box.
I managed to run my tests with PF_RING 6.1.1, ixgbe zc drivers and tcpdump with
and without using ZC on my 10g network interface.
I installed the binary n2disk package to do some more tests and while it's
working fine without using ZC, I can't get it to work with ZC. n2disk stop
reading from the interface after the 4925th packets.
I need some help figuring what I did wrong with n2disk when I use ZC.
The box is running Ubuntu 12.04 without updates.
PF_RING and n2disk were installed by using the .deb binary files from the ntop
website with all the necessary dependencies:
e1000e-zc-dkms_3.0.4.1.69_all.deb
i40e-zc-dkms_1.1.23.69_all.deb
igb-zc-dkms_5.2.5.69_all.deb
ixgbe-zc-dkms_3.22.3.69_all.deb
libnl1_1.1-7_amd64.deb
libnuma1_2.0.8~rc3-1_amd64.deb
libnuma-dev_2.0.8~rc3-1_amd64.deb
n2disk_2.3.150617-4707_amd64.deb
pfring_6.1.1-69_amd64.deb
pfring-dkms_6.1.1_all.deb
pfring-drivers-zc-dkms_1.2_all.deb
I have a license for ZC for the network interface.
I have no license for n2disk.
The pf_ring module is loaded with the options: options pf_ring
transparent_mode=2 min_num_slots=250000 enable_tx_capture=0
The ixgbe module is loaded with the options: options ixgbe MQ=1,1 RSS=8,8
I have reserved 2048 hugepages for my tests.
The capture interface (Intel X520-SR2 based on 82599EB chipset) is configured
as follow:
# ethtool -g eth8
Ring parameters for eth8:
Pre-set maximums:
RX: 32768
RX Mini: 0
RX Jumbo: 0
TX: 32768
Current hardware settings:
RX: 32768
RX Mini: 0
RX Jumbo: 0
TX: 8192
# ethtool -i eth8
driver: ixgbe
version: 3.22.3
firmware-version: 0x61bd0001
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
The command line for a working test with n2disk without ZC is: n2disk10gzc -i
eth8 -o /mnt/captures/zc\:eth8/ -p 1000 -b 2000 -q 1 -C 4096 -s 500 -c 4 -w 12
-U
The output is the following:
18/Jun/2015 15:36:37 [n2disk_zc.c:4158] WARNING: Invalid license found for
93C406BC9105A1EF
18/Jun/2015 15:36:37 [n2disk_zc.c:4159] WARNING: (See /etc/n2disk10g.license)
[Missing license file]
18/Jun/2015 15:36:37 [n2disk_zc.c:4585] WARNING:
***************************************************
18/Jun/2015 15:36:37 [n2disk_zc.c:4586] WARNING: **
**
18/Jun/2015 15:36:37 [n2disk_zc.c:4587] WARNING: ** Switching to DEMO MODE due
to license error **
18/Jun/2015 15:36:37 [n2disk_zc.c:4588] WARNING: **
**
18/Jun/2015 15:36:37 [n2disk_zc.c:4589] WARNING:
***************************************************
18/Jun/2015 15:36:37 [n2disk_zc.c:4590] WARNING: ** Dumping will stop
after 5 min **
18/Jun/2015 15:36:37 [n2disk_zc.c:4592] WARNING:
***************************************************
18/Jun/2015 15:36:37 [n2disk_zc.c:4596] Welcome to n2disk10g v.2.3.150617
(r4707)
18/Jun/2015 15:36:37 [n2disk_zc.c:4622] Running on 2 node(s) system with 16
core(s). NUMA affinity set to node 1.
18/Jun/2015 15:36:37 [n2disk_zc.c:4696] Using PF_RING for packet capture
18/Jun/2015 15:36:37 [n2disk_zc.c:4700] Multithread support enabled
18/Jun/2015 15:36:37 [n2disk_zc.c:4811] Dump files max size is set to 1000 MB
18/Jun/2015 15:36:37 [n2disk_zc.c:4830] Buffer memory is set to 1 GB (2 pcap
files buffer)
18/Jun/2015 15:36:37 [n2disk_zc.c:4867] Using directory /mnt/captures/zc:eth8
for dump files
18/Jun/2015 15:36:37 [n2disk_zc.c:4872] No sub-directories will be created
18/Jun/2015 15:36:37 [n2disk_zc.c:4874] Up to 1000 files will be written before
overwriting
18/Jun/2015 15:36:37 [n2disk_zc.c:4899] Dumping data in 4.0 MB chunks
18/Jun/2015 15:36:37 [n2disk_zc.c:5111] Memory allocated successfully
18/Jun/2015 15:36:37 [n2disk_zc.c:3323] Using packet timestamps from pf_ring
18/Jun/2015 15:36:37 [n2disk_zc.c:3362] Using PF_RING v.6.1.1
18/Jun/2015 15:36:37 [n2disk_zc.c:3372] Dumping traffic statistics on
/proc/net/pf_ring/stats/7498-eth8.23
18/Jun/2015 15:36:37 [n2disk_zc.c:3381] Started PF_RING packet reader thread
for device eth8
18/Jun/2015 15:36:37 [n2disk_zc.c:3768] [reader] Packet capture started
18/Jun/2015 15:36:37 [n2disk_zc.c:5198] WARNING: Unable to change user when
using hugepages
# cat /proc/net/pf_ring/7498-eth8.23
Bound Device(s) : eth8
Active : 1
Breed : Standard
Appl. Name : n2disk10g
Socket Mode : RX only
Capture Direction : RX only
Sampling Rate : 1
IP Defragment : No
BPF Filtering : Disabled
# Sw Filt. Rules : 0
# Hw Filt. Rules : 0
Poll Pkt Watermark : 128
Num Poll Calls : 465146
Channel Id Mask : 0xFFFFFFFFFFFFFFFF
Cluster Id : 0
Slot Version : 16 [6.1.1]
Min Num Slots : 250014
Bucket Len : 500
Slot Len : 544 [bucket+header]
Tot Memory : 136015872
Tot Packets : 17307755
Tot Pkt Lost : 0
Tot Insert : 17307755
Tot Read : 17307746
Insert Offset : 95943832
Remove Offset : 95940152
Num Free Slots : 250003
Reflect: Fwd Ok : 0
Reflect: Fwd Errors: 0
Here we see that the breed used is standard and not ZC.
When I stop the capture I get this output along with 19GB of pcap files:
^C18/Jun/2015 15:41:02 [n2disk_zc.c:875] Caught termination signal 2...
18/Jun/2015 15:41:02 [n2disk_zc.c:780] [PF_RING] Total stats: 63421671 pkts
rcvd/63421671 pkts filtered/0 pkts dropped [0.0%]
18/Jun/2015 15:41:02 [n2disk_zc.c:793] [n2disk] Processed 63421671 pkts
18/Jun/2015 15:41:02 [n2disk_zc.c:798] Capture Duration:
0:00:04:24:245
18/Jun/2015 15:41:02 [n2disk_zc.c:800] Average Capture Throughput: 0.73 Gbit /
0.24 Mpps
18/Jun/2015 15:41:03 [n2disk_zc.c:3994] Packet capture thread terminated
18/Jun/2015 15:41:03 [n2disk_zc.c:5229] Reader thread terminated
18/Jun/2015 15:41:04 [n2disk_zc.c:5239] Writer thread terminated
18/Jun/2015 15:41:04 [n2disk_zc.c:5265] Destroying hugepages
I'm using tcpreplay to generate the packets using 1 dump file replayed 1000
times (last about 1 minute).
When I do the same test with ZC, I'm using this command: n2disk10gzc -i zc:eth8
-o /mnt/captures/zc\:eth8/ -p 1000 -b 2000 -q 1 -C 4096 -s 500 -S 5 -c 4 -w 12
The output is:
18/Jun/2015 17:19:24 [n2disk_zc.c:4158] WARNING: Invalid license found for
93C406BC9105A1EF
18/Jun/2015 17:19:24 [n2disk_zc.c:4159] WARNING: (See /etc/n2disk10g.license)
[Missing license file]
18/Jun/2015 17:19:24 [n2disk_zc.c:4585] WARNING:
***************************************************
18/Jun/2015 17:19:24 [n2disk_zc.c:4586] WARNING: **
**
18/Jun/2015 17:19:24 [n2disk_zc.c:4587] WARNING: ** Switching to DEMO MODE due
to license error **
18/Jun/2015 17:19:24 [n2disk_zc.c:4588] WARNING: **
**
18/Jun/2015 17:19:24 [n2disk_zc.c:4589] WARNING:
***************************************************
18/Jun/2015 17:19:24 [n2disk_zc.c:4590] WARNING: ** Dumping will stop
after 5 min **
18/Jun/2015 17:19:24 [n2disk_zc.c:4592] WARNING:
***************************************************
18/Jun/2015 17:19:24 [n2disk_zc.c:4596] Welcome to n2disk10g v.2.3.150617
(r4707)
18/Jun/2015 17:19:24 [n2disk_zc.c:4622] Running on 2 node(s) system with 16
core(s). NUMA affinity set to node 1.
18/Jun/2015 17:19:24 [n2disk_zc.c:4675] WARNING: If you are using standard
drivers (packet capture via kernel) please disable time-pulse thread
18/Jun/2015 17:19:24 [n2disk_zc.c:4696] Using PF_RING for packet capture
18/Jun/2015 17:19:24 [n2disk_zc.c:4700] Multithread support enabled
18/Jun/2015 17:19:24 [n2disk_zc.c:4811] Dump files max size is set to 1000 MB
18/Jun/2015 17:19:24 [n2disk_zc.c:4830] Buffer memory is set to 1 GB (2 pcap
files buffer)
18/Jun/2015 17:19:24 [n2disk_zc.c:4867] Using directory /mnt/captures/zc:eth8
for dump files
18/Jun/2015 17:19:24 [n2disk_zc.c:4872] No sub-directories will be created
18/Jun/2015 17:19:24 [n2disk_zc.c:4874] Up to 1000 files will be written before
overwriting
18/Jun/2015 17:19:24 [n2disk_zc.c:4899] Dumping data in 4.0 MB chunks
18/Jun/2015 17:19:25 [n2disk_zc.c:5111] Memory allocated successfully
18/Jun/2015 17:19:25 [n2disk_zc.c:3315] Using time pulse timestamps
18/Jun/2015 17:19:25 [n2disk_zc.c:3362] Using PF_RING v.6.1.1
18/Jun/2015 17:19:25 [n2disk_zc.c:3372] Dumping traffic statistics on
/proc/net/pf_ring/stats/5414-eth8.7
18/Jun/2015 17:19:25 [n2disk_zc.c:3381] Started PF_RING packet reader thread
for device zc:eth8
18/Jun/2015 17:19:25 [n2disk_zc.c:2739] Time pulse thread started
18/Jun/2015 17:19:25 [n2disk_zc.c:3768] [reader] Packet capture started
18/Jun/2015 17:19:26 [n2disk_zc.c:391] n2disk10g changed user to n2disk
# cat /proc/net/pf_ring/5414-eth8.7
Bound Device(s) :
Active : 1
Breed : ZC
Appl. Name : n2disk10g
Socket Mode : RX only
Capture Direction : RX only
Sampling Rate : 1
IP Defragment : No
BPF Filtering : Disabled
# Sw Filt. Rules : 0
# Hw Filt. Rules : 0
Poll Pkt Watermark : 128
Num Poll Calls : 507415
Channel Id : 0
Num RX Slots : 32768
Tot Memory : 655360 bytes
But when I look at the stats in "/proc/net/pf_ring/stats/5414-eth8.7", they are
stuck after 4925 packets (over 77 millions packets are normally sent during a test):
Duration: 0:00:12:41:082
Throughput: 0.00 Mpps 0.00 Gbps
Packets: 4925
Filtered: 4925
Dropped: 0
Bytes: 2100613
DumpedBytes: 0
DumpedFiles: 0
SlowSlavesLoops: 0
SlowStorageLoops: 0
tcpreplay is stuck too until I stop the capture with n2disk.
When I stop the capture, I have one file of 1,7 MB with the 3925 packets but
nothing more.
Have anyone encountered this behaviour before ?
Thanks in advance.
Olivier.
_______________________________________________
Ntop-misc mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop-misc
_______________________________________________
Ntop-misc mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop-misc
_______________________________________________
Ntop-misc mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop-misc
