Hi Alfredo, I played around with ZC, too, and noticed kind of the same problem. Would using multiple queues with ZC provide any benefit and how can I specify them in case of?
Thanks, Jan ________________________________________ From: [email protected] [[email protected]] on behalf of Alfredo Cardigliano [[email protected]] Sent: Friday, June 19, 2015 09:49 To: [email protected] Subject: Re: [Ntop-misc] Trouble capturing packets using ZC and n2disk10gzc Hi Olivier please load the driver with RSS=1,1 when running n2disk on top of ZC (it does capture from a single interface/queue unless you explicitly pass all the queues) Alfredo > On 18 Jun 2015, at 18:21, Olivier Prin <[email protected]> wrote: > > Hi everyone, > > I am in the process of doing benchmarks for capturing 10Gbps network traffic > on a freshly installed Linux Ubuntu 12.04 box. > > I managed to run my tests with PF_RING 6.1.1, ixgbe zc drivers and tcpdump > with and without using ZC on my 10g network interface. > > I installed the binary n2disk package to do some more tests and while it's > working fine without using ZC, I can't get it to work with ZC. n2disk stop > reading from the interface after the 4925th packets. > > I need some help figuring what I did wrong with n2disk when I use ZC. > > The box is running Ubuntu 12.04 without updates. > PF_RING and n2disk were installed by using the .deb binary files from the > ntop website with all the necessary dependencies: > e1000e-zc-dkms_3.0.4.1.69_all.deb > i40e-zc-dkms_1.1.23.69_all.deb > igb-zc-dkms_5.2.5.69_all.deb > ixgbe-zc-dkms_3.22.3.69_all.deb > libnl1_1.1-7_amd64.deb > libnuma1_2.0.8~rc3-1_amd64.deb > libnuma-dev_2.0.8~rc3-1_amd64.deb > n2disk_2.3.150617-4707_amd64.deb > pfring_6.1.1-69_amd64.deb > pfring-dkms_6.1.1_all.deb > pfring-drivers-zc-dkms_1.2_all.deb > > I have a license for ZC for the network interface. > I have no license for n2disk. > > The pf_ring module is loaded with the options: options pf_ring > transparent_mode=2 min_num_slots=250000 enable_tx_capture=0 > > The ixgbe module is loaded with the options: options ixgbe MQ=1,1 RSS=8,8 > > I have reserved 2048 hugepages for my tests. > > The capture interface (Intel X520-SR2 based on 82599EB chipset) is configured > as follow: > # ethtool -g eth8 > Ring parameters for eth8: > Pre-set maximums: > RX: 32768 > RX Mini: 0 > RX Jumbo: 0 > TX: 32768 > Current hardware settings: > RX: 32768 > RX Mini: 0 > RX Jumbo: 0 > TX: 8192 > > # ethtool -i eth8 > driver: ixgbe > version: 3.22.3 > firmware-version: 0x61bd0001 > bus-info: 0000:04:00.0 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > > The command line for a working test with n2disk without ZC is: n2disk10gzc -i > eth8 -o /mnt/captures/zc\:eth8/ -p 1000 -b 2000 -q 1 -C 4096 -s 500 -c 4 -w > 12 -U > > The output is the following: > 18/Jun/2015 15:36:37 [n2disk_zc.c:4158] WARNING: Invalid license found for > 93C406BC9105A1EF > 18/Jun/2015 15:36:37 [n2disk_zc.c:4159] WARNING: (See /etc/n2disk10g.license) > [Missing license file] > 18/Jun/2015 15:36:37 [n2disk_zc.c:4585] WARNING: > *************************************************** > 18/Jun/2015 15:36:37 [n2disk_zc.c:4586] WARNING: ** > ** > 18/Jun/2015 15:36:37 [n2disk_zc.c:4587] WARNING: ** Switching to DEMO MODE > due to license error ** > 18/Jun/2015 15:36:37 [n2disk_zc.c:4588] WARNING: ** > ** > 18/Jun/2015 15:36:37 [n2disk_zc.c:4589] WARNING: > *************************************************** > 18/Jun/2015 15:36:37 [n2disk_zc.c:4590] WARNING: ** Dumping will stop > after 5 min ** > 18/Jun/2015 15:36:37 [n2disk_zc.c:4592] WARNING: > *************************************************** > 18/Jun/2015 15:36:37 [n2disk_zc.c:4596] Welcome to n2disk10g v.2.3.150617 > (r4707) > 18/Jun/2015 15:36:37 [n2disk_zc.c:4622] Running on 2 node(s) system with 16 > core(s). NUMA affinity set to node 1. > 18/Jun/2015 15:36:37 [n2disk_zc.c:4696] Using PF_RING for packet capture > 18/Jun/2015 15:36:37 [n2disk_zc.c:4700] Multithread support enabled > 18/Jun/2015 15:36:37 [n2disk_zc.c:4811] Dump files max size is set to 1000 MB > 18/Jun/2015 15:36:37 [n2disk_zc.c:4830] Buffer memory is set to 1 GB (2 pcap > files buffer) > 18/Jun/2015 15:36:37 [n2disk_zc.c:4867] Using directory /mnt/captures/zc:eth8 > for dump files > 18/Jun/2015 15:36:37 [n2disk_zc.c:4872] No sub-directories will be created > 18/Jun/2015 15:36:37 [n2disk_zc.c:4874] Up to 1000 files will be written > before overwriting > 18/Jun/2015 15:36:37 [n2disk_zc.c:4899] Dumping data in 4.0 MB chunks > 18/Jun/2015 15:36:37 [n2disk_zc.c:5111] Memory allocated successfully > 18/Jun/2015 15:36:37 [n2disk_zc.c:3323] Using packet timestamps from pf_ring > 18/Jun/2015 15:36:37 [n2disk_zc.c:3362] Using PF_RING v.6.1.1 > 18/Jun/2015 15:36:37 [n2disk_zc.c:3372] Dumping traffic statistics on > /proc/net/pf_ring/stats/7498-eth8.23 > 18/Jun/2015 15:36:37 [n2disk_zc.c:3381] Started PF_RING packet reader thread > for device eth8 > 18/Jun/2015 15:36:37 [n2disk_zc.c:3768] [reader] Packet capture started > 18/Jun/2015 15:36:37 [n2disk_zc.c:5198] WARNING: Unable to change user when > using hugepages > > # cat /proc/net/pf_ring/7498-eth8.23 > Bound Device(s) : eth8 > Active : 1 > Breed : Standard > Appl. Name : n2disk10g > Socket Mode : RX only > Capture Direction : RX only > Sampling Rate : 1 > IP Defragment : No > BPF Filtering : Disabled > # Sw Filt. Rules : 0 > # Hw Filt. Rules : 0 > Poll Pkt Watermark : 128 > Num Poll Calls : 465146 > Channel Id Mask : 0xFFFFFFFFFFFFFFFF > Cluster Id : 0 > Slot Version : 16 [6.1.1] > Min Num Slots : 250014 > Bucket Len : 500 > Slot Len : 544 [bucket+header] > Tot Memory : 136015872 > Tot Packets : 17307755 > Tot Pkt Lost : 0 > Tot Insert : 17307755 > Tot Read : 17307746 > Insert Offset : 95943832 > Remove Offset : 95940152 > Num Free Slots : 250003 > Reflect: Fwd Ok : 0 > Reflect: Fwd Errors: 0 > > Here we see that the breed used is standard and not ZC. > > When I stop the capture I get this output along with 19GB of pcap files: > ^C18/Jun/2015 15:41:02 [n2disk_zc.c:875] Caught termination signal 2... > 18/Jun/2015 15:41:02 [n2disk_zc.c:780] [PF_RING] Total stats: 63421671 pkts > rcvd/63421671 pkts filtered/0 pkts dropped [0.0%] > 18/Jun/2015 15:41:02 [n2disk_zc.c:793] [n2disk] Processed 63421671 pkts > 18/Jun/2015 15:41:02 [n2disk_zc.c:798] Capture Duration: > 0:00:04:24:245 > 18/Jun/2015 15:41:02 [n2disk_zc.c:800] Average Capture Throughput: 0.73 Gbit > / 0.24 Mpps > 18/Jun/2015 15:41:03 [n2disk_zc.c:3994] Packet capture thread terminated > 18/Jun/2015 15:41:03 [n2disk_zc.c:5229] Reader thread terminated > 18/Jun/2015 15:41:04 [n2disk_zc.c:5239] Writer thread terminated > 18/Jun/2015 15:41:04 [n2disk_zc.c:5265] Destroying hugepages > > I'm using tcpreplay to generate the packets using 1 dump file replayed 1000 > times (last about 1 minute). > > When I do the same test with ZC, I'm using this command: n2disk10gzc -i > zc:eth8 -o /mnt/captures/zc\:eth8/ -p 1000 -b 2000 -q 1 -C 4096 -s 500 -S 5 > -c 4 -w 12 > > The output is: > 18/Jun/2015 17:19:24 [n2disk_zc.c:4158] WARNING: Invalid license found for > 93C406BC9105A1EF > 18/Jun/2015 17:19:24 [n2disk_zc.c:4159] WARNING: (See /etc/n2disk10g.license) > [Missing license file] > 18/Jun/2015 17:19:24 [n2disk_zc.c:4585] WARNING: > *************************************************** > 18/Jun/2015 17:19:24 [n2disk_zc.c:4586] WARNING: ** > ** > 18/Jun/2015 17:19:24 [n2disk_zc.c:4587] WARNING: ** Switching to DEMO MODE > due to license error ** > 18/Jun/2015 17:19:24 [n2disk_zc.c:4588] WARNING: ** > ** > 18/Jun/2015 17:19:24 [n2disk_zc.c:4589] WARNING: > *************************************************** > 18/Jun/2015 17:19:24 [n2disk_zc.c:4590] WARNING: ** Dumping will stop > after 5 min ** > 18/Jun/2015 17:19:24 [n2disk_zc.c:4592] WARNING: > *************************************************** > 18/Jun/2015 17:19:24 [n2disk_zc.c:4596] Welcome to n2disk10g v.2.3.150617 > (r4707) > 18/Jun/2015 17:19:24 [n2disk_zc.c:4622] Running on 2 node(s) system with 16 > core(s). NUMA affinity set to node 1. > 18/Jun/2015 17:19:24 [n2disk_zc.c:4675] WARNING: If you are using standard > drivers (packet capture via kernel) please disable time-pulse thread > 18/Jun/2015 17:19:24 [n2disk_zc.c:4696] Using PF_RING for packet capture > 18/Jun/2015 17:19:24 [n2disk_zc.c:4700] Multithread support enabled > 18/Jun/2015 17:19:24 [n2disk_zc.c:4811] Dump files max size is set to 1000 MB > 18/Jun/2015 17:19:24 [n2disk_zc.c:4830] Buffer memory is set to 1 GB (2 pcap > files buffer) > 18/Jun/2015 17:19:24 [n2disk_zc.c:4867] Using directory /mnt/captures/zc:eth8 > for dump files > 18/Jun/2015 17:19:24 [n2disk_zc.c:4872] No sub-directories will be created > 18/Jun/2015 17:19:24 [n2disk_zc.c:4874] Up to 1000 files will be written > before overwriting > 18/Jun/2015 17:19:24 [n2disk_zc.c:4899] Dumping data in 4.0 MB chunks > 18/Jun/2015 17:19:25 [n2disk_zc.c:5111] Memory allocated successfully > 18/Jun/2015 17:19:25 [n2disk_zc.c:3315] Using time pulse timestamps > 18/Jun/2015 17:19:25 [n2disk_zc.c:3362] Using PF_RING v.6.1.1 > 18/Jun/2015 17:19:25 [n2disk_zc.c:3372] Dumping traffic statistics on > /proc/net/pf_ring/stats/5414-eth8.7 > 18/Jun/2015 17:19:25 [n2disk_zc.c:3381] Started PF_RING packet reader thread > for device zc:eth8 > 18/Jun/2015 17:19:25 [n2disk_zc.c:2739] Time pulse thread started > 18/Jun/2015 17:19:25 [n2disk_zc.c:3768] [reader] Packet capture started > 18/Jun/2015 17:19:26 [n2disk_zc.c:391] n2disk10g changed user to n2disk > > # cat /proc/net/pf_ring/5414-eth8.7 > Bound Device(s) : > Active : 1 > Breed : ZC > Appl. Name : n2disk10g > Socket Mode : RX only > Capture Direction : RX only > Sampling Rate : 1 > IP Defragment : No > BPF Filtering : Disabled > # Sw Filt. Rules : 0 > # Hw Filt. Rules : 0 > Poll Pkt Watermark : 128 > Num Poll Calls : 507415 > Channel Id : 0 > Num RX Slots : 32768 > Tot Memory : 655360 bytes > > But when I look at the stats in "/proc/net/pf_ring/stats/5414-eth8.7", they > are stuck after 4925 packets (over 77 millions packets are normally sent > during a test): > Duration: 0:00:12:41:082 > Throughput: 0.00 Mpps 0.00 Gbps > Packets: 4925 > Filtered: 4925 > Dropped: 0 > Bytes: 2100613 > DumpedBytes: 0 > DumpedFiles: 0 > SlowSlavesLoops: 0 > SlowStorageLoops: 0 > > tcpreplay is stuck too until I stop the capture with n2disk. > > When I stop the capture, I have one file of 1,7 MB with the 3925 packets but > nothing more. > > Have anyone encountered this behaviour before ? > > Thanks in advance. > > Olivier. > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc _______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
