> On 19 Jun 2015, at 10:48, Olivier Prin <[email protected]> wrote: > > Hi Alfredo, > > Thanks a lot, it works with RSS=1,1 ! > > Does it mean that I don't need to use MQ to capture 10g at wire rate ?
Not with n2disk, n2disk has an internal multithreaded architecture which does not need multiple queues for packet capture. > If I understand this then when I had threads to the capture with -R, it add > reader threads to pull packets faster from the unique queue instead of adding > thread to read from multiple queues ? Correct. Alfredo > > Regards, > > Olivier. > > Le 19/06/2015 09:49, Alfredo Cardigliano a écrit : >> Hi Olivier >> please load the driver with RSS=1,1 when running n2disk on top of ZC (it >> does capture from a single interface/queue unless you explicitly pass all >> the queues) >> >> Alfredo >> >>> On 18 Jun 2015, at 18:21, Olivier Prin <[email protected]> >>> <mailto:[email protected]> wrote: >>> >>> Hi everyone, >>> >>> I am in the process of doing benchmarks for capturing 10Gbps network >>> traffic on a freshly installed Linux Ubuntu 12.04 box. >>> >>> I managed to run my tests with PF_RING 6.1.1, ixgbe zc drivers and tcpdump >>> with and without using ZC on my 10g network interface. >>> >>> I installed the binary n2disk package to do some more tests and while it's >>> working fine without using ZC, I can't get it to work with ZC. n2disk stop >>> reading from the interface after the 4925th packets. >>> >>> I need some help figuring what I did wrong with n2disk when I use ZC. >>> >>> The box is running Ubuntu 12.04 without updates. >>> PF_RING and n2disk were installed by using the .deb binary files from the >>> ntop website with all the necessary dependencies: >>> e1000e-zc-dkms_3.0.4.1.69_all.deb >>> i40e-zc-dkms_1.1.23.69_all.deb >>> igb-zc-dkms_5.2.5.69_all.deb >>> ixgbe-zc-dkms_3.22.3.69_all.deb >>> libnl1_1.1-7_amd64.deb >>> libnuma1_2.0.8~rc3-1_amd64.deb >>> libnuma-dev_2.0.8~rc3-1_amd64.deb >>> n2disk_2.3.150617-4707_amd64.deb >>> pfring_6.1.1-69_amd64.deb >>> pfring-dkms_6.1.1_all.deb >>> pfring-drivers-zc-dkms_1.2_all.deb >>> >>> I have a license for ZC for the network interface. >>> I have no license for n2disk. >>> >>> The pf_ring module is loaded with the options: options pf_ring >>> transparent_mode=2 min_num_slots=250000 enable_tx_capture=0 >>> >>> The ixgbe module is loaded with the options: options ixgbe MQ=1,1 RSS=8,8 >>> >>> I have reserved 2048 hugepages for my tests. >>> >>> The capture interface (Intel X520-SR2 based on 82599EB chipset) is >>> configured as follow: >>> # ethtool -g eth8 >>> Ring parameters for eth8: >>> Pre-set maximums: >>> RX: 32768 >>> RX Mini: 0 >>> RX Jumbo: 0 >>> TX: 32768 >>> Current hardware settings: >>> RX: 32768 >>> RX Mini: 0 >>> RX Jumbo: 0 >>> TX: 8192 >>> >>> # ethtool -i eth8 >>> driver: ixgbe >>> version: 3.22.3 >>> firmware-version: 0x61bd0001 >>> bus-info: 0000:04:00.0 >>> supports-statistics: yes >>> supports-test: yes >>> supports-eeprom-access: yes >>> supports-register-dump: yes >>> >>> The command line for a working test with n2disk without ZC is: n2disk10gzc >>> -i eth8 -o /mnt/captures/zc\:eth8/ -p 1000 -b 2000 -q 1 -C 4096 -s 500 -c 4 >>> -w 12 -U >>> >>> The output is the following: >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:4158] WARNING: Invalid license found for >>> 93C406BC9105A1EF >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:4159] WARNING: (See >>> /etc/n2disk10g.license) [Missing license file] >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:4585] WARNING: >>> *************************************************** >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:4586] WARNING: ** >>> ** >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:4587] WARNING: ** Switching to DEMO MODE >>> due to license error ** >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:4588] WARNING: ** >>> ** >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:4589] WARNING: >>> *************************************************** >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:4590] WARNING: ** Dumping will >>> stop after 5 min ** >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:4592] WARNING: >>> *************************************************** >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:4596] Welcome to n2disk10g v.2.3.150617 >>> (r4707) >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:4622] Running on 2 node(s) system with 16 >>> core(s). NUMA affinity set to node 1. >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:4696] Using PF_RING for packet capture >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:4700] Multithread support enabled >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:4811] Dump files max size is set to 1000 >>> MB >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:4830] Buffer memory is set to 1 GB (2 >>> pcap files buffer) >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:4867] Using directory >>> /mnt/captures/zc:eth8 for dump files >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:4872] No sub-directories will be created >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:4874] Up to 1000 files will be written >>> before overwriting >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:4899] Dumping data in 4.0 MB chunks >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:5111] Memory allocated successfully >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:3323] Using packet timestamps from pf_ring >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:3362] Using PF_RING v.6.1.1 >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:3372] Dumping traffic statistics on >>> /proc/net/pf_ring/stats/7498-eth8.23 >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:3381] Started PF_RING packet reader >>> thread for device eth8 >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:3768] [reader] Packet capture started >>> 18/Jun/2015 15:36:37 [n2disk_zc.c:5198] WARNING: Unable to change user when >>> using hugepages >>> >>> # cat /proc/net/pf_ring/7498-eth8.23 >>> Bound Device(s) : eth8 >>> Active : 1 >>> Breed : Standard >>> Appl. Name : n2disk10g >>> Socket Mode : RX only >>> Capture Direction : RX only >>> Sampling Rate : 1 >>> IP Defragment : No >>> BPF Filtering : Disabled >>> # Sw Filt. Rules : 0 >>> # Hw Filt. Rules : 0 >>> Poll Pkt Watermark : 128 >>> Num Poll Calls : 465146 >>> Channel Id Mask : 0xFFFFFFFFFFFFFFFF >>> Cluster Id : 0 >>> Slot Version : 16 [6.1.1] >>> Min Num Slots : 250014 >>> Bucket Len : 500 >>> Slot Len : 544 [bucket+header] >>> Tot Memory : 136015872 >>> Tot Packets : 17307755 >>> Tot Pkt Lost : 0 >>> Tot Insert : 17307755 >>> Tot Read : 17307746 >>> Insert Offset : 95943832 >>> Remove Offset : 95940152 >>> Num Free Slots : 250003 >>> Reflect: Fwd Ok : 0 >>> Reflect: Fwd Errors: 0 >>> >>> Here we see that the breed used is standard and not ZC. >>> >>> When I stop the capture I get this output along with 19GB of pcap files: >>> ^C18/Jun/2015 15:41:02 [n2disk_zc.c:875] Caught termination signal 2... >>> 18/Jun/2015 15:41:02 [n2disk_zc.c:780] [PF_RING] Total stats: 63421671 pkts >>> rcvd/63421671 pkts filtered/0 pkts dropped [0.0%] >>> 18/Jun/2015 15:41:02 [n2disk_zc.c:793] [n2disk] Processed 63421671 pkts >>> 18/Jun/2015 15:41:02 [n2disk_zc.c:798] Capture Duration: >>> 0:00:04:24:245 >>> 18/Jun/2015 15:41:02 [n2disk_zc.c:800] Average Capture Throughput: 0.73 >>> Gbit / 0.24 Mpps >>> 18/Jun/2015 15:41:03 [n2disk_zc.c:3994] Packet capture thread terminated >>> 18/Jun/2015 15:41:03 [n2disk_zc.c:5229] Reader thread terminated >>> 18/Jun/2015 15:41:04 [n2disk_zc.c:5239] Writer thread terminated >>> 18/Jun/2015 15:41:04 [n2disk_zc.c:5265] Destroying hugepages >>> >>> I'm using tcpreplay to generate the packets using 1 dump file replayed 1000 >>> times (last about 1 minute). >>> >>> When I do the same test with ZC, I'm using this command: n2disk10gzc -i >>> zc:eth8 -o /mnt/captures/zc\:eth8/ -p 1000 -b 2000 -q 1 -C 4096 -s 500 -S 5 >>> -c 4 -w 12 >>> >>> The output is: >>> 18/Jun/2015 17:19:24 [n2disk_zc.c:4158] WARNING: Invalid license found for >>> 93C406BC9105A1EF >>> 18/Jun/2015 17:19:24 [n2disk_zc.c:4159] WARNING: (See >>> /etc/n2disk10g.license) [Missing license file] >>> 18/Jun/2015 17:19:24 [n2disk_zc.c:4585] WARNING: >>> *************************************************** >>> 18/Jun/2015 17:19:24 [n2disk_zc.c:4586] WARNING: ** >>> ** >>> 18/Jun/2015 17:19:24 [n2disk_zc.c:4587] WARNING: ** Switching to DEMO MODE >>> due to license error ** >>> 18/Jun/2015 17:19:24 [n2disk_zc.c:4588] WARNING: ** >>> ** >>> 18/Jun/2015 17:19:24 [n2disk_zc.c:4589] WARNING: >>> *************************************************** >>> 18/Jun/2015 17:19:24 [n2disk_zc.c:4590] WARNING: ** Dumping will >>> stop after 5 min ** >>> 18/Jun/2015 17:19:24 [n2disk_zc.c:4592] WARNING: >>> *************************************************** >>> 18/Jun/2015 17:19:24 [n2disk_zc.c:4596] Welcome to n2disk10g v.2.3.150617 >>> (r4707) >>> 18/Jun/2015 17:19:24 [n2disk_zc.c:4622] Running on 2 node(s) system with 16 >>> core(s). NUMA affinity set to node 1. >>> 18/Jun/2015 17:19:24 [n2disk_zc.c:4675] WARNING: If you are using standard >>> drivers (packet capture via kernel) please disable time-pulse thread >>> 18/Jun/2015 17:19:24 [n2disk_zc.c:4696] Using PF_RING for packet capture >>> 18/Jun/2015 17:19:24 [n2disk_zc.c:4700] Multithread support enabled >>> 18/Jun/2015 17:19:24 [n2disk_zc.c:4811] Dump files max size is set to 1000 >>> MB >>> 18/Jun/2015 17:19:24 [n2disk_zc.c:4830] Buffer memory is set to 1 GB (2 >>> pcap files buffer) >>> 18/Jun/2015 17:19:24 [n2disk_zc.c:4867] Using directory >>> /mnt/captures/zc:eth8 for dump files >>> 18/Jun/2015 17:19:24 [n2disk_zc.c:4872] No sub-directories will be created >>> 18/Jun/2015 17:19:24 [n2disk_zc.c:4874] Up to 1000 files will be written >>> before overwriting >>> 18/Jun/2015 17:19:24 [n2disk_zc.c:4899] Dumping data in 4.0 MB chunks >>> 18/Jun/2015 17:19:25 [n2disk_zc.c:5111] Memory allocated successfully >>> 18/Jun/2015 17:19:25 [n2disk_zc.c:3315] Using time pulse timestamps >>> 18/Jun/2015 17:19:25 [n2disk_zc.c:3362] Using PF_RING v.6.1.1 >>> 18/Jun/2015 17:19:25 [n2disk_zc.c:3372] Dumping traffic statistics on >>> /proc/net/pf_ring/stats/5414-eth8.7 >>> 18/Jun/2015 17:19:25 [n2disk_zc.c:3381] Started PF_RING packet reader >>> thread for device zc:eth8 >>> 18/Jun/2015 17:19:25 [n2disk_zc.c:2739] Time pulse thread started >>> 18/Jun/2015 17:19:25 [n2disk_zc.c:3768] [reader] Packet capture started >>> 18/Jun/2015 17:19:26 [n2disk_zc.c:391] n2disk10g changed user to n2disk >>> >>> # cat /proc/net/pf_ring/5414-eth8.7 >>> Bound Device(s) : >>> Active : 1 >>> Breed : ZC >>> Appl. Name : n2disk10g >>> Socket Mode : RX only >>> Capture Direction : RX only >>> Sampling Rate : 1 >>> IP Defragment : No >>> BPF Filtering : Disabled >>> # Sw Filt. Rules : 0 >>> # Hw Filt. Rules : 0 >>> Poll Pkt Watermark : 128 >>> Num Poll Calls : 507415 >>> Channel Id : 0 >>> Num RX Slots : 32768 >>> Tot Memory : 655360 bytes >>> >>> But when I look at the stats in "/proc/net/pf_ring/stats/5414-eth8.7", they >>> are stuck after 4925 packets (over 77 millions packets are normally sent >>> during a test): >>> Duration: 0:00:12:41:082 >>> Throughput: 0.00 Mpps 0.00 Gbps >>> Packets: 4925 >>> Filtered: 4925 >>> Dropped: 0 >>> Bytes: 2100613 >>> DumpedBytes: 0 >>> DumpedFiles: 0 >>> SlowSlavesLoops: 0 >>> SlowStorageLoops: 0 >>> >>> tcpreplay is stuck too until I stop the capture with n2disk. >>> >>> When I stop the capture, I have one file of 1,7 MB with the 3925 packets >>> but nothing more. >>> >>> Have anyone encountered this behaviour before ? >>> >>> Thanks in advance. >>> >>> Olivier. >>> _______________________________________________ >>> Ntop-misc mailing list >>> [email protected] <mailto:[email protected]> >>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >> >> >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] <mailto:[email protected]> >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
