Hi Alfredo, spreading traffic across multiple cores is actually my use case as I am using Bro. I tried to use multiple receive queues but this crashed the whole machine. Then I was told to use zbalance_ipc to load balance the traffic by connection (-m 1) and use only a single receive queue. This worked for me but I still wonder if there may be any benefit in using more queues on the NIC and how PF_RING ZC would handle this.
Jan ________________________________________ From: [email protected] [[email protected]] on behalf of Alfredo Cardigliano [[email protected]] Sent: Friday, June 19, 2015 10:30 To: [email protected] Subject: Re: [Ntop-misc] Trouble capturing packets using ZC and n2disk10gzc Hi Jan multiple queues are needed when you need to spread the load across multiple cores, for example when using multiple instances of nProbe or Snort, or multithreaded application with native support to multiple interfaces/queues. n2disk does not need multiqueue since it needs to serialise traffic before dumping it to disk. Alfredo > On 19 Jun 2015, at 10:26, Jan Grashofer <[email protected]> wrote: > > Hi Alfredo, > > I played around with ZC, too, and noticed kind of the same problem. Would > using multiple queues with ZC provide any benefit and how can I specify them > in case of? > > Thanks, > Jan > > ________________________________________ > From: [email protected] > [[email protected]] on behalf of Alfredo Cardigliano > [[email protected]] > Sent: Friday, June 19, 2015 09:49 > To: [email protected] > Subject: Re: [Ntop-misc] Trouble capturing packets using ZC and n2disk10gzc > > Hi Olivier > please load the driver with RSS=1,1 when running n2disk on top of ZC (it does > capture from a single interface/queue unless you explicitly pass all the > queues) > > Alfredo > >> On 18 Jun 2015, at 18:21, Olivier Prin <[email protected]> wrote: >> >> Hi everyone, >> >> I am in the process of doing benchmarks for capturing 10Gbps network traffic >> on a freshly installed Linux Ubuntu 12.04 box. >> >> I managed to run my tests with PF_RING 6.1.1, ixgbe zc drivers and tcpdump >> with and without using ZC on my 10g network interface. >> >> I installed the binary n2disk package to do some more tests and while it's >> working fine without using ZC, I can't get it to work with ZC. n2disk stop >> reading from the interface after the 4925th packets. >> >> I need some help figuring what I did wrong with n2disk when I use ZC. >> >> The box is running Ubuntu 12.04 without updates. >> PF_RING and n2disk were installed by using the .deb binary files from the >> ntop website with all the necessary dependencies: >> e1000e-zc-dkms_3.0.4.1.69_all.deb >> i40e-zc-dkms_1.1.23.69_all.deb >> igb-zc-dkms_5.2.5.69_all.deb >> ixgbe-zc-dkms_3.22.3.69_all.deb >> libnl1_1.1-7_amd64.deb >> libnuma1_2.0.8~rc3-1_amd64.deb >> libnuma-dev_2.0.8~rc3-1_amd64.deb >> n2disk_2.3.150617-4707_amd64.deb >> pfring_6.1.1-69_amd64.deb >> pfring-dkms_6.1.1_all.deb >> pfring-drivers-zc-dkms_1.2_all.deb >> >> I have a license for ZC for the network interface. >> I have no license for n2disk. >> >> The pf_ring module is loaded with the options: options pf_ring >> transparent_mode=2 min_num_slots=250000 enable_tx_capture=0 >> >> The ixgbe module is loaded with the options: options ixgbe MQ=1,1 RSS=8,8 >> >> I have reserved 2048 hugepages for my tests. >> >> The capture interface (Intel X520-SR2 based on 82599EB chipset) is >> configured as follow: >> # ethtool -g eth8 >> Ring parameters for eth8: >> Pre-set maximums: >> RX: 32768 >> RX Mini: 0 >> RX Jumbo: 0 >> TX: 32768 >> Current hardware settings: >> RX: 32768 >> RX Mini: 0 >> RX Jumbo: 0 >> TX: 8192 >> >> # ethtool -i eth8 >> driver: ixgbe >> version: 3.22.3 >> firmware-version: 0x61bd0001 >> bus-info: 0000:04:00.0 >> supports-statistics: yes >> supports-test: yes >> supports-eeprom-access: yes >> supports-register-dump: yes >> >> The command line for a working test with n2disk without ZC is: n2disk10gzc >> -i eth8 -o /mnt/captures/zc\:eth8/ -p 1000 -b 2000 -q 1 -C 4096 -s 500 -c 4 >> -w 12 -U >> >> The output is the following: >> 18/Jun/2015 15:36:37 [n2disk_zc.c:4158] WARNING: Invalid license found for >> 93C406BC9105A1EF >> 18/Jun/2015 15:36:37 [n2disk_zc.c:4159] WARNING: (See >> /etc/n2disk10g.license) [Missing license file] >> 18/Jun/2015 15:36:37 [n2disk_zc.c:4585] WARNING: >> *************************************************** >> 18/Jun/2015 15:36:37 [n2disk_zc.c:4586] WARNING: ** >> ** >> 18/Jun/2015 15:36:37 [n2disk_zc.c:4587] WARNING: ** Switching to DEMO MODE >> due to license error ** >> 18/Jun/2015 15:36:37 [n2disk_zc.c:4588] WARNING: ** >> ** >> 18/Jun/2015 15:36:37 [n2disk_zc.c:4589] WARNING: >> *************************************************** >> 18/Jun/2015 15:36:37 [n2disk_zc.c:4590] WARNING: ** Dumping will >> stop after 5 min ** >> 18/Jun/2015 15:36:37 [n2disk_zc.c:4592] WARNING: >> *************************************************** >> 18/Jun/2015 15:36:37 [n2disk_zc.c:4596] Welcome to n2disk10g v.2.3.150617 >> (r4707) >> 18/Jun/2015 15:36:37 [n2disk_zc.c:4622] Running on 2 node(s) system with 16 >> core(s). NUMA affinity set to node 1. >> 18/Jun/2015 15:36:37 [n2disk_zc.c:4696] Using PF_RING for packet capture >> 18/Jun/2015 15:36:37 [n2disk_zc.c:4700] Multithread support enabled >> 18/Jun/2015 15:36:37 [n2disk_zc.c:4811] Dump files max size is set to 1000 MB >> 18/Jun/2015 15:36:37 [n2disk_zc.c:4830] Buffer memory is set to 1 GB (2 pcap >> files buffer) >> 18/Jun/2015 15:36:37 [n2disk_zc.c:4867] Using directory >> /mnt/captures/zc:eth8 for dump files >> 18/Jun/2015 15:36:37 [n2disk_zc.c:4872] No sub-directories will be created >> 18/Jun/2015 15:36:37 [n2disk_zc.c:4874] Up to 1000 files will be written >> before overwriting >> 18/Jun/2015 15:36:37 [n2disk_zc.c:4899] Dumping data in 4.0 MB chunks >> 18/Jun/2015 15:36:37 [n2disk_zc.c:5111] Memory allocated successfully >> 18/Jun/2015 15:36:37 [n2disk_zc.c:3323] Using packet timestamps from pf_ring >> 18/Jun/2015 15:36:37 [n2disk_zc.c:3362] Using PF_RING v.6.1.1 >> 18/Jun/2015 15:36:37 [n2disk_zc.c:3372] Dumping traffic statistics on >> /proc/net/pf_ring/stats/7498-eth8.23 >> 18/Jun/2015 15:36:37 [n2disk_zc.c:3381] Started PF_RING packet reader thread >> for device eth8 >> 18/Jun/2015 15:36:37 [n2disk_zc.c:3768] [reader] Packet capture started >> 18/Jun/2015 15:36:37 [n2disk_zc.c:5198] WARNING: Unable to change user when >> using hugepages >> >> # cat /proc/net/pf_ring/7498-eth8.23 >> Bound Device(s) : eth8 >> Active : 1 >> Breed : Standard >> Appl. Name : n2disk10g >> Socket Mode : RX only >> Capture Direction : RX only >> Sampling Rate : 1 >> IP Defragment : No >> BPF Filtering : Disabled >> # Sw Filt. Rules : 0 >> # Hw Filt. Rules : 0 >> Poll Pkt Watermark : 128 >> Num Poll Calls : 465146 >> Channel Id Mask : 0xFFFFFFFFFFFFFFFF >> Cluster Id : 0 >> Slot Version : 16 [6.1.1] >> Min Num Slots : 250014 >> Bucket Len : 500 >> Slot Len : 544 [bucket+header] >> Tot Memory : 136015872 >> Tot Packets : 17307755 >> Tot Pkt Lost : 0 >> Tot Insert : 17307755 >> Tot Read : 17307746 >> Insert Offset : 95943832 >> Remove Offset : 95940152 >> Num Free Slots : 250003 >> Reflect: Fwd Ok : 0 >> Reflect: Fwd Errors: 0 >> >> Here we see that the breed used is standard and not ZC. >> >> When I stop the capture I get this output along with 19GB of pcap files: >> ^C18/Jun/2015 15:41:02 [n2disk_zc.c:875] Caught termination signal 2... >> 18/Jun/2015 15:41:02 [n2disk_zc.c:780] [PF_RING] Total stats: 63421671 pkts >> rcvd/63421671 pkts filtered/0 pkts dropped [0.0%] >> 18/Jun/2015 15:41:02 [n2disk_zc.c:793] [n2disk] Processed 63421671 pkts >> 18/Jun/2015 15:41:02 [n2disk_zc.c:798] Capture Duration: >> 0:00:04:24:245 >> 18/Jun/2015 15:41:02 [n2disk_zc.c:800] Average Capture Throughput: 0.73 Gbit >> / 0.24 Mpps >> 18/Jun/2015 15:41:03 [n2disk_zc.c:3994] Packet capture thread terminated >> 18/Jun/2015 15:41:03 [n2disk_zc.c:5229] Reader thread terminated >> 18/Jun/2015 15:41:04 [n2disk_zc.c:5239] Writer thread terminated >> 18/Jun/2015 15:41:04 [n2disk_zc.c:5265] Destroying hugepages >> >> I'm using tcpreplay to generate the packets using 1 dump file replayed 1000 >> times (last about 1 minute). >> >> When I do the same test with ZC, I'm using this command: n2disk10gzc -i >> zc:eth8 -o /mnt/captures/zc\:eth8/ -p 1000 -b 2000 -q 1 -C 4096 -s 500 -S 5 >> -c 4 -w 12 >> >> The output is: >> 18/Jun/2015 17:19:24 [n2disk_zc.c:4158] WARNING: Invalid license found for >> 93C406BC9105A1EF >> 18/Jun/2015 17:19:24 [n2disk_zc.c:4159] WARNING: (See >> /etc/n2disk10g.license) [Missing license file] >> 18/Jun/2015 17:19:24 [n2disk_zc.c:4585] WARNING: >> *************************************************** >> 18/Jun/2015 17:19:24 [n2disk_zc.c:4586] WARNING: ** >> ** >> 18/Jun/2015 17:19:24 [n2disk_zc.c:4587] WARNING: ** Switching to DEMO MODE >> due to license error ** >> 18/Jun/2015 17:19:24 [n2disk_zc.c:4588] WARNING: ** >> ** >> 18/Jun/2015 17:19:24 [n2disk_zc.c:4589] WARNING: >> *************************************************** >> 18/Jun/2015 17:19:24 [n2disk_zc.c:4590] WARNING: ** Dumping will >> stop after 5 min ** >> 18/Jun/2015 17:19:24 [n2disk_zc.c:4592] WARNING: >> *************************************************** >> 18/Jun/2015 17:19:24 [n2disk_zc.c:4596] Welcome to n2disk10g v.2.3.150617 >> (r4707) >> 18/Jun/2015 17:19:24 [n2disk_zc.c:4622] Running on 2 node(s) system with 16 >> core(s). NUMA affinity set to node 1. >> 18/Jun/2015 17:19:24 [n2disk_zc.c:4675] WARNING: If you are using standard >> drivers (packet capture via kernel) please disable time-pulse thread >> 18/Jun/2015 17:19:24 [n2disk_zc.c:4696] Using PF_RING for packet capture >> 18/Jun/2015 17:19:24 [n2disk_zc.c:4700] Multithread support enabled >> 18/Jun/2015 17:19:24 [n2disk_zc.c:4811] Dump files max size is set to 1000 MB >> 18/Jun/2015 17:19:24 [n2disk_zc.c:4830] Buffer memory is set to 1 GB (2 pcap >> files buffer) >> 18/Jun/2015 17:19:24 [n2disk_zc.c:4867] Using directory >> /mnt/captures/zc:eth8 for dump files >> 18/Jun/2015 17:19:24 [n2disk_zc.c:4872] No sub-directories will be created >> 18/Jun/2015 17:19:24 [n2disk_zc.c:4874] Up to 1000 files will be written >> before overwriting >> 18/Jun/2015 17:19:24 [n2disk_zc.c:4899] Dumping data in 4.0 MB chunks >> 18/Jun/2015 17:19:25 [n2disk_zc.c:5111] Memory allocated successfully >> 18/Jun/2015 17:19:25 [n2disk_zc.c:3315] Using time pulse timestamps >> 18/Jun/2015 17:19:25 [n2disk_zc.c:3362] Using PF_RING v.6.1.1 >> 18/Jun/2015 17:19:25 [n2disk_zc.c:3372] Dumping traffic statistics on >> /proc/net/pf_ring/stats/5414-eth8.7 >> 18/Jun/2015 17:19:25 [n2disk_zc.c:3381] Started PF_RING packet reader thread >> for device zc:eth8 >> 18/Jun/2015 17:19:25 [n2disk_zc.c:2739] Time pulse thread started >> 18/Jun/2015 17:19:25 [n2disk_zc.c:3768] [reader] Packet capture started >> 18/Jun/2015 17:19:26 [n2disk_zc.c:391] n2disk10g changed user to n2disk >> >> # cat /proc/net/pf_ring/5414-eth8.7 >> Bound Device(s) : >> Active : 1 >> Breed : ZC >> Appl. Name : n2disk10g >> Socket Mode : RX only >> Capture Direction : RX only >> Sampling Rate : 1 >> IP Defragment : No >> BPF Filtering : Disabled >> # Sw Filt. Rules : 0 >> # Hw Filt. Rules : 0 >> Poll Pkt Watermark : 128 >> Num Poll Calls : 507415 >> Channel Id : 0 >> Num RX Slots : 32768 >> Tot Memory : 655360 bytes >> >> But when I look at the stats in "/proc/net/pf_ring/stats/5414-eth8.7", they >> are stuck after 4925 packets (over 77 millions packets are normally sent >> during a test): >> Duration: 0:00:12:41:082 >> Throughput: 0.00 Mpps 0.00 Gbps >> Packets: 4925 >> Filtered: 4925 >> Dropped: 0 >> Bytes: 2100613 >> DumpedBytes: 0 >> DumpedFiles: 0 >> SlowSlavesLoops: 0 >> SlowStorageLoops: 0 >> >> tcpreplay is stuck too until I stop the capture with n2disk. >> >> When I stop the capture, I have one file of 1,7 MB with the 3925 packets but >> nothing more. >> >> Have anyone encountered this behaviour before ? >> >> Thanks in advance. >> >> Olivier. >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc _______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
