Hi Jason I will have a look at what it does (how it generates traffic) and come back to you.
Alfredo > On 10 Jun 2016, at 15:56, Jason Lixfeld <[email protected]> wrote: > > Hi Alfredo, > > I can run MoonGen’s benchmark application (60 byte UDP packets) on one core > and achieve full literate. > > root@pgen:~/MoonGen# ./build/MoonGen examples/benchmark/udp-throughput.lua 1:1 > [INFO] Initializing DPDK. This will take a few seconds... > [INFO] Found 7 usable devices: > Device 0: 00:0C:BD:08:80:9C (82599EB 10-Gigabit SFI/SFP+ Network Connection) > Device 1: 00:0C:BD:08:80:9D (82599EB 10-Gigabit SFI/SFP+ Network Connection) > Device 2: 00:0C:BD:08:80:9A (82599EB 10-Gigabit SFI/SFP+ Network Connection) > Device 3: 00:0C:BD:08:80:9B (82599EB 10-Gigabit SFI/SFP+ Network Connection) > Device 4: 00:0C:BD:08:80:98 (82599EB 10-Gigabit SFI/SFP+ Network Connection) > Device 5: 00:0C:BD:08:80:99 (82599EB 10-Gigabit SFI/SFP+ Network Connection) > Device 6: 0C:C4:7A:AB:F9:C1 (I350 Gigabit Network Connection) > [WARN] You are running Linux >= 3.14, DDIO might not be working with DPDK in > this setup! > [WARN] This can cause a huge performance impact (one memory access per > packet!) preventing MoonGen from reaching line rate. > [WARN] Try using an older kernel (we recommend 3.13) if you see a low > performance or huge cache miss ratio. > [INFO] Waiting for devices to come up... > [INFO] Device 1 (00:0C:BD:08:80:9D) is up: full-duplex 10000 MBit/s > [INFO] 1 devices are up. > [Device: id=1] Sent 14878261 packets, current rate 14.88 Mpps, 7617.60 > MBit/s, 9998.09 MBit/s wire rate. > [Device: id=1] Sent 29758773 packets, current rate 14.88 Mpps, 7618.79 > MBit/s, 9999.66 MBit/s wire rate. > [Device: id=1] Sent 44639285 packets, current rate 14.88 Mpps, 7618.79 > MBit/s, 9999.66 MBit/s wire rate. > ^C[Device: id=1] Sent 51834112 packets with 3317383168 bytes payload > (including CRC). > [Device: id=1] Sent 14.880450 (StdDev 0.000001) Mpps, 7618.790143 (StdDev > 0.000847) MBit/s, 9999.662143 (StdDev 0.000999) MBit/s wire rate on average. > PMD: ixgbe_dev_tx_queue_stop(): Could not disable Tx Queue 0 > root@pgen:~/MoonGen# > > I am also to do 2 cores and get literate on two ports: > > root@pgen:~/MoonGen# ./build/MoonGen examples/benchmark/udp-throughput.lua > 1:1 4:1 > [INFO] Initializing DPDK. This will take a few seconds... > [INFO] Found 7 usable devices: > Device 0: 00:0C:BD:08:80:9C (82599EB 10-Gigabit SFI/SFP+ Network Connection) > Device 1: 00:0C:BD:08:80:9D (82599EB 10-Gigabit SFI/SFP+ Network Connection) > Device 2: 00:0C:BD:08:80:9A (82599EB 10-Gigabit SFI/SFP+ Network Connection) > Device 3: 00:0C:BD:08:80:9B (82599EB 10-Gigabit SFI/SFP+ Network Connection) > Device 4: 00:0C:BD:08:80:98 (82599EB 10-Gigabit SFI/SFP+ Network Connection) > Device 5: 00:0C:BD:08:80:99 (82599EB 10-Gigabit SFI/SFP+ Network Connection) > Device 6: 0C:C4:7A:AB:F9:C1 (I350 Gigabit Network Connection) > [WARN] You are running Linux >= 3.14, DDIO might not be working with DPDK in > this setup! > [WARN] This can cause a huge performance impact (one memory access per > packet!) preventing MoonGen from reaching line rate. > [WARN] Try using an older kernel (we recommend 3.13) if you see a low > performance or huge cache miss ratio. > [INFO] Waiting for devices to come up... > [INFO] Device 1 (00:0C:BD:08:80:9D) is up: full-duplex 10000 MBit/s > [INFO] Device 4 (00:0C:BD:08:80:98) is up: full-duplex 10000 MBit/s > [INFO] 2 devices are up. > [Device: id=1] Sent 14876880 packets, current rate 14.88 Mpps, 7616.92 > MBit/s, 9997.21 MBit/s wire rate. > [Device: id=4] Sent 14876732 packets, current rate 14.88 Mpps, 7616.78 > MBit/s, 9997.02 MBit/s wire rate. > [Device: id=1] Sent 29757364 packets, current rate 14.88 Mpps, 7618.78 > MBit/s, 9999.65 MBit/s wire rate. > [Device: id=4] Sent 29756984 packets, current rate 14.88 Mpps, 7618.69 > MBit/s, 9999.53 MBit/s wire rate. > [Device: id=1] Sent 44637898 packets, current rate 14.88 Mpps, 7618.80 > MBit/s, 9999.68 MBit/s wire rate. > [Device: id=4] Sent 44637237 packets, current rate 14.88 Mpps, 7618.69 > MBit/s, 9999.53 MBit/s wire rate. > [Device: id=1] Sent 59518743 packets, current rate 14.88 Mpps, 7618.98 > MBit/s, 9999.91 MBit/s wire rate. > [Device: id=4] Sent 59517493 packets, current rate 14.88 Mpps, 7618.69 > MBit/s, 9999.53 MBit/s wire rate. > ^C[Device: id=1] Sent 69843968 packets with 4470013952 bytes payload > (including CRC). > [Device: id=1] Sent 14.880561 (StdDev 0.000194) Mpps, 7618.853606 (StdDev > 0.110259) MBit/s, 9999.743332 (StdDev 0.141375) MBit/s wire rate on average. > [Device: id=4] Sent 69767424 packets with 4465115136 bytes payload (including > CRC). > [Device: id=4] Sent 14.880251 (StdDev 0.000002) Mpps, 7618.688334 (StdDev > 0.001419) MBit/s, 9999.528438 (StdDev 0.001742) MBit/s wire rate on average. > PMD: ixgbe_dev_tx_queue_stop(): Could not disable Tx Queue 0 > PMD: ixgbe_dev_tx_queue_stop(): Could not disable Tx Queue 0 > root@pgen:~/MoonGen# > >> On Jun 10, 2016, at 9:07 AM, Alfredo Cardigliano <[email protected]> >> wrote: >> >> Hi Jason >> it seems that other applications are interfering with zsend affecting the >> transmission rate, >> this could depend on several factor including core isolation (other >> applications using the >> core where zsend is running) and memory bandwidth (it seems that starting >> pfcount on >> another core affects zsend, thus this seems to be the case). >> Were you able to run MoonGen or other applications with better performance >> than zsend >> on the same machine? >> >> Thank you >> Alfredo >> >>> On 10 Jun 2016, at 13:27, Jason Lixfeld <[email protected]> wrote: >>> >>> Good day! >>> >>> I’m just circling back to see if anyone has any further insights on how I >>> might be able to get a steady 14.88Mpps? >>> >>> Admittedly, I spec’d this box to be able to use MoonGen. According to >>> their specs, I should be able to get a full 14.88Mpps across all 6 ports on >>> this NIC. That said, after discovering that pf_ring:zc is able to perform >>> as well as MoonGen, I’d rather go this route so I can (hopefully) us >>> Ostinato as a front-end. >>> >>> Thanks! >>> >>>> On Jun 8, 2016, at 5:55 PM, Jason Lixfeld <[email protected]> wrote: >>>> >>>> Without zcount and ./zsend -i zc:eth7 -g 1 -c 1 -a, it starts off at >>>> 14.88/10Gbps. It stays like that until I start doing stuff in another >>>> window (ssh’d in) like running top or changing directories. At which point >>>> it dropped down to 14.5Mpps and seems to be hovering there now. >>>> >>>> On Jun 8, 2016, at 5:43 PM, Alfredo Cardigliano <[email protected]> >>>> wrote: >>>> >>>> It could be related to available memory bandwidth, do you see the same >>>> when zcount is not running? >>>> >>>> Alfredo >>>> >>>> On 08 Jun 2016, at 23:40, Jason Lixfeld <[email protected]> wrote: >>>> >>>> I did. zsend to one core, zcount to another, but I can’t seen to quote get >>>> up there. send sometimes has the tendency to start off strong, about >>>> 14.86Mpps, but slowly ramps down to about 14.25Mpps after about 15 seconds. >>>> >>>> On Jun 8, 2016, at 4:48 PM, Alfredo Cardigliano <[email protected]> >>>> wrote: >>>> >>>> Did you bind pfsend/zsend to a cpu core? >>>> >>>> Alfredo >>>> >>>> On 08 Jun 2016, at 22:44, Jason Lixfeld <[email protected]> wrote: >>>> >>>> I don’t seem to be able to transmit the full 14.88Mpps to get full >>>> linerate. Is there anything else I can tweak? I’ve added the -a option >>>> which has given gotten me a bit closer, but not all the way. >>>> >>>> My CPU is a 6 core Intel® Xeon® CPU E5-2620 v3 @ 2.40GHz >>>> >>>> Thanks once again! >>>> >>>> On Jun 8, 2016, at 4:21 PM, Alfredo Cardigliano <[email protected]> >>>> wrote: >>>> >>>> Please use RSS=1,1,1,1,1,1 as you have 6 ports. >>>> >>>> Alfredo >>>> >>>> On 08 Jun 2016, at 22:11, [email protected] wrote: >>>> >>>> Thanks Alfredo, >>>> >>>> If I’m reading this correctly, eth2 has 12 Tx and Rx queues while eth7 has >>>> 1 each: >>>> >>>> root@pgen:/home/jlixfeld/PF_RING/drivers/intel/ixgbe/ixgbe-4.1.5-zc/src# >>>> cat /proc/net/pf_ring/dev/eth2/info Name: eth2 Index: 86 Address: >>>> 00:0C:BD:08:80:98 Polling Mode: NAPI/ZC Type: Ethernet Family: Intel ixgbe >>>> 82599 Max # TX Queues: 12 # Used RX Queues: 12 Num RX Slots: 32768 Num TX >>>> Slots: 32768 >>>> root@pgen:/home/jlixfeld/PF_RING/drivers/intel/ixgbe/ixgbe-4.1.5-zc/src# >>>> cat /proc/net/pf_ring/dev/eth7/info Name: eth7 Index: 83 Address: >>>> 00:0C:BD:08:80:9D Polling Mode: NAPI/ZC Type: Ethernet Family: Intel ixgbe >>>> 82599 Max # TX Queues: 1 # Used RX Queues: 1 Num RX Slots: 32768 Num TX >>>> Slots: 32768 >>>> root@pgen:/home/jlixfeld/PF_RING/drivers/intel/ixgbe/ixgbe-4.1.5-zc/src# >>>> >>>> load_driver.sh seems to be set to disable multi queue so I’m not quite >>>> sure how this got this way, or how to correct it? >>>> >>>> root@pgen:/home/jlixfeld/PF_RING/drivers/intel/ixgbe/ixgbe-4.1.5-zc/src# >>>> grep -i rss load_driver.sh #insmod ./ixgbe.ko RSS=0,0,0,0 insmod >>>> ./ixgbe.ko RSS=1,1,1,1 #insmod ./ixgbe.ko RSS=1,1,1,1 low_latency_tx=1 >>>> #insmod ./ixgbe.ko MQ=1,1,1,1 RSS=16,16,16,16 #insmod ./ixgbe.ko >>>> RSS=1,1,1,1 FdirPballoc=3,3,3,3 #insmod ./ixgbe.ko RSS=1,1,1,1 >>>> numa_cpu_affinity=0,0,0,0 >>>> root@pgen:/home/jlixfeld/PF_RING/drivers/intel/ixgbe/ixgbe-4.1.5-zc/src# >>>> >>>> On Jun 8, 2016, at 3:55 PM, Alfredo Cardigliano <[email protected]> >>>> wrote: >>>> >>>> Hi Jason no problem, I was able to read something :-) Please check that >>>> both the interfaces are configured with a single RSS queue (take a look at >>>> /proc/net/pf_ring/dev/eth2/info) >>>> >>>> Alfredo >>>> >>>> On 08 Jun 2016, at 21:09, Jason Lixfeld <[email protected]> wrote: >>>> >>>> My gosh! I’m so sorry for the way this is formatted. My mailer insists >>>> that this message was sent in plain-text, not whatever the heck this is! >>>> >>>> I’m sorry this is so impossible to read :( >>>> >>>> On Jun 8, 2016, at 3:05 PM, [email protected] wrote: >>>> >>>> Hello, >>>> >>>> My first run through with poring. :) >>>> >>>> I’ve compiled the zc variant of pfring in an attempt to get linerate >>>> between two of the ports which are looped together. >>>> >>>> The NIC is a 6 port 82599 based one. >>>> >>>> root@pgen:/home/jlixfeld/PF_RING/drivers/intel/ixgbe/ixgbe-4.1.5-zc/src# >>>> ./load_driver.sh irqbalance: no process found Configuring eth6 IFACE CORE >>>> MASK -> FILE ======================= eth6 0 1 -> /proc/irq/87/smp_affinity >>>> Configuring eth7 IFACE CORE MASK -> FILE ======================= eth7 0 1 >>>> -> /proc/irq/89/smp_affinity Configuring eth2 IFACE CORE MASK -> FILE >>>> ======================= eth2 0 1 -> /proc/irq/96/smp_affinity eth2 1 2 -> >>>> /proc/irq/97/smp_affinity eth2 2 4 -> /proc/irq/98/smp_affinity eth2 3 8 >>>> -> /proc/irq/99/smp_affinity eth2 4 10 -> /proc/irq/100/smp_affinity eth2 >>>> 5 20 -> /proc/irq/101/smp_affinity eth2 6 40 -> /proc/irq/102/smp_affinity >>>> eth2 7 80 -> /proc/irq/103/smp_affinity eth2 8 100 -> >>>> /proc/irq/104/smp_affinity eth2 9 200 -> /proc/irq/105/smp_affinity eth2 >>>> 10 400 -> /proc/irq/106/smp_affinity eth2 11 800 -> >>>> /proc/irq/107/smp_affinity Configuring eth3 IFACE CORE MASK -> FILE >>>> ======================= eth3 0 1 -> /proc/irq/109/smp_affinity eth3 1 2 -> >>>> /proc/irq/110/smp_affinity eth3 2 4 -> /proc/irq/111/smp_affinity eth3 3 8 >>>> -> /proc/irq/112/smp_affinity eth3 4 10 -> /proc/irq/113/smp_affinity eth3 >>>> 5 20 -> /proc/irq/114/smp_affinity eth3 6 40 -> /proc/irq/115/smp_affinity >>>> eth3 7 80 -> /proc/irq/116/smp_affinity eth3 8 100 -> >>>> /proc/irq/117/smp_affinity eth3 9 200 -> /proc/irq/118/smp_affinity eth3 >>>> 10 400 -> /proc/irq/119/smp_affinity eth3 11 800 -> >>>> /proc/irq/120/smp_affinity Configuring eth4 IFACE CORE MASK -> FILE >>>> ======================= eth4 0 1 -> /proc/irq/91/smp_affinity Configuring >>>> eth5 IFACE CORE MASK -> FILE ======================= eth5 0 1 -> >>>> /proc/irq/93/smp_affinity >>>> root@pgen:/home/jlixfeld/PF_RING/drivers/intel/ixgbe/ixgbe-4.1.5-zc/src# >>>> >>>> My issue is that I only get 10Gbps in one direction. If zc:eth7 is the >>>> sender, zc:eth2 only sees Rx @ 0.54Gbps: >>>> >>>> root@pgen:/home/jlixfeld/PF_RING/userland/examples_zc# ./zsend -i zc:eth7 >>>> -c 2 >>>> >>>> Absolute Stats: 111'707'057 pkts – 9'383'392'788 bytes Actual Stats: >>>> 13'983'520.42 pps – 9.40 Gbps [1133946996 bytes / 1.0 sec] >>>> >>>> root@pgen:/home/jlixfeld/PF_RING/userland/examples_zc# ./zcount -i zc:eth2 >>>> -c 1 >>>> >>>> Absolute Stats: 5'699'096 pkts (33'135'445 drops) – 478'724'064 bytes >>>> Actual Stats: 802'982.00 pps (4'629'316.93 drops) – 0.54 Gbps >>>> >>>> But, if zc:eth2 is the sender, zc:eth7 sees rates more in-line with what >>>> zc:eth2 is sending. >>>> >>>> root@pgen:/home/jlixfeld/PF_RING/userland/examples_zc# ./zsend -i zc:eth2 >>>> -c 2 >>>> >>>> Absolute Stats: 28'285'274 pkts – 2'375'963'016 bytes Actual Stats: >>>> 14'114'355.24 pps – 9.48 Gbps [1185800280 bytes / 1.0 sec] >>>> >>>> root@pgen:/home/jlixfeld/PF_RING/userland/examples_zc# ./zcount -i zc:eth7 >>>> -c 1 >>>> >>>> Absolute Stats: 28'007'460 pkts (0 drops) – 2'352'626'640 bytes Actual >>>> Stats: 14'044'642.54 pps (0.00 drops) – 9.44 Gbps >>>> >>>> I’ve done some reading, but I haven’t found anything that has pointed me >>>> towards a possible reason why this is happening. I’m wondering if anyone >>>> has any thoughts? >>>> >>>> Thanks! >>>> >>>> _____________________________________ Ntop-misc mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>> >>>> _______________________________________ Ntop-misc mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>> >>>> _______________________________________ Ntop-misc mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>> >>>> _________________________________________ Ntop-misc mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>> >>>> _________________________________________ Ntop-misc mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>> >>>> ___________________________________________ Ntop-misc mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>> >>>> ___________________________________________ Ntop-misc mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>> >>>> _____________________________________________ Ntop-misc mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>> >>>> _____________________________________________ Ntop-misc mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>> >>>> >>>> _______________________________________________ >>>> Ntop-misc mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>> >>> _______________________________________________ >>> Ntop-misc mailing list >>> [email protected] >>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc > > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
