On 05.11.2014 19:18, Evandro Nunes wrote: > On Wed, Nov 5, 2014 at 1:52 PM, Andrey V. Elsukov <bu7c...@yandex.ru> wrote: > >> On 05.11.2014 18:39, George Neville-Neil wrote: >>> Howdy, >>> >>> Last night (Pacific Time) I committed a change so that GENERIC, on HEAD >>> has the netmap >>> device enabled. This is to increase the breadth of our testing of that >>> feature prior >>> to the release of FreeBSD 11. >>> >>> In two weeks I will enable IPSec by default, again in preparation for 11. >> >> Hi, >> >> recently we did some IP forwarding tests and the GENERIC kernel is >> several times faster than GENERIC+IPSEC. Even when IPSEC has no SA. >> >> I didn't do test on vanilla kernel, but our kernel is able forward >> IPv4/IPv6 on rate close to 8.6 Mpps. The same kernel compiled with IPSEC >> can forward only 180 kpps. I think this problem should be solved before >> enabling it in GENERIC. >> > > this forward rate you mention is related to netmap? or usual > forwarding/fastforwarding? this is a huge number, do you mind sharing your > dmesg output and top -PSH output so I can check for interrupt CPU usage and > other relevant stuff?
This is patched kernel without netmap and fastforwarding. We removed all lock contention on the forwarding path to be sure that it doesn't affect IPSEC. Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz (2200.05-MHz K8-class CPU) FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs FreeBSD/SMP: 2 package(s) x 8 core(s) x 2 SMT threads real memory = 68736253952 (65552 MB) avail memory = 66370662400 (63295 MB) ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.5.15> port 0x7020-0x703f mem 0xde680000-0xde6fffff,0xde704000-0xde707fff irq 32 at device 0.0 on pci4 ix0: Using MSIX interrupts with 16 vectors ix0: Ethernet address: 90:e2:ba:0d:73:54 ix0: PCI Express Bus: Speed 5.0GT/s Width x8 This is IPv6 forwarding test - 6 /64 prefixes each has 200 random addresses. They are routed between 6 vlans. # netstat -I ix0 -w 1 input (ix0) output packets errs idrops bytes packets errs bytes colls 8917043 0 0 571436864 8149880 0 522587200 0 8943391 0 0 571598336 8179318 0 525085504 0 8928155 0 0 571262144 8168254 0 522712192 0 8921342 0 937 571693504 8128132 0 521997184 0 8924322 0 0 571170048 8211500 0 520264320 0 8934564 0 0 571483584 8180040 0 524475264 0 8937039 0 0 571384640 8234779 0 525686080 0 8926528 0 0 571481728 8160380 0 524265920 0 8923160 0 0 571397248 8229839 0 522569408 0 8930070 0 1705 571594944 8216092 0 528481152 0 8916249 0 0 571294784 8184286 0 524399360 0 8937301 0 0 571391040 8221895 0 526383744 0 8927967 0 0 571613312 8164779 0 524997760 0 8936306 0 0 571251712 8167960 0 519575744 0 8922983 0 306 571430528 8216466 0 525893056 0 8916209 0 0 571434240 8202692 0 526046336 0 8945608 0 0 571426624 8265756 0 524815552 0 8925548 0 1045 571444480 8229681 0 530935232 0 8932145 0 0 571747200 8149710 0 523409536 0 8929339 0 0 571683200 8186790 0 520719040 0 8917697 0 0 571585152 8212635 0 525775680 0 # top -PSH last pid: 2788; load averages: 12.01, 4.76, 1.92 up 0+00:04:38 20:58:48 471 processes: 45 running, 344 sleeping, 82 waiting CPU 0: 0.0% user, 0.0% nice, 21.6% system, 68.2% interrupt, 10.2% idle CPU 1: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 2: 0.0% user, 0.0% nice, 2.7% system, 84.3% interrupt, 12.9% idle CPU 3: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 4: 0.0% user, 0.0% nice, 3.9% system, 86.7% interrupt, 9.4% idle CPU 5: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 6: 0.0% user, 0.0% nice, 5.5% system, 88.6% interrupt, 5.9% idle CPU 7: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 8: 0.0% user, 0.0% nice, 3.5% system, 90.2% interrupt, 6.3% idle CPU 9: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 10: 0.0% user, 0.0% nice, 3.1% system, 87.1% interrupt, 9.8% idle CPU 11: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 12: 0.0% user, 0.0% nice, 27.5% system, 62.0% interrupt, 10.6% idle CPU 13: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 14: 0.0% user, 0.0% nice, 6.3% system, 85.9% interrupt, 7.8% idle CPU 15: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 16: 0.0% user, 0.0% nice, 17.6% system, 79.6% interrupt, 2.7% idle CPU 17: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 18: 0.0% user, 0.0% nice, 2.4% system, 92.2% interrupt, 5.5% idle CPU 19: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 20: 0.0% user, 0.0% nice, 7.8% system, 86.7% interrupt, 5.5% idle CPU 21: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 22: 0.0% user, 0.0% nice, 6.3% system, 87.5% interrupt, 6.3% idle CPU 23: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 24: 0.0% user, 0.0% nice, 1.6% system, 89.4% interrupt, 9.0% idle CPU 25: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 26: 0.0% user, 0.0% nice, 2.0% system, 91.8% interrupt, 6.3% idle CPU 27: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 28: 0.0% user, 0.0% nice, 2.7% system, 87.8% interrupt, 9.4% idle CPU 29: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 30: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 31: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle Mem: 17M Active, 13M Inact, 716M Wired, 61G Free ARC: 28M Total, 6102K MFU, 20M MRU, 16K Anon, 373K Header, 1492K Other Swap: 16G Total, 16G Free # top -HSIzts1 last pid: 2805; load averages: 13.88, 7.92, 3.56 up 0+00:06:40 21:00:50 469 processes: 45 running, 342 sleeping, 82 waiting CPU: 0.0% user, 0.0% nice, 3.5% system, 39.3% interrupt, 57.1% idle Mem: 17M Active, 13M Inact, 716M Wired, 61G Free ARC: 28M Total, 6105K MFU, 20M MRU, 16K Anon, 373K Header, 1494K Other Swap: 16G Total, 16G Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12 root -92 - 0K 1504K CPU22 22 3:38 87.35% intr{irq293: ix0:que } 12 root -92 - 0K 1504K WAIT 26 3:41 86.57% intr{irq295: ix0:que } 12 root -92 - 0K 1504K CPU18 18 3:41 86.47% intr{irq291: ix0:que } 12 root -92 - 0K 1504K CPU28 28 3:41 86.47% intr{irq296: ix0:que } 12 root -92 - 0K 1504K CPU24 24 3:39 86.18% intr{irq294: ix0:que } 12 root -92 - 0K 1504K WAIT 4 3:35 86.18% intr{irq284: ix0:que } 12 root -92 - 0K 1504K CPU8 8 3:35 86.18% intr{irq286: ix0:que } 12 root -92 - 0K 1504K CPU6 6 3:33 85.60% intr{irq285: ix0:que } 12 root -92 - 0K 1504K CPU20 20 3:32 85.50% intr{irq292: ix0:que } 12 root -92 - 0K 1504K CPU10 10 3:32 84.86% intr{irq287: ix0:que } 12 root -92 - 0K 1504K CPU2 2 3:32 84.67% intr{irq283: ix0:que } 12 root -92 - 0K 1504K WAIT 14 3:31 84.38% intr{irq289: ix0:que } 12 root -92 - 0K 1504K CPU16 16 3:11 77.59% intr{irq290: ix0:que } 12 root -92 - 0K 1504K CPU0 0 2:42 64.70% intr{irq282: ix0:que } 12 root -92 - 0K 1504K CPU12 12 2:44 63.67% intr{irq288: ix0:que } 0 root -92 0 0K 4672K - 12 1:01 26.66% kernel{ix0 que} 0 root -92 0 0K 4672K - 0 0:58 23.00% kernel{ix0 que} 0 root -92 0 0K 4672K - 16 0:48 18.16% kernel{ix0 que} 0 root -92 0 0K 4672K - 20 0:20 7.86% kernel{ix0 que} 0 root -92 0 0K 4672K - 14 0:15 5.57% kernel{ix0 que} 0 root -92 0 0K 4672K CPU6 6 0:14 5.57% kernel{ix0 que} 0 root -92 0 0K 4672K - 22 0:12 4.59% kernel{ix0 que} 0 root -92 0 0K 4672K - 18 0:06 4.59% kernel{ix0 que} 0 root -92 0 0K 4672K - 28 0:06 4.49% kernel{ix0 que} 0 root -92 0 0K 4672K - 26 0:06 4.39% kernel{ix0 que} 0 root -92 0 0K 4672K - 24 0:05 4.05% kernel{ix0 que} 0 root -92 0 0K 4672K - 8 0:09 3.47% kernel{ix0 que} 0 root -92 0 0K 4672K - 4 0:08 3.17% kernel{ix0 que} 0 root -92 0 0K 4672K - 10 0:07 2.98% kernel{ix0 que} 0 root -92 0 0K 4672K - 2 0:05 2.39% kernel{ix0 que} # pmcstat -TS llc-misses -w10 PMC: [llc-misses] Samples: 3949 (100.0%) , 0 unresolved %SAMP IMAGE FUNCTION CALLERS 28.6 kernel ixgbe_rxeof ixgbe_msix_que:25.9 ixgbe_handle_que:2.8 14.7 kernel bcmp netisr_dispatch_src:13.5 rtalloc_fib_nolock:1.0 5.9 kernel mb_ctor_mbuf uma_zalloc_arg 5.8 kernel ixgbe_mq_start vlan_transmit 4.6 kernel _mtx_trylock ixgbe_mq_start 3.8 kernel ether_nh_input netisr_dispatch_src 2.6 kernel ether_input ixgbe_msix_que 2.4 kernel m_tag_delete_chain m_tag_copy_chain:1.2 uma_zfree_arg:0.9 2.4 kernel netisr_dispatch_src ixgbe_rxeof:1.6 ixgbe_msix_que:0.6 2.2 kernel cpu_search_highest cpu_search_highest 2.2 kernel ixgbe_txeof ixgbe_msix_que 2.1 kernel uma_zfree_arg m_freem 2.0 kernel netisr_dispatch ixgbe_msix_que:1.1 ixgbe_rxeof:0.7 2.0 kernel critical_exit uma_zfree_arg 1.9 kernel _thread_lock_flags ithread_loop:1.2 intr_event_schedule_thread:0.6 1.9 kernel _bus_dmamap_sync ixgbe_rxeof 1.3 kernel ixgbe_xmit ixgbe_mq_start_locked 0.9 kernel ixgbe_msix_que intr_event_execute_handlers -- WBR, Andrey V. Elsukov
signature.asc
Description: OpenPGP digital signature