nothing obvious on my mind... -- Damjan
> On 30 Oct 2018, at 13:25, Xuekun <[email protected]> wrote: > > Yes, I know. Iām also doing the performance comparison between tapv2 vs. > af_packet. > I just found below strange problem, and would like to know why? š > > Thanks. > > <>From: Damjan Marion <[email protected]> > Sent: Tuesday, October 30, 2018 8:18 PM > To: Hu, Xuekun <[email protected]> > Cc: [email protected] > Subject: Re: [vpp-dev] AF_PACKET performance question > > > If you are interested in performance, i suggest using tap interface instead > of af_packet. > With tap (by using vhost-net backend) there is no memcpy involved, kernel > puts buffer data straight into VPP buffers... > > -- > Damjan > > > On 30 Oct 2018, at 13:07, Xuekun <[email protected] > <mailto:[email protected]>> wrote: > > Hi, All > > Recently I found a big performance difference (~5Gbps vs. ~300Mbps by iperf3) > between two settings, which AF_PACKET related. > > The testing topology: > Two servers are directly connected through 10G link; One is running VPP on > the 10G NIC and AF_PACKET interface. And the iperf client (TCP) is sending > packets to the AF_PACKET interface. The other is running iperf server. > > The startup.conf settings: > ... > cpu { > main-core 18 > corelist-workers 19,20,21,22 > } > dev 0000:af:00.1 { > workers 0 > # workers 1 > } > .... > > The only difference of two testing is the "workers" setting, one (test1) is > setting workers to 0, the other (test2) is setting workers to 1. And I found > the iperf3 perf is ~5Gbps for test1, while only ~300Mbps for test2. > > So I run "show run" to see what's happening and what's the difference. > > Below is the output of "show run" that I only paste the working core. > > Test1 (workers to 0) > Thread 1 vpp_wk_0 (lcore 19) > Time 1.2, average vectors/node 32.01, last 128 main loops .05 per node 1.00 > vector rates in 4.7033e5, out 4.7033e5, drop 0.0000e0, punt 0.0000e0 > Name State > Calls Vectors Suspends Clocks Vectors/Call > TenGigabitEthernetaf/0/1-outpu active 8134 528460 > 0 1.32e1 64.97 > TenGigabitEthernetaf/0/1-tx active 8134 > 528460 0 1.32e2 64.97 > af-packet-input interrupt wa 184542 > 528460 0 8.31e2 2.86 > dpdk-input polling > 038663 40390 0 5.87e3 .02 > ethernet-input active > 8134 528460 0 2.91e1 64.97 > host-vpp0-output active 11729 > 40390 0 8.22e1 3.44 > host-vpp0-tx active > 11729 40390 0 4.37e4 3.44 > ip4-input active > 8134 528460 0 3.89e1 64.97 > ip4-input-no-checksum active 11729 > 40390 0 2.03e2 3.44 > ip4-lookup active > 19863 568850 0 3.68e1 28.64 > ip4-rewrite active > 19863 568850 0 3.62e1 28.64 > > Test2 (workers to 1) > Thread 1 vpp_wk_0 (lcore 19) > Time 1.3, average vectors/node 256.00, last 128 main loops 0.00 per node 0.00 > vector rates in 2.4971e4, out 2.4971e4, drop 0.0000e0, punt 0.0000e0 > Name State Calls > Vectors Suspends Clocks Vectors/Call > TenGigabitEthernetaf/0/1-outpu active 130 33280 > 0 9.40e0 256.00 > TenGigabitEthernetaf/0/1-tx active 130 > 33280 0 1.17e2 256.00 > af-packet-input interrupt wa 8146 > 33280 0 6.34e2 4.09 > ethernet-input active 130 > 33280 0 2.84e1 256.00 > ip4-input active > 130 33280 0 3.56e1 256.00 > ip4-lookup active 130 > 33280 0 3.13e1 256.00 > ip4-rewrite active 130 > 33280 0 2.97e1 256.00 > unix-epoll-input polling 9317 > 0 0 4.62e5 0.00 > --------------- > Thread 2 vpp_wk_1 (lcore 20) > Time 1.3, average vectors/node 1.44, last 128 main loops 0.00 per node 0.00 > vector rates in 5.3198e2, out 5.3198e2, drop 0.0000e0, punt 0.0000e0 > Name State Calls > Vectors Suspends Clocks Vectors/Call > dpdk-input polling 18232975 > 709 0 2.29e6 0.00 > host-vpp0-output active 492 > 709 0 1.91e2 1.44 > host-vpp0-tx active 492 > 709 0 1.51e5 1.44 > ip4-input-no-checksum active 492 > 709 0 3.20e2 1.44 > ip4-lookup active 492 > 709 0 2.16e2 1.44 > ip4-rewrite active 492 > 709 0 1.88e2 1.44 > > My observation is : For test1, the dpdk-input and af-packet-input are running > in the same core (core19), while for test2, dpdk-input (core 20) and > af-packet-input (core19) are running in the different core. > > My question: Is it the reason for such huge performance difference? If yes, > seems like af-packet-input (by default) is handling by the first worker core. > Any optimization method to bind the af-packet-input and dpdk-input in the > same core? > > My VPP config is simple: > set interface state TenGigabitEthernetaf/0/1 up > set interface ip address TenGigabitEthernetaf/0/1 10.1.1.1/24 > set interface promiscuous on TenGigabitEthernetaf/0/1 > create host-interface name vpp0 > set interface state host-vpp0 up > set interface ip address host-vpp0 10.1.2.1/24 > set interface promiscuous on host-vpp0 > > Many thanks. > > Thx, Xuekun > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#11030): https://lists.fd.io/g/vpp-dev/message/11030 > <https://lists.fd.io/g/vpp-dev/message/11030> > Mute This Topic: https://lists.fd.io/mt/27791437/675642 > <https://lists.fd.io/mt/27791437/675642> > Group Owner: [email protected] <mailto:[email protected]> > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub > <https://lists.fd.io/g/vpp-dev/unsub> [[email protected] > <mailto:[email protected]>] > -=-=-=-=-=-=-=-=-=-=-=- > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#11033): https://lists.fd.io/g/vpp-dev/message/11033 > <https://lists.fd.io/g/vpp-dev/message/11033> > Mute This Topic: https://lists.fd.io/mt/27791437/675642 > <https://lists.fd.io/mt/27791437/675642> > Group Owner: [email protected] <mailto:[email protected]> > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub > <https://lists.fd.io/g/vpp-dev/unsub> [[email protected] > <mailto:[email protected]>] > -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11034): https://lists.fd.io/g/vpp-dev/message/11034 Mute This Topic: https://lists.fd.io/mt/27791437/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
