In VPP packet stays on the same core where it is received in majority of cases.
Handing over packet to different core is performance expensive process and we
are trying to avoid it.
You likely need to utilise RSS on rx side to equally load your cores,
but in this specific case VPP is not overloaded, your vector rate is ~2....
--
Damjan
> On 1 Aug 2018, at 20:22, chakravarthy.arise...@viasat.com wrote:
>
> Hi Damjan,
>
> Thanks for your feedback. I'm running the test in AWS instances. Thus, I have
> got only VFs. I do not have access to PF. So, I'm trying to get help from AWS
> to find out.
> Once I get the info, I'll post it over here. In the mean time, I looked at
> the counters that you suggested me to focus on. It looks like the packets are
> scheduled on only one core in transmit direction. Is there a way to change?
>
> I have 3 dedicated cores (1 main core thread for stats/mgmt and 2 cores for
> the worker threads). All the Tx queues are pinned to worker thread 1. So,
> worker thread 2 is not used for transmit path at all. Is there way to spread
> the transmit queues across the threads?
>
> Thanks
> Chakri
>
> vpp# sh threads
> ID Name Type LWP Sched Policy (Priority)
> lcore Core Socket State
> 0 vpp_main 1733 other (0)
> 1 1 0
> 1 vpp_wk_0 workers 1745 other (0) 2
> 2 0
> 2 vpp_wk_1 workers 1746 other (0) 3
> 3 0
> 3 stats 1747 other (0)
> 0 0 0
>
> vpp# sh run
> Thread 0 vpp_main (lcore 1)
> Time 5125.9, average vectors/node 0.00, last 128 main loops 0.00 per node 0.00
> vector rates in 0.0000e0, out 0.0000e0, drop 0.0000e0, punt 0.0000e0
> Name State Calls Vectors
> Suspends Clocks Vectors/Call
> api-rx-from-ring any wait 0 0
> 364 1.19e4 0.00
> cdp-process any wait 0 0
> 992 1.98e3 0.00
> dhcp-client-process any wait 0 0
> 51 3.41e3 0.00
> dns-resolver-process any wait 0 0
> 5 4.06e3 0.00
> dpdk-process any wait 0 0
> 1709 5.13e4 0.00
> fib-walk any wait 0 0
> 2563 1.37e3 0.00
> ikev2-manager-process any wait 0 0
> 5124 7.25e2 0.00
> ip-route-resolver-process any wait 0 0
> 51 2.64e3 0.00
> ip4-reassembly-expire-walk any wait 0 0
> 513 3.85e3 0.00
> ip6-icmp-neighbor-discovery-ev any wait 0 0
> 5124 6.92e2 0.00
> ip6-reassembly-expire-walk any wait 0 0
> 513 3.84e3 0.00
> lisp-retry-service any wait 0 0
> 2563 1.57e3 0.00
> memif-process any wait 0 0
> 1709 2.10e3 0.00
> rd-cp-process any wait 0 0
> 237212380 3.21e2 0.00
> unix-cli-local:17 active 0 0
> 580 2.05e5 0.00
> unix-epoll-input polling 96172305 0
> 0 1.19e4 0.00
> vpe-oam-process any wait 0 0
> 2513 1.23e3 0.00
> ---------------
> Thread 1 vpp_wk_0 (lcore 2)
> Time 5125.9, average vectors/node 4.82, last 128 main loops 0.00 per node 0.00
> vector rates in 9.5578e3, out 8.4052e3, drop 0.0000e0, punt 0.0000e0
> Name State Calls Vectors
> Suspends Clocks Vectors/Call
> VirtualFunctionEthernet0/6/0-o active 91 91
> 0 8.59e2 1.00
> VirtualFunctionEthernet0/6/0-t active 91 91
> 0 2.82e3 1.00
> VirtualFunctionEthernet0/7/0-o active 5334164 32661561
> 0 4.33e1 6.12
> VirtualFunctionEthernet0/7/0-t active 5334164 26753703
> 0 3.83e2 5.02
> arp-input active 182 182
> 0 7.25e3 1.00
> dpdk-input polling 16550217513 16330917
> 0 4.05e5 0.00
> ethernet-input active 5334255 32661652
> 0 7.97e1 6.12
> interface-output active 182 182
> 0 6.58e2 1.00
> ip4-input active 4685453 16330735
> 0 9.48e1 3.49
> ip4-load-balance active 5334073 32661470
> 0 4.85e1 6.12
> ip4-local active 4685453 16330735
> 0 9.92e1 3.49
> ip4-lookup active 4685453 16330735
> 0 1.05e2 3.49
> ip4-rewrite active 5334073 32661470
> 0 5.57e1 6.12
> ip4-udp-lookup active 4685453 16330735
> 0 8.96e1 3.49
> l2-fwd active 10019526 48992205
> 0 5.56e1 4.89
> l2-input active 10019526 48992205
> 0 6.03e1 4.89
> l2-learn active 10019526 48992205
> 0 6.75e1 4.89
> l2-output active 10019526 48992205
> 0 6.16e1 4.89
> memif-input polling 16550217513 32661470
> 0 2.33e5 0.00
> unix-epoll-input polling 1817493 0
> 0 1.18e4 0.00
> vxlan4-encap active 5334073 32661470
> 0 1.09e2 6.12
> vxlan4-input active 4685453 16330735
> 0 1.19e2 3.49
> memif1/1-output active 4685453 16330735
> 0 1.34e2 3.49
> memif1/1-tx active 4685453 16330735
> 0 1.53e3 3.49
> ---------------
> Thread 2 vpp_wk_1 (lcore 3)
> Time 5125.9, average vectors/node 1.67, last 128 main loops 0.00 per node 0.00
> vector rates in 3.1859e3, out 3.1859e3, drop 0.0000e0, punt 0.0000e0
> Name State Calls Vectors
> Suspends Clocks Vectors/Call
> dpdk-input polling 16679496489 16330735
> 0 4.24e5 0.00
> ip4-input active 9785099 16330735
> 0 1.47e2 1.67
> ip4-local active 9785099 16330735
> 0 1.33e2 1.67
> ip4-lookup active 9785099 16330735
> 0 1.25e2 1.67
> ip4-udp-lookup active 9785099 16330735
> 0 1.18e2 1.67
> l2-fwd active 9785099 16330735
> 0 1.06e2 1.67
> l2-input active 9785099 16330735
> 0 1.32e2 1.67
> l2-learn active 9785099 16330735
> 0 1.31e2 1.67
> l2-output active 9785099 16330735
> 0 9.05e1 1.67
> memif-input polling 16679496489 0
> 0 4.38e2 0.00
> unix-epoll-input polling 1130721 0
> 0 1.14e4 0.00
> vxlan4-input active 9785099 16330735
> 0 1.45e2 1.67
> memif1/1-output active 9785099 16330735
> 0 1.01e2 1.67
> memif1/1-tx active 9785099 16330735
> 0 2.09e3 1.67
>
> vpp# sh hardware
> [80/1816]
> Name Idx Link Hardware
> VirtualFunctionEthernet0/6/0 1 up VirtualFunctionEthernet0/6/0
> Ethernet address 06:3a:20:ff:aa:d0
> AWS ENA VF
> carrier up full duplex speed 10000 mtu 9216
> rx queues 2, rx desc 1024, tx queues 3, tx desc 1024
> cpu socket 0
>
> tx frames ok 2
> tx bytes ok 84
> rx frames ok 21146632
> rx bytes ok 33665435044
> extended stats:
> rx good packets 21146632
> tx good packets 2
> rx good bytes 33665435044
> tx good bytes 84
> VirtualFunctionEthernet0/7/0 2 up VirtualFunctionEthernet0/7/0
> Ethernet address 06:90:5e:ca:8f:6c
> AWS ENA VF
> carrier up full duplex speed 10000 mtu 9216
> rx queues 2, rx desc 1024, tx queues 3, tx desc 1024
> cpu socket 0
>
> tx frames ok 17322383
> tx bytes ok 27577230636
> rx frames ok 2
> rx bytes ok 84
> extended stats:
> rx good packets 2
> tx good packets 17322383
> rx good bytes 84
> tx good bytes 27577230636
> local0 0 down local0
> local
> loop1 3 up loop1
> Ethernet address de:ad:00:00:00:01
> loop2 5 up loop2
> Ethernet address de:ad:00:00:00:02
> memif1/1 7 up memif1/1
> Ethernet address 02:fe:95:70:02:bc
> MEMIF interface
> instance 0
> memif2/2 8 up memif2/2
> Ethernet address 02:fe:6d:04:8f:40
> MEMIF interface
> instance 1
> vxlan_tunnel1 4 up vxlan_tunnel1
> VXLAN
> vxlan_tunnel2 6 up vxlan_tunnel2
> VXLAN
>
> vpp# show int
> Name Idx State Counter
> Count
> VirtualFunctionEthernet0/6/0 1 up rx packets
> 21146633
> rx bytes
> 33665435086
> tx packets
> 3
> tx bytes
> 126
> ip4
> 21146630
> VirtualFunctionEthernet0/7/0 2 up rx packets
> 3
> rx bytes
> 126
> tx packets
> 21146633
> tx bytes
> 33665435086
> tx-error
> 3824249
> local0 0 up
> loop1 3 up
> loop2 5 up
> memif1/1 7 up tx packets
> 21146630
> tx bytes
> 32608103460
> memif2/2 8 up rx packets
> 21146630
> rx bytes
> 32608103460
> vxlan_tunnel1 4 up rx packets
> 21146630
> rx bytes
> 32608103460
> vxlan_tunnel2 6 up tx packets
> 21146630
> tx bytes
> 33369382140
>
> vpp# show error
> Count Node Reason
> 120878944 vxlan4-input good packets decapsulated
> 241757861 vxlan4-encap good packets encapsulated
> 362636805 l2-output L2 output packets
> 362636805 l2-learn L2 learn packets
> 362636805 l2-input L2 input packets
> 2616 arp-input ARP replies sent
> 8 arp-input ARP probe or announcement
> dropped
> 18841192 VirtualFunctionEthernet0/7/0-tx Tx packet drops (dpdk tx
> failure)
> 120878917 vxlan4-input good packets decapsulated
> 120878917 l2-output L2 output packets
> 120878917 l2-learn L2 learn packets
> 120878917 l2-input L2 input packets
>
> Startup config snippet
> =================
> dev 0000:00:06.0 {
> num-rx-queues 2
> num-rx-desc 1024
> num-tx-desc 1024
> }
>
> dev 0000:00:07.0 {
> num-rx-queues 2
> num-rx-desc 1024
> num-tx-desc 1024
> }
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
>
> View/Reply Online (#10012): https://lists.fd.io/g/vpp-dev/message/10012
> Mute This Topic: https://lists.fd.io/mt/23982730/675642
> Group Owner: vpp-dev+ow...@lists.fd.io
> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [dmar...@me.com]
> -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#10014): https://lists.fd.io/g/vpp-dev/message/10014
Mute This Topic: https://lists.fd.io/mt/23982730/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
