In VPP packet stays on the same core where it is received in majority of cases.
Handing over packet to different core is performance expensive process and we 
are trying to avoid it.
You likely need to utilise RSS on rx side to equally load your cores,
but in this specific case VPP is not overloaded, your vector rate is ~2....

-- 
Damjan

> On 1 Aug 2018, at 20:22, chakravarthy.arise...@viasat.com wrote:
> 
> Hi Damjan,
> 
> Thanks for your feedback. I'm running the test in AWS instances. Thus, I have 
> got only VFs. I do not have access to PF. So, I'm trying to get help from AWS 
> to find out.
> Once I get the info, I'll post it over here. In the mean time, I looked at 
> the counters that you suggested me to focus on. It looks like the packets are 
> scheduled on only one core in transmit direction. Is there a way to change?
> 
> I have 3 dedicated cores (1 main core thread for stats/mgmt and 2 cores for 
> the worker threads). All the Tx queues are pinned to worker thread 1. So, 
> worker thread 2 is not used for transmit path at all. Is there way to spread 
> the transmit queues across the threads?
> 
> Thanks
> Chakri
> 
> vpp# sh threads
> ID     Name                Type            LWP    Sched Policy (Priority)  
> lcore  Core   Socket State
> 0      vpp_main                              1733    other (0)                
> 1      1      0
> 1      vpp_wk_0            workers     1745    other (0)                2     
>  2      0
> 2      vpp_wk_1            workers     1746    other (0)                3     
>  3      0
> 3                                   stats         1747    other (0)           
>      0      0      0
> 
> vpp# sh run
> Thread 0 vpp_main (lcore 1)
> Time 5125.9, average vectors/node 0.00, last 128 main loops 0.00 per node 0.00
>   vector rates in 0.0000e0, out 0.0000e0, drop 0.0000e0, punt 0.0000e0
>              Name                 State         Calls          Vectors        
> Suspends         Clocks       Vectors/Call
> api-rx-from-ring                any wait                 0               0    
>          364          1.19e4            0.00
> cdp-process                     any wait                 0               0    
>          992          1.98e3            0.00
> dhcp-client-process             any wait                 0               0    
>           51          3.41e3            0.00
> dns-resolver-process            any wait                 0               0    
>            5          4.06e3            0.00
> dpdk-process                    any wait                 0               0    
>         1709          5.13e4            0.00
> fib-walk                        any wait                 0               0    
>         2563          1.37e3            0.00
> ikev2-manager-process           any wait                 0               0    
>         5124          7.25e2            0.00
> ip-route-resolver-process       any wait                 0               0    
>           51          2.64e3            0.00
> ip4-reassembly-expire-walk      any wait                 0               0    
>          513          3.85e3            0.00
> ip6-icmp-neighbor-discovery-ev  any wait                 0               0    
>         5124          6.92e2            0.00
> ip6-reassembly-expire-walk      any wait                 0               0    
>          513          3.84e3            0.00
> lisp-retry-service              any wait                 0               0    
>         2563          1.57e3            0.00
> memif-process                   any wait                 0               0    
>         1709          2.10e3            0.00
> rd-cp-process                   any wait                 0               0    
>    237212380          3.21e2            0.00
> unix-cli-local:17                active                  0               0    
>          580          2.05e5            0.00
> unix-epoll-input                 polling          96172305               0    
>            0          1.19e4            0.00
> vpe-oam-process                 any wait                 0               0    
>         2513          1.23e3            0.00
> ---------------
> Thread 1 vpp_wk_0 (lcore 2)
> Time 5125.9, average vectors/node 4.82, last 128 main loops 0.00 per node 0.00
>   vector rates in 9.5578e3, out 8.4052e3, drop 0.0000e0, punt 0.0000e0
>              Name                 State         Calls          Vectors        
> Suspends         Clocks       Vectors/Call
> VirtualFunctionEthernet0/6/0-o   active                 91              91    
>            0          8.59e2            1.00
> VirtualFunctionEthernet0/6/0-t   active                 91              91    
>            0          2.82e3            1.00
> VirtualFunctionEthernet0/7/0-o   active            5334164        32661561    
>            0          4.33e1            6.12
> VirtualFunctionEthernet0/7/0-t   active            5334164        26753703    
>            0          3.83e2            5.02
> arp-input                        active                182             182    
>            0          7.25e3            1.00
> dpdk-input                       polling       16550217513        16330917    
>            0          4.05e5            0.00
> ethernet-input                   active            5334255        32661652    
>            0          7.97e1            6.12
> interface-output                 active                182             182    
>            0          6.58e2            1.00
> ip4-input                        active            4685453        16330735    
>            0          9.48e1            3.49
> ip4-load-balance                 active            5334073        32661470    
>            0          4.85e1            6.12
> ip4-local                        active            4685453        16330735    
>            0          9.92e1            3.49
> ip4-lookup                       active            4685453        16330735    
>            0          1.05e2            3.49
> ip4-rewrite                      active            5334073        32661470    
>            0          5.57e1            6.12
> ip4-udp-lookup                   active            4685453        16330735    
>            0          8.96e1            3.49
> l2-fwd                           active           10019526        48992205    
>            0          5.56e1            4.89
> l2-input                         active           10019526        48992205    
>            0          6.03e1            4.89
> l2-learn                         active           10019526        48992205    
>            0          6.75e1            4.89
> l2-output                        active           10019526        48992205    
>            0          6.16e1            4.89
> memif-input                      polling       16550217513        32661470    
>            0          2.33e5            0.00
> unix-epoll-input                 polling           1817493               0    
>            0          1.18e4            0.00
> vxlan4-encap                     active            5334073        32661470    
>            0          1.09e2            6.12
> vxlan4-input                     active            4685453        16330735    
>            0          1.19e2            3.49
> memif1/1-output                  active            4685453        16330735    
>            0          1.34e2            3.49
> memif1/1-tx                      active            4685453        16330735    
>            0          1.53e3            3.49
> ---------------
> Thread 2 vpp_wk_1 (lcore 3)
> Time 5125.9, average vectors/node 1.67, last 128 main loops 0.00 per node 0.00
>   vector rates in 3.1859e3, out 3.1859e3, drop 0.0000e0, punt 0.0000e0
>              Name                 State         Calls          Vectors        
> Suspends         Clocks       Vectors/Call
> dpdk-input                       polling       16679496489        16330735    
>            0          4.24e5            0.00
> ip4-input                        active            9785099        16330735    
>            0          1.47e2            1.67
> ip4-local                        active            9785099        16330735    
>            0          1.33e2            1.67
> ip4-lookup                       active            9785099        16330735    
>            0          1.25e2            1.67
> ip4-udp-lookup                   active            9785099        16330735    
>            0          1.18e2            1.67
> l2-fwd                           active            9785099        16330735    
>            0          1.06e2            1.67
> l2-input                         active            9785099        16330735    
>            0          1.32e2            1.67
> l2-learn                         active            9785099        16330735    
>            0          1.31e2            1.67
> l2-output                        active            9785099        16330735    
>            0          9.05e1            1.67
> memif-input                      polling       16679496489               0    
>            0          4.38e2            0.00
> unix-epoll-input                 polling           1130721               0    
>            0          1.14e4            0.00
> vxlan4-input                     active            9785099        16330735    
>            0          1.45e2            1.67
> memif1/1-output                  active            9785099        16330735    
>            0          1.01e2            1.67
> memif1/1-tx                      active            9785099        16330735    
>            0          2.09e3            1.67
> 
> vpp# sh hardware                                                              
>                                                 [80/1816]
>               Name                Idx   Link  Hardware
> VirtualFunctionEthernet0/6/0       1     up   VirtualFunctionEthernet0/6/0
>   Ethernet address 06:3a:20:ff:aa:d0
>   AWS ENA VF
>     carrier up full duplex speed 10000 mtu 9216
>     rx queues 2, rx desc 1024, tx queues 3, tx desc 1024
>     cpu socket 0
>  
>     tx frames ok                                           2
>     tx bytes ok                                           84
>     rx frames ok                                    21146632
>     rx bytes ok                                  33665435044
>     extended stats:
>       rx good packets                               21146632
>       tx good packets                                      2
>       rx good bytes                              33665435044
>       tx good bytes                                       84
> VirtualFunctionEthernet0/7/0       2     up   VirtualFunctionEthernet0/7/0
>   Ethernet address 06:90:5e:ca:8f:6c
>   AWS ENA VF
>     carrier up full duplex speed 10000 mtu 9216
>     rx queues 2, rx desc 1024, tx queues 3, tx desc 1024
>     cpu socket 0
>  
>     tx frames ok                                    17322383
>     tx bytes ok                                  27577230636
>     rx frames ok                                           2
>     rx bytes ok                                           84
>     extended stats:
>       rx good packets                                      2
>       tx good packets                               17322383
>       rx good bytes                                       84
>       tx good bytes                              27577230636
> local0                             0    down  local0
>   local
> loop1                              3     up   loop1
>   Ethernet address de:ad:00:00:00:01
> loop2                              5     up   loop2
>   Ethernet address de:ad:00:00:00:02
> memif1/1                           7     up   memif1/1
>   Ethernet address 02:fe:95:70:02:bc
>   MEMIF interface
>      instance 0
> memif2/2                           8     up   memif2/2
>   Ethernet address 02:fe:6d:04:8f:40
>   MEMIF interface
>      instance 1
> vxlan_tunnel1                      4     up   vxlan_tunnel1
>   VXLAN
> vxlan_tunnel2                      6     up   vxlan_tunnel2
>   VXLAN
> 
> vpp# show int
>               Name               Idx       State          Counter          
> Count
> VirtualFunctionEthernet0/6/0      1         up       rx packets              
> 21146633
>                                                      rx bytes             
> 33665435086
>                                                      tx packets               
>       3
>                                                      tx bytes                 
>     126
>                                                      ip4                     
> 21146630
> VirtualFunctionEthernet0/7/0      2         up       rx packets               
>       3
>                                                      rx bytes                 
>     126
>                                                      tx packets              
> 21146633
>                                                      tx bytes             
> 33665435086
>                                                      tx-error                 
> 3824249
> local0                            0         up
> loop1                             3         up
> loop2                             5         up
> memif1/1                          7         up       tx packets              
> 21146630
>                                                      tx bytes             
> 32608103460
> memif2/2                          8         up       rx packets              
> 21146630
>                                                      rx bytes             
> 32608103460
> vxlan_tunnel1                     4         up       rx packets              
> 21146630
>                                                      rx bytes             
> 32608103460
> vxlan_tunnel2                     6         up       tx packets              
> 21146630
>                                                      tx bytes             
> 33369382140
> 
> vpp# show error
>    Count                    Node                  Reason
>  120878944              vxlan4-input              good packets decapsulated
>  241757861              vxlan4-encap              good packets encapsulated
>  362636805                l2-output               L2 output packets
>  362636805                l2-learn                L2 learn packets
>  362636805                l2-input                L2 input packets
>       2616                arp-input               ARP replies sent
>          8                arp-input               ARP probe or announcement 
> dropped
>   18841192     VirtualFunctionEthernet0/7/0-tx    Tx packet drops (dpdk tx 
> failure)
>  120878917              vxlan4-input              good packets decapsulated
>  120878917                l2-output               L2 output packets
>  120878917                l2-learn                L2 learn packets
>  120878917                l2-input                L2 input packets
> 
> Startup config snippet
> =================
>         dev 0000:00:06.0 {
>                 num-rx-queues 2
>                 num-rx-desc 1024
>                 num-tx-desc 1024
>         }
>  
>         dev 0000:00:07.0 {
>                 num-rx-queues 2
>                 num-rx-desc 1024
>                 num-tx-desc 1024
>         }
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> 
> View/Reply Online (#10012): https://lists.fd.io/g/vpp-dev/message/10012
> Mute This Topic: https://lists.fd.io/mt/23982730/675642
> Group Owner: vpp-dev+ow...@lists.fd.io
> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [dmar...@me.com]
> -=-=-=-=-=-=-=-=-=-=-=-

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#10014): https://lists.fd.io/g/vpp-dev/message/10014
Mute This Topic: https://lists.fd.io/mt/23982730/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Reply via email to