We’ve seen 4.6Gbps IPsec on a single core of a C5 instance, and that’s 
throttled by the 5gbps in and out of the VPC “limit” courtesy of Amazon.  (The 
other 400Mbps is framing overhead.)
But here, obviously, “large packets” (we were using iperf3 on the tunnel).

Jim

> On Oct 5, 2018, at 6:38 AM, Dave Barach via Lists.Fd.Io 
> <[email protected]> wrote:
> 
> I’ll let others jump in on the tx [and rx] drops. At some level, it looks the 
> vfio device simply isn’t disposing of traffic rapidly enough. Perhaps too few 
> buffers have been configured, but I don’t think that’s the most likely 
> explanation.
>  
> The “show run” stats are not helpful because they include over 2000 seconds’s 
> worth of [mostly] dead airtime: please start traffic, “clear run”, wait a few 
> seconds, then “show run”.
>  
> You didn’t mention the packet size that you’re using. It wouldn’t shock me to 
> learn that a “10xGE capable instance” comes with a few stars in the margin: 
> perhaps “10xGE capable at or above 256 byte packets” or some such. 
>  
> HTH... Dave
>  
> From: [email protected] <mailto:[email protected]> <[email protected] 
> <mailto:[email protected]>> On Behalf Of Santosh Kumar Dornal
> Sent: Thursday, October 4, 2018 9:40 PM
> To: [email protected] <mailto:[email protected]>
> Subject: [vpp-dev] Tx packet drops - dpdk tx failure
>  
> Hi 
>  
> I am using VPP 18.04 on AWS (c5.xlarge instance with ena interfaces) and 
> running into Tx packet drops when pushing traffic over 2.5Gbps 
> (bidirectional). The instance is capable of doing 10gig. Can somebody help?
>  
> vpp# show version
> vpp v18.04-16~gca7a68e0 built by vagrant on debian9.localdomain at Wed Oct  3 
> 11:57:44 PDT 2018
> vpp# show errors
>    Count                    Node                  Reason
>         68                arp-input               ARP replies sent
>   82693102     VirtualFunctionEthernet0/6/0-tx    Tx packet drops (dpdk tx 
> failure)
>   83200033     VirtualFunctionEthernet0/7/0-tx    Tx packet drops (dpdk tx 
> failure)
> vpp#
>  
> vpp# show hardware-interfaces  detail
>               Name                Idx   Link  Hardware
> VirtualFunctionEthernet0/6/0       1     up   VirtualFunctionEthernet0/6/0
>   Ethernet address 02:d2:6c:6e:b0:1c
>   AWS ENA VF
>     carrier up full duplex speed 10000 mtu 9216
>     pci id:            device 1d0f:ec20 subsystem 0000:0000
>     pci address:       0000:00:06.00
>     max rx packet len: 9216
>     max num of queues: rx 8 tx 8
>     promiscuous:       unicast off all-multicast off
>     vlan offload:      strip off filter off qinq off
>     rx offload caps:
>     tx offload caps:   ipv4-cksum udp-cksum tcp-cksum
>     rss active:        ipv4-sctp ipv4-other ipv6-udp-ex ipv6-frag ipv6-tcp
>                        l2-payload
>     rss supported:     none
>     rx queues 1, rx desc 1024, tx queues 1, tx desc 1024
>     cpu socket 0
> 
>     tx frames ok                                   379327687
>     tx bytes ok                                 573877720248 
> <tel:573877720248>
>     rx frames ok                                   481696392
>     rx bytes ok                                 729288200634 
> <tel:729288200634>
>     rx missed                                         576271
>     extended stats:
>       rx good packets                              481696392
>       tx good packets                              379327687
>       rx good bytes                             729288200634 
> <tel:729288200634>
>       tx good bytes                             573877720248 
> <tel:573877720248>
>       rx missed errors                                576271
>       rx errors                                            0
>       tx errors                                            0
>       rx mbuf allocation errors                            0
>       rx q0packets                                         0
>       rx q0bytes                                           0
>       rx q0errors                                          0
>       tx q0packets                                         0
>       tx q0bytes                                           0
> VirtualFunctionEthernet0/7/0       2     up   VirtualFunctionEthernet0/7/0
>   Ethernet address 02:cb:fc:e1:b2:e6
>   AWS ENA VF
>     carrier up full duplex speed 10000 mtu 9216
>     pci id:            device 1d0f:ec20 subsystem 0000:0000
>     pci address:       0000:00:07.00
>     max rx packet len: 9216
>     max num of queues: rx 8 tx 8
>     promiscuous:       unicast off all-multicast off
>     vlan offload:      strip off filter off qinq off
>     rx offload caps:
>     tx offload caps:   ipv4-cksum udp-cksum tcp-cksum
>     rss active:        ipv4-sctp ipv4-other ipv6-udp-ex ipv6-frag ipv6-tcp
>                        l2-payload
>     rss supported:     none
>     rx queues 1, rx desc 1024, tx queues 1, tx desc 1024
>     cpu socket 0
> 
>     tx frames ok                                   378158827
>     tx bytes ok                                 572532333080 
> <tel:572532333080>
>     rx frames ok                                   483669396
>     rx bytes ok                                 731851064762 
> <tel:731851064762>
>     rx missed                                         576803
>     extended stats:
>       rx good packets                              483669396
>       tx good packets                              378158827
>       rx good bytes                             731851064762 
> <tel:731851064762>
>       tx good bytes                             572532333080 
> <tel:572532333080>
>       rx missed errors                                576803
>       rx errors                                            0
>       tx errors                                            0
>       rx mbuf allocation errors                            0
>       rx q0packets                                         0
>       rx q0bytes                                           0
>       rx q0errors                                          0
>       tx q0packets                                         0
>       tx q0bytes                                           0
> local0                             0    down  local0
>   local
> vpp#
>  
>  
> vpp# show interface
>               Name               Idx       State          Counter          
> Count
> VirtualFunctionEthernet0/6/0      1         up       rx packets             
> 481696393
>                                                      rx bytes            
> 729288200676 <tel:729288200676>
>                                                      tx packets             
> 483669396
>                                                      tx bytes            
> 731851064730 <tel:731851064730>
>                                                      drops                    
>       4
>                                                      ip4                    
> 481696346
>                                                      rx-miss                  
>  576271
>                                                      tx-error               
> 104341708
> VirtualFunctionEthernet0/7/0      2         up       rx packets             
> 483669397
>                                                      rx bytes            
> 731851064804 <tel:731851064804>
>                                                      tx packets             
> 481696392
>                                                      tx bytes            
> 729288200602 <tel:729288200602>
>                                                      drops                    
>       4
>                                                      ip4                    
> 483669350
>                                                      rx-miss                  
>  576803
>                                                      tx-error               
> 103537564
> local0                            0        down
>  
>  
> vpp# show runtime
> Time 2390.3, average vectors/node 2.46, last 128 main loops 0.00 per node 0.00
>   vector rates in 3.9979e5, out 3.1282e5, drop 2.5101e-3, punt 0.0000e0
>              Name                 State         Calls          Vectors        
> Suspends         Clocks       Vectors/Call
> VirtualFunctionEthernet0/6/0-o   active          203354869       478794413    
>            0          7.81e1            2.35
> VirtualFunctionEthernet0/6/0-t   active          203354869       374452705    
>            0          4.01e2            1.84
> VirtualFunctionEthernet0/7/0-o   active          203863473       476819589    
>            0          7.60e1            2.34
> VirtualFunctionEthernet0/7/0-t   active          203863473       373282025    
>            0          3.89e2            1.83
> api-rx-from-ring                any wait                 0               0    
>          120          1.19e4            0.00
> arp-input                        active                 61              88    
>            0          2.75e4            1.44
> dhcp-client-process             any wait                 0               0    
>           24          5.04e3            0.00
> dns-resolver-process            any wait                 0               0    
>            2          4.74e3            0.00
> dpdk-input                       polling       16754698175 <tel:16754698175>  
>      955614004               0          4.47e3             .06
> dpdk-process                    any wait                 0               0    
>          797          1.01e5            0.00
> error-drop                       active                  6               6    
>            0          3.91e3            1.00
> ethernet-input                   active                 61              88    
>            0          3.33e3            1.44
> fib-walk                        any wait                 0               0    
>         1195          4.96e3            0.00
> ikev2-manager-process           any wait                 0               0    
>         2390          2.49e3            0.00
> interface-output                 active                 57              84    
>            0          2.79e3            1.47
> ip-route-resolver-process       any wait                 0               0    
>           24          3.33e3            0.00
> ip4-glean                        active                  2               2    
>            0          2.06e4            1.00
> ip4-input                        active          382393615       955613916    
>            0          1.37e2            2.49
> ip4-lookup                       active          382393615       955613916    
>            0          9.37e1            2.49
> ip4-reassembly-expire-walk      any wait                 0               0    
>          239          5.99e3            0.00
> ip4-rewrite                      active          382393613       955613914    
>            0          9.20e1            2.49
> ip6-icmp-neighbor-discovery-ev  any wait                 0               0    
>         2390          2.26e3            0.00
> ip6-reassembly-expire-walk      any wait                 0               0    
>          239          5.69e3            0.00
> lisp-retry-service              any wait                 0               0    
>         1195          5.34e3            0.00
> rd-cp-process                   any wait                 0               0    
>    118658595          3.15e2            0.00
> unix-cli-local:2                 active                  2               0    
>         1363          3.00e9            0.00
> unix-epoll-input                 polling          16346047               0    
>            0          1.07e3            0.00
> vpe-oam-process                 any wait                 0               0    
>         1171          3.69e3            0.00
> vpp#
>  
> --
> Santosh
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> 
> View/Reply Online (#10759): https://lists.fd.io/g/vpp-dev/message/10759 
> <https://lists.fd.io/g/vpp-dev/message/10759>
> Mute This Topic: https://lists.fd.io/mt/26799215/675164 
> <https://lists.fd.io/mt/26799215/675164>
> Group Owner: [email protected] <mailto:[email protected]>
> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub 
> <https://lists.fd.io/g/vpp-dev/unsub>  [[email protected] 
> <mailto:[email protected]>]
> -=-=-=-=-=-=-=-=-=-=-=-

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#10760): https://lists.fd.io/g/vpp-dev/message/10760
Mute This Topic: https://lists.fd.io/mt/26799215/21656
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Reply via email to