Note the vector size. Vpp isn’t even working hard enough to run efficiently. The tx side of the interface you’re using appears to be throttled.
D. From: Santosh Kumar Dornal <santoshdor...@gmail.com> Sent: Friday, October 5, 2018 6:30 PM To: j...@netgate.com Cc: vpp-dev@lists.fd.io; Dave Barach (dbarach) <dbar...@cisco.com> Subject: Re: [vpp-dev] Tx packet drops - dpdk tx failure Hi Dave Here is the output of "show run". I am using Trex tool and generating 1500 byte udp packets. vpp# show errors Count Node Reason 16 arp-input ARP replies sent 15611 VirtualFunctionEthernet0/7/0-tx Tx packet drops (dpdk tx failure) vpp# vpp# show runtime Time 30.5, average vectors/node 2.34, last 128 main loops 0.00 per node 0.00 vector rates in 3.4424e5, out 3.4373e5, drop 0.0000e0, punt 0.0000e0 Name State Calls Vectors Suspends Clocks Vectors/Call VirtualFunctionEthernet0/6/0-o active 1 1 0 8.02e2 1.00 VirtualFunctionEthernet0/6/0-t active 1 1 0 3.83e3 1.00 VirtualFunctionEthernet0/7/0-o active 4483078 10493156 0 7.35e1 2.34 VirtualFunctionEthernet0/7/0-t active 4483078 10477545 0 3.11e2 2.34 api-rx-from-ring any wait 0 0 1 1.02e4 0.00 arp-input active 1 2 0 2.03e4 2.00 dhcp-client-process any wait 0 0 1 3.77e3 0.00 dpdk-input polling 218410075 10493157 0 5.23e3 .05 dpdk-process any wait 0 0 10 9.76e4 0.00 ethernet-input active 1 2 0 2.86e3 2.00 fib-walk any wait 0 0 15 5.84e3 0.00 ikev2-manager-process any wait 0 0 31 2.86e3 0.00 interface-output active 1 2 0 2.51e3 2.00 ip-route-resolver-process any wait 0 0 1 3.60e3 0.00 ip4-input active 4483078 10493155 0 1.45e2 2.34 ip4-lookup active 4483078 10493155 0 9.87e1 2.34 ip4-reassembly-expire-walk any wait 0 0 3 5.86e3 0.00 ip4-rewrite active 4483078 10493155 0 8.12e1 2.34 ip6-icmp-neighbor-discovery-ev any wait 0 0 31 2.71e3 0.00 ip6-reassembly-expire-walk any wait 0 0 3 5.53e3 0.00 lisp-retry-service any wait 0 0 15 6.00e3 0.00 rd-cp-process any wait 0 0 1514830 3.02e2 0.00 unix-cli-local:6 active 0 0 97 1.59e5 0.00 unix-epoll-input polling 213083 0 0 1.11e3 0.00 vpe-oam-process any wait 0 0 15 3.99e3 0.00 vpp# -- Santosh On Fri, Oct 5, 2018 at 7:29 AM Jim Thompson <j...@netgate.com<mailto:j...@netgate.com>> wrote: We’ve seen 4.6Gbps IPsec on a single core of a C5 instance, and that’s throttled by the 5gbps in and out of the VPC “limit” courtesy of Amazon. (The other 400Mbps is framing overhead.) But here, obviously, “large packets” (we were using iperf3 on the tunnel). Jim On Oct 5, 2018, at 6:38 AM, Dave Barach via Lists.Fd.Io<http://Lists.Fd.Io> <dbarach=cisco....@lists.fd.io<mailto:dbarach=cisco....@lists.fd.io>> wrote: I’ll let others jump in on the tx [and rx] drops. At some level, it looks the vfio device simply isn’t disposing of traffic rapidly enough. Perhaps too few buffers have been configured, but I don’t think that’s the most likely explanation. The “show run” stats are not helpful because they include over 2000 seconds’s worth of [mostly] dead airtime: please start traffic, “clear run”, wait a few seconds, then “show run”. You didn’t mention the packet size that you’re using. It wouldn’t shock me to learn that a “10xGE capable instance” comes with a few stars in the margin: perhaps “10xGE capable at or above 256 byte packets” or some such. HTH... Dave From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> On Behalf Of Santosh Kumar Dornal Sent: Thursday, October 4, 2018 9:40 PM To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: [vpp-dev] Tx packet drops - dpdk tx failure Hi I am using VPP 18.04 on AWS (c5.xlarge instance with ena interfaces) and running into Tx packet drops when pushing traffic over 2.5Gbps (bidirectional). The instance is capable of doing 10gig. Can somebody help? vpp# show version vpp v18.04-16~gca7a68e0 built by vagrant on debian9.localdomain at Wed Oct 3 11:57:44 PDT 2018 vpp# show errors Count Node Reason 68 arp-input ARP replies sent 82693102 VirtualFunctionEthernet0/6/0-tx Tx packet drops (dpdk tx failure) 83200033 VirtualFunctionEthernet0/7/0-tx Tx packet drops (dpdk tx failure) vpp# vpp# show hardware-interfaces detail Name Idx Link Hardware VirtualFunctionEthernet0/6/0 1 up VirtualFunctionEthernet0/6/0 Ethernet address 02:d2:6c:6e:b0:1c AWS ENA VF carrier up full duplex speed 10000 mtu 9216 pci id: device 1d0f:ec20 subsystem 0000:0000 pci address: 0000:00:06.00 max rx packet len: 9216 max num of queues: rx 8 tx 8 promiscuous: unicast off all-multicast off vlan offload: strip off filter off qinq off rx offload caps: tx offload caps: ipv4-cksum udp-cksum tcp-cksum rss active: ipv4-sctp ipv4-other ipv6-udp-ex ipv6-frag ipv6-tcp l2-payload rss supported: none rx queues 1, rx desc 1024, tx queues 1, tx desc 1024 cpu socket 0 tx frames ok 379327687 tx bytes ok 573877720248<tel:573877720248> rx frames ok 481696392 rx bytes ok 729288200634<tel:729288200634> rx missed 576271 extended stats: rx good packets 481696392 tx good packets 379327687 rx good bytes 729288200634<tel:729288200634> tx good bytes 573877720248<tel:573877720248> rx missed errors 576271 rx errors 0 tx errors 0 rx mbuf allocation errors 0 rx q0packets 0 rx q0bytes 0 rx q0errors 0 tx q0packets 0 tx q0bytes 0 VirtualFunctionEthernet0/7/0 2 up VirtualFunctionEthernet0/7/0 Ethernet address 02:cb:fc:e1:b2:e6 AWS ENA VF carrier up full duplex speed 10000 mtu 9216 pci id: device 1d0f:ec20 subsystem 0000:0000 pci address: 0000:00:07.00 max rx packet len: 9216 max num of queues: rx 8 tx 8 promiscuous: unicast off all-multicast off vlan offload: strip off filter off qinq off rx offload caps: tx offload caps: ipv4-cksum udp-cksum tcp-cksum rss active: ipv4-sctp ipv4-other ipv6-udp-ex ipv6-frag ipv6-tcp l2-payload rss supported: none rx queues 1, rx desc 1024, tx queues 1, tx desc 1024 cpu socket 0 tx frames ok 378158827 tx bytes ok 572532333080<tel:572532333080> rx frames ok 483669396 rx bytes ok 731851064762<tel:731851064762> rx missed 576803 extended stats: rx good packets 483669396 tx good packets 378158827 rx good bytes 731851064762<tel:731851064762> tx good bytes 572532333080<tel:572532333080> rx missed errors 576803 rx errors 0 tx errors 0 rx mbuf allocation errors 0 rx q0packets 0 rx q0bytes 0 rx q0errors 0 tx q0packets 0 tx q0bytes 0 local0 0 down local0 local vpp# vpp# show interface Name Idx State Counter Count VirtualFunctionEthernet0/6/0 1 up rx packets 481696393 rx bytes 729288200676<tel:729288200676> tx packets 483669396 tx bytes 731851064730<tel:731851064730> drops 4 ip4 481696346 rx-miss 576271 tx-error 104341708 VirtualFunctionEthernet0/7/0 2 up rx packets 483669397 rx bytes 731851064804<tel:731851064804> tx packets 481696392 tx bytes 729288200602<tel:729288200602> drops 4 ip4 483669350 rx-miss 576803 tx-error 103537564 local0 0 down vpp# show runtime Time 2390.3, average vectors/node 2.46, last 128 main loops 0.00 per node 0.00 vector rates in 3.9979e5, out 3.1282e5, drop 2.5101e-3, punt 0.0000e0 Name State Calls Vectors Suspends Clocks Vectors/Call VirtualFunctionEthernet0/6/0-o active 203354869 478794413 0 7.81e1 2.35 VirtualFunctionEthernet0/6/0-t active 203354869 374452705 0 4.01e2 1.84 VirtualFunctionEthernet0/7/0-o active 203863473 476819589 0 7.60e1 2.34 VirtualFunctionEthernet0/7/0-t active 203863473 373282025 0 3.89e2 1.83 api-rx-from-ring any wait 0 0 120 1.19e4 0.00 arp-input active 61 88 0 2.75e4 1.44 dhcp-client-process any wait 0 0 24 5.04e3 0.00 dns-resolver-process any wait 0 0 2 4.74e3 0.00 dpdk-input polling 16754698175<tel:16754698175> 955614004 0 4.47e3 .06 dpdk-process any wait 0 0 797 1.01e5 0.00 error-drop active 6 6 0 3.91e3 1.00 ethernet-input active 61 88 0 3.33e3 1.44 fib-walk any wait 0 0 1195 4.96e3 0.00 ikev2-manager-process any wait 0 0 2390 2.49e3 0.00 interface-output active 57 84 0 2.79e3 1.47 ip-route-resolver-process any wait 0 0 24 3.33e3 0.00 ip4-glean active 2 2 0 2.06e4 1.00 ip4-input active 382393615 955613916 0 1.37e2 2.49 ip4-lookup active 382393615 955613916 0 9.37e1 2.49 ip4-reassembly-expire-walk any wait 0 0 239 5.99e3 0.00 ip4-rewrite active 382393613 955613914 0 9.20e1 2.49 ip6-icmp-neighbor-discovery-ev any wait 0 0 2390 2.26e3 0.00 ip6-reassembly-expire-walk any wait 0 0 239 5.69e3 0.00 lisp-retry-service any wait 0 0 1195 5.34e3 0.00 rd-cp-process any wait 0 0 118658595 3.15e2 0.00 unix-cli-local:2 active 2 0 1363 3.00e9 0.00 unix-epoll-input polling 16346047 0 0 1.07e3 0.00 vpe-oam-process any wait 0 0 1171 3.69e3 0.00 vpp# -- Santosh -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10759): https://lists.fd.io/g/vpp-dev/message/10759 Mute This Topic: https://lists.fd.io/mt/26799215/675164 Group Owner: vpp-dev+ow...@lists.fd.io<mailto:vpp-dev+ow...@lists.fd.io> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [j...@netgate.com<mailto:j...@netgate.com>] -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10763): https://lists.fd.io/g/vpp-dev/message/10763 Mute This Topic: https://lists.fd.io/mt/26799215/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-