We’ve seen 4.6Gbps IPsec on a single core of a C5 instance, and that’s throttled by the 5gbps in and out of the VPC “limit” courtesy of Amazon. (The other 400Mbps is framing overhead.) But here, obviously, “large packets” (we were using iperf3 on the tunnel).
Jim > On Oct 5, 2018, at 6:38 AM, Dave Barach via Lists.Fd.Io > <[email protected]> wrote: > > I’ll let others jump in on the tx [and rx] drops. At some level, it looks the > vfio device simply isn’t disposing of traffic rapidly enough. Perhaps too few > buffers have been configured, but I don’t think that’s the most likely > explanation. > > The “show run” stats are not helpful because they include over 2000 seconds’s > worth of [mostly] dead airtime: please start traffic, “clear run”, wait a few > seconds, then “show run”. > > You didn’t mention the packet size that you’re using. It wouldn’t shock me to > learn that a “10xGE capable instance” comes with a few stars in the margin: > perhaps “10xGE capable at or above 256 byte packets” or some such. > > HTH... Dave > > From: [email protected] <mailto:[email protected]> <[email protected] > <mailto:[email protected]>> On Behalf Of Santosh Kumar Dornal > Sent: Thursday, October 4, 2018 9:40 PM > To: [email protected] <mailto:[email protected]> > Subject: [vpp-dev] Tx packet drops - dpdk tx failure > > Hi > > I am using VPP 18.04 on AWS (c5.xlarge instance with ena interfaces) and > running into Tx packet drops when pushing traffic over 2.5Gbps > (bidirectional). The instance is capable of doing 10gig. Can somebody help? > > vpp# show version > vpp v18.04-16~gca7a68e0 built by vagrant on debian9.localdomain at Wed Oct 3 > 11:57:44 PDT 2018 > vpp# show errors > Count Node Reason > 68 arp-input ARP replies sent > 82693102 VirtualFunctionEthernet0/6/0-tx Tx packet drops (dpdk tx > failure) > 83200033 VirtualFunctionEthernet0/7/0-tx Tx packet drops (dpdk tx > failure) > vpp# > > vpp# show hardware-interfaces detail > Name Idx Link Hardware > VirtualFunctionEthernet0/6/0 1 up VirtualFunctionEthernet0/6/0 > Ethernet address 02:d2:6c:6e:b0:1c > AWS ENA VF > carrier up full duplex speed 10000 mtu 9216 > pci id: device 1d0f:ec20 subsystem 0000:0000 > pci address: 0000:00:06.00 > max rx packet len: 9216 > max num of queues: rx 8 tx 8 > promiscuous: unicast off all-multicast off > vlan offload: strip off filter off qinq off > rx offload caps: > tx offload caps: ipv4-cksum udp-cksum tcp-cksum > rss active: ipv4-sctp ipv4-other ipv6-udp-ex ipv6-frag ipv6-tcp > l2-payload > rss supported: none > rx queues 1, rx desc 1024, tx queues 1, tx desc 1024 > cpu socket 0 > > tx frames ok 379327687 > tx bytes ok 573877720248 > <tel:573877720248> > rx frames ok 481696392 > rx bytes ok 729288200634 > <tel:729288200634> > rx missed 576271 > extended stats: > rx good packets 481696392 > tx good packets 379327687 > rx good bytes 729288200634 > <tel:729288200634> > tx good bytes 573877720248 > <tel:573877720248> > rx missed errors 576271 > rx errors 0 > tx errors 0 > rx mbuf allocation errors 0 > rx q0packets 0 > rx q0bytes 0 > rx q0errors 0 > tx q0packets 0 > tx q0bytes 0 > VirtualFunctionEthernet0/7/0 2 up VirtualFunctionEthernet0/7/0 > Ethernet address 02:cb:fc:e1:b2:e6 > AWS ENA VF > carrier up full duplex speed 10000 mtu 9216 > pci id: device 1d0f:ec20 subsystem 0000:0000 > pci address: 0000:00:07.00 > max rx packet len: 9216 > max num of queues: rx 8 tx 8 > promiscuous: unicast off all-multicast off > vlan offload: strip off filter off qinq off > rx offload caps: > tx offload caps: ipv4-cksum udp-cksum tcp-cksum > rss active: ipv4-sctp ipv4-other ipv6-udp-ex ipv6-frag ipv6-tcp > l2-payload > rss supported: none > rx queues 1, rx desc 1024, tx queues 1, tx desc 1024 > cpu socket 0 > > tx frames ok 378158827 > tx bytes ok 572532333080 > <tel:572532333080> > rx frames ok 483669396 > rx bytes ok 731851064762 > <tel:731851064762> > rx missed 576803 > extended stats: > rx good packets 483669396 > tx good packets 378158827 > rx good bytes 731851064762 > <tel:731851064762> > tx good bytes 572532333080 > <tel:572532333080> > rx missed errors 576803 > rx errors 0 > tx errors 0 > rx mbuf allocation errors 0 > rx q0packets 0 > rx q0bytes 0 > rx q0errors 0 > tx q0packets 0 > tx q0bytes 0 > local0 0 down local0 > local > vpp# > > > vpp# show interface > Name Idx State Counter > Count > VirtualFunctionEthernet0/6/0 1 up rx packets > 481696393 > rx bytes > 729288200676 <tel:729288200676> > tx packets > 483669396 > tx bytes > 731851064730 <tel:731851064730> > drops > 4 > ip4 > 481696346 > rx-miss > 576271 > tx-error > 104341708 > VirtualFunctionEthernet0/7/0 2 up rx packets > 483669397 > rx bytes > 731851064804 <tel:731851064804> > tx packets > 481696392 > tx bytes > 729288200602 <tel:729288200602> > drops > 4 > ip4 > 483669350 > rx-miss > 576803 > tx-error > 103537564 > local0 0 down > > > vpp# show runtime > Time 2390.3, average vectors/node 2.46, last 128 main loops 0.00 per node 0.00 > vector rates in 3.9979e5, out 3.1282e5, drop 2.5101e-3, punt 0.0000e0 > Name State Calls Vectors > Suspends Clocks Vectors/Call > VirtualFunctionEthernet0/6/0-o active 203354869 478794413 > 0 7.81e1 2.35 > VirtualFunctionEthernet0/6/0-t active 203354869 374452705 > 0 4.01e2 1.84 > VirtualFunctionEthernet0/7/0-o active 203863473 476819589 > 0 7.60e1 2.34 > VirtualFunctionEthernet0/7/0-t active 203863473 373282025 > 0 3.89e2 1.83 > api-rx-from-ring any wait 0 0 > 120 1.19e4 0.00 > arp-input active 61 88 > 0 2.75e4 1.44 > dhcp-client-process any wait 0 0 > 24 5.04e3 0.00 > dns-resolver-process any wait 0 0 > 2 4.74e3 0.00 > dpdk-input polling 16754698175 <tel:16754698175> > 955614004 0 4.47e3 .06 > dpdk-process any wait 0 0 > 797 1.01e5 0.00 > error-drop active 6 6 > 0 3.91e3 1.00 > ethernet-input active 61 88 > 0 3.33e3 1.44 > fib-walk any wait 0 0 > 1195 4.96e3 0.00 > ikev2-manager-process any wait 0 0 > 2390 2.49e3 0.00 > interface-output active 57 84 > 0 2.79e3 1.47 > ip-route-resolver-process any wait 0 0 > 24 3.33e3 0.00 > ip4-glean active 2 2 > 0 2.06e4 1.00 > ip4-input active 382393615 955613916 > 0 1.37e2 2.49 > ip4-lookup active 382393615 955613916 > 0 9.37e1 2.49 > ip4-reassembly-expire-walk any wait 0 0 > 239 5.99e3 0.00 > ip4-rewrite active 382393613 955613914 > 0 9.20e1 2.49 > ip6-icmp-neighbor-discovery-ev any wait 0 0 > 2390 2.26e3 0.00 > ip6-reassembly-expire-walk any wait 0 0 > 239 5.69e3 0.00 > lisp-retry-service any wait 0 0 > 1195 5.34e3 0.00 > rd-cp-process any wait 0 0 > 118658595 3.15e2 0.00 > unix-cli-local:2 active 2 0 > 1363 3.00e9 0.00 > unix-epoll-input polling 16346047 0 > 0 1.07e3 0.00 > vpe-oam-process any wait 0 0 > 1171 3.69e3 0.00 > vpp# > > -- > Santosh > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#10759): https://lists.fd.io/g/vpp-dev/message/10759 > <https://lists.fd.io/g/vpp-dev/message/10759> > Mute This Topic: https://lists.fd.io/mt/26799215/675164 > <https://lists.fd.io/mt/26799215/675164> > Group Owner: [email protected] <mailto:[email protected]> > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub > <https://lists.fd.io/g/vpp-dev/unsub> [[email protected] > <mailto:[email protected]>] > -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10760): https://lists.fd.io/g/vpp-dev/message/10760 Mute This Topic: https://lists.fd.io/mt/26799215/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
