Note the vector size. Vpp isn’t even working hard enough to run efficiently.
The tx side of the interface you’re using appears to be throttled.
D.
From: Santosh Kumar Dornal <santoshdor...@gmail.com>
Sent: Friday, October 5, 2018 6:30 PM
To: j...@netgate.com
Cc: vpp-dev@lists.fd.io; Dave Barach (dbarach) <dbar...@cisco.com>
Subject: Re: [vpp-dev] Tx packet drops - dpdk tx failure
Hi Dave
Here is the output of "show run". I am using Trex tool and generating 1500 byte
udp packets.
vpp# show errors
Count Node Reason
16 arp-input ARP replies sent
15611 VirtualFunctionEthernet0/7/0-tx Tx packet drops (dpdk tx
failure)
vpp#
vpp# show runtime
Time 30.5, average vectors/node 2.34, last 128 main loops 0.00 per node 0.00
vector rates in 3.4424e5, out 3.4373e5, drop 0.0000e0, punt 0.0000e0
Name State Calls Vectors
Suspends Clocks Vectors/Call
VirtualFunctionEthernet0/6/0-o active 1 1
0 8.02e2 1.00
VirtualFunctionEthernet0/6/0-t active 1 1
0 3.83e3 1.00
VirtualFunctionEthernet0/7/0-o active 4483078 10493156
0 7.35e1 2.34
VirtualFunctionEthernet0/7/0-t active 4483078 10477545
0 3.11e2 2.34
api-rx-from-ring any wait 0 0
1 1.02e4 0.00
arp-input active 1 2
0 2.03e4 2.00
dhcp-client-process any wait 0 0
1 3.77e3 0.00
dpdk-input polling 218410075 10493157
0 5.23e3 .05
dpdk-process any wait 0 0
10 9.76e4 0.00
ethernet-input active 1 2
0 2.86e3 2.00
fib-walk any wait 0 0
15 5.84e3 0.00
ikev2-manager-process any wait 0 0
31 2.86e3 0.00
interface-output active 1 2
0 2.51e3 2.00
ip-route-resolver-process any wait 0 0
1 3.60e3 0.00
ip4-input active 4483078 10493155
0 1.45e2 2.34
ip4-lookup active 4483078 10493155
0 9.87e1 2.34
ip4-reassembly-expire-walk any wait 0 0
3 5.86e3 0.00
ip4-rewrite active 4483078 10493155
0 8.12e1 2.34
ip6-icmp-neighbor-discovery-ev any wait 0 0
31 2.71e3 0.00
ip6-reassembly-expire-walk any wait 0 0
3 5.53e3 0.00
lisp-retry-service any wait 0 0
15 6.00e3 0.00
rd-cp-process any wait 0 0
1514830 3.02e2 0.00
unix-cli-local:6 active 0 0
97 1.59e5 0.00
unix-epoll-input polling 213083 0
0 1.11e3 0.00
vpe-oam-process any wait 0 0
15 3.99e3 0.00
vpp#
--
Santosh
On Fri, Oct 5, 2018 at 7:29 AM Jim Thompson
<j...@netgate.com<mailto:j...@netgate.com>> wrote:
We’ve seen 4.6Gbps IPsec on a single core of a C5 instance, and that’s
throttled by the 5gbps in and out of the VPC “limit” courtesy of Amazon. (The
other 400Mbps is framing overhead.)
But here, obviously, “large packets” (we were using iperf3 on the tunnel).
Jim
On Oct 5, 2018, at 6:38 AM, Dave Barach via Lists.Fd.Io<http://Lists.Fd.Io>
<dbarach=cisco....@lists.fd.io<mailto:dbarach=cisco....@lists.fd.io>> wrote:
I’ll let others jump in on the tx [and rx] drops. At some level, it looks the
vfio device simply isn’t disposing of traffic rapidly enough. Perhaps too few
buffers have been configured, but I don’t think that’s the most likely
explanation.
The “show run” stats are not helpful because they include over 2000 seconds’s
worth of [mostly] dead airtime: please start traffic, “clear run”, wait a few
seconds, then “show run”.
You didn’t mention the packet size that you’re using. It wouldn’t shock me to
learn that a “10xGE capable instance” comes with a few stars in the margin:
perhaps “10xGE capable at or above 256 byte packets” or some such.
HTH... Dave
From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>
<vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> On Behalf Of Santosh Kumar
Dornal
Sent: Thursday, October 4, 2018 9:40 PM
To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>
Subject: [vpp-dev] Tx packet drops - dpdk tx failure
Hi
I am using VPP 18.04 on AWS (c5.xlarge instance with ena interfaces) and
running into Tx packet drops when pushing traffic over 2.5Gbps (bidirectional).
The instance is capable of doing 10gig. Can somebody help?
vpp# show version
vpp v18.04-16~gca7a68e0 built by vagrant on debian9.localdomain at Wed Oct 3
11:57:44 PDT 2018
vpp# show errors
Count Node Reason
68 arp-input ARP replies sent
82693102 VirtualFunctionEthernet0/6/0-tx Tx packet drops (dpdk tx
failure)
83200033 VirtualFunctionEthernet0/7/0-tx Tx packet drops (dpdk tx
failure)
vpp#
vpp# show hardware-interfaces detail
Name Idx Link Hardware
VirtualFunctionEthernet0/6/0 1 up VirtualFunctionEthernet0/6/0
Ethernet address 02:d2:6c:6e:b0:1c
AWS ENA VF
carrier up full duplex speed 10000 mtu 9216
pci id: device 1d0f:ec20 subsystem 0000:0000
pci address: 0000:00:06.00
max rx packet len: 9216
max num of queues: rx 8 tx 8
promiscuous: unicast off all-multicast off
vlan offload: strip off filter off qinq off
rx offload caps:
tx offload caps: ipv4-cksum udp-cksum tcp-cksum
rss active: ipv4-sctp ipv4-other ipv6-udp-ex ipv6-frag ipv6-tcp
l2-payload
rss supported: none
rx queues 1, rx desc 1024, tx queues 1, tx desc 1024
cpu socket 0
tx frames ok 379327687
tx bytes ok 573877720248<tel:573877720248>
rx frames ok 481696392
rx bytes ok 729288200634<tel:729288200634>
rx missed 576271
extended stats:
rx good packets 481696392
tx good packets 379327687
rx good bytes 729288200634<tel:729288200634>
tx good bytes 573877720248<tel:573877720248>
rx missed errors 576271
rx errors 0
tx errors 0
rx mbuf allocation errors 0
rx q0packets 0
rx q0bytes 0
rx q0errors 0
tx q0packets 0
tx q0bytes 0
VirtualFunctionEthernet0/7/0 2 up VirtualFunctionEthernet0/7/0
Ethernet address 02:cb:fc:e1:b2:e6
AWS ENA VF
carrier up full duplex speed 10000 mtu 9216
pci id: device 1d0f:ec20 subsystem 0000:0000
pci address: 0000:00:07.00
max rx packet len: 9216
max num of queues: rx 8 tx 8
promiscuous: unicast off all-multicast off
vlan offload: strip off filter off qinq off
rx offload caps:
tx offload caps: ipv4-cksum udp-cksum tcp-cksum
rss active: ipv4-sctp ipv4-other ipv6-udp-ex ipv6-frag ipv6-tcp
l2-payload
rss supported: none
rx queues 1, rx desc 1024, tx queues 1, tx desc 1024
cpu socket 0
tx frames ok 378158827
tx bytes ok 572532333080<tel:572532333080>
rx frames ok 483669396
rx bytes ok 731851064762<tel:731851064762>
rx missed 576803
extended stats:
rx good packets 483669396
tx good packets 378158827
rx good bytes 731851064762<tel:731851064762>
tx good bytes 572532333080<tel:572532333080>
rx missed errors 576803
rx errors 0
tx errors 0
rx mbuf allocation errors 0
rx q0packets 0
rx q0bytes 0
rx q0errors 0
tx q0packets 0
tx q0bytes 0
local0 0 down local0
local
vpp#
vpp# show interface
Name Idx State Counter Count
VirtualFunctionEthernet0/6/0 1 up rx packets
481696393
rx bytes
729288200676<tel:729288200676>
tx packets
483669396
tx bytes
731851064730<tel:731851064730>
drops
4
ip4
481696346
rx-miss
576271
tx-error
104341708
VirtualFunctionEthernet0/7/0 2 up rx packets
483669397
rx bytes
731851064804<tel:731851064804>
tx packets
481696392
tx bytes
729288200602<tel:729288200602>
drops
4
ip4
483669350
rx-miss
576803
tx-error
103537564
local0 0 down
vpp# show runtime
Time 2390.3, average vectors/node 2.46, last 128 main loops 0.00 per node 0.00
vector rates in 3.9979e5, out 3.1282e5, drop 2.5101e-3, punt 0.0000e0
Name State Calls Vectors
Suspends Clocks Vectors/Call
VirtualFunctionEthernet0/6/0-o active 203354869 478794413
0 7.81e1 2.35
VirtualFunctionEthernet0/6/0-t active 203354869 374452705
0 4.01e2 1.84
VirtualFunctionEthernet0/7/0-o active 203863473 476819589
0 7.60e1 2.34
VirtualFunctionEthernet0/7/0-t active 203863473 373282025
0 3.89e2 1.83
api-rx-from-ring any wait 0 0
120 1.19e4 0.00
arp-input active 61 88
0 2.75e4 1.44
dhcp-client-process any wait 0 0
24 5.04e3 0.00
dns-resolver-process any wait 0 0
2 4.74e3 0.00
dpdk-input polling 16754698175<tel:16754698175>
955614004 0 4.47e3 .06
dpdk-process any wait 0 0
797 1.01e5 0.00
error-drop active 6 6
0 3.91e3 1.00
ethernet-input active 61 88
0 3.33e3 1.44
fib-walk any wait 0 0
1195 4.96e3 0.00
ikev2-manager-process any wait 0 0
2390 2.49e3 0.00
interface-output active 57 84
0 2.79e3 1.47
ip-route-resolver-process any wait 0 0
24 3.33e3 0.00
ip4-glean active 2 2
0 2.06e4 1.00
ip4-input active 382393615 955613916
0 1.37e2 2.49
ip4-lookup active 382393615 955613916
0 9.37e1 2.49
ip4-reassembly-expire-walk any wait 0 0
239 5.99e3 0.00
ip4-rewrite active 382393613 955613914
0 9.20e1 2.49
ip6-icmp-neighbor-discovery-ev any wait 0 0
2390 2.26e3 0.00
ip6-reassembly-expire-walk any wait 0 0
239 5.69e3 0.00
lisp-retry-service any wait 0 0
1195 5.34e3 0.00
rd-cp-process any wait 0 0
118658595 3.15e2 0.00
unix-cli-local:2 active 2 0
1363 3.00e9 0.00
unix-epoll-input polling 16346047 0
0 1.07e3 0.00
vpe-oam-process any wait 0 0
1171 3.69e3 0.00
vpp#
--
Santosh
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#10759): https://lists.fd.io/g/vpp-dev/message/10759
Mute This Topic: https://lists.fd.io/mt/26799215/675164
Group Owner: vpp-dev+ow...@lists.fd.io<mailto:vpp-dev+ow...@lists.fd.io>
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub
[j...@netgate.com<mailto:j...@netgate.com>]
-=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#10763): https://lists.fd.io/g/vpp-dev/message/10763
Mute This Topic: https://lists.fd.io/mt/26799215/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
