Hi Dom,
> On Dec 12, 2019, at 12:29 PM, dch...@akouto.com wrote:
>
> Hi Florin,
>
> The saga continues, a little progress and more questions. In order to reduce
> the variables, I am now only using VPP on one of the VMs: iperf3 server is
> running on a VM with native Linux networking, and iperf3+VCL client running
> on the second VM.
FC: Okay!
>
> I've pasted the output from a few commands during this test run below and
> have a few questions if you don't mind.
> The "show errors" command indicates "Tx packet drops (dpdk tx failure)". I
> have done quite a bit of searching, found other mentions of this in other
> threads but no tips as to where to look or hints on how it was / can be
> solved. Any thoughts?
FC: The number of drops is not that large, so we can ignore for now.
> I'm not really sure how to interpret the results of "show run" but nothing
> jumps out at me, do you see anything useful in there?
FC: Nothing apart from the fact that one of vpp’s workers is moderately loaded
(you’re still running 3 workers).
> Some of the startup.conf options were not working for me, so I switched to
> building from source (I chose to use tag v20.01-rc0 for some stability).
> Still no luck with some of the options:
> When I try to use tcp { tso } I get this: 0: tcp_config_fn: unknown input `
> tso'
FC: You need to get “closer” to master HEAD. That tag was laid when 19.08 was
released but tso support was merged afterwards. Typically our CI infra is good
enough to keep things running so you might want to try master latest.
> When I try to use num-mbufs in the dpdk section, I get 0: dpdk_config:
> unknown input `num-mbufs 65535’
FC: This was deprecated at one point. The new stanza is "buffers {
buffers-per-numa <number> }"
>
> Do you know if these options are supported? I can't figure out a way to
> increase mbufs since the above option does not work, and when I try to use
> socket-mem (which according to the documentation is needed if there is a need
> for a larger number of mbufs) I get this: dpdk_config:1408: socket-mem
> argument is deprecated
FC: Yes, this was also deprecated.
>
> To answer some of your questions from your previous reply:
> I have indeed been using taaskset and watching CPU load with top to make sure
> things are going where I expect them to go
> I am not trying to use jumbo buffers, increasing "default data-size" was just
> an attempt to see if there would be a difference
> Thanks for the cubic congestion algo suggestion, made the change but no
> improvement
FC: Understood! I guess that means we should try tso. I just tested it and it
seems dpdk stanza needs an extra "dpdk {enable-tcp-udp-checksum}” apart from
“dpdk { dev <id> { tso on } }”. Let me know if you hit any other issues with
it. You’ll know that it’s running if you do “show session verbose 2” and you
see “TSO" in the cfg flags, instead of “TSO off”.
Regards,
Florin
> Thank you for all the help, it is very much appreciated.
>
> Regards,
> Dom
>
> vpp# sh int
> Name Idx State MTU (L3/IP4/IP6/MPLS)
> Counter Count
> GigabitEthernet0/3/0 1 up 9000/0/0/0 rx
> packets 1642537
> rx bytes
> 108676814
> tx
> packets 5216493
> tx bytes
> 7793319472
> drops
> 392
> ip4
> 1642178
> tx-error
> 475
> local0 0 down 0/0/0/0 drops
> 1
>
> vpp# sh err
> Count Node Reason
> 1 ip4-glean ARP requests sent
> 7 dpdk-input no error
> 5216424 session-queue Packets transmitted
> 1 tcp4-rcv-process Pure ACKs received
> 2 tcp4-syn-sent SYN-ACKs received
> 7 tcp4-established Packets pushed into rx fifo
> 1619850 tcp4-established Pure ACKs received
> 22219 tcp4-established Duplicate ACK
> 1 tcp4-established Resets received
> 62 tcp4-established Connection closed
> 1 tcp4-established FINs received
> 62 tcp4-output Resets sent
> 2 arp-reply ARP replies sent
> 33 ip4-input unknown ip protocol
> 1 ip4-input Multicast RPF check failed
> 1 ip4-glean ARP requests sent
> 351 llc-input unknown llc ssap/dsap
> 475 GigabitEthernet0/3/0-tx Tx packet drops (dpdk tx
> failure)
>
> vpp# sh run
> Thread 0 vpp_main (lcore 7)
> Time 94.7, average vectors/node 1.00, last 128 main loops 0.00 per node 0.00
> vector rates in 0.0000e0, out 3.1669e-2, drop 1.0556e-2, punt 0.0000e0
> Name State Calls Vectors
> Suspends Clocks Vectors/Call
> GigabitEthernet0/3/0-output active 3 3
> 0 3.29e4 1.00
> GigabitEthernet0/3/0-tx active 3 3
> 0 3.73e4 1.00
> acl-plugin-fa-cleaner-process event wait 0 0
> 1 2.78e4 0.00
> admin-up-down-process event wait 0 0
> 1 2.24e3 0.00
> api-rx-from-ring any wait 0 0
> 24 1.01e6 0.00
> avf-process event wait 0 0
> 1 2.15e4 0.00
> bfd-process event wait 0 0
> 1 1.49e4 0.00
> bond-process event wait 0 0
> 1 1.43e4 0.00
> dhcp-client-process any wait 0 0
> 1 2.72e4 0.00
> dhcp6-client-cp-process any wait 0 0
> 1 1.07e4 0.00
> dhcp6-pd-client-cp-process any wait 0 0
> 1 2.03e4 0.00
> dhcp6-pd-reply-publisher-proce event wait 0 0
> 1 1.31e4 0.00
> dhcp6-reply-publisher-process event wait 0 0
> 1 1.37e4 0.00
> dpdk-ipsec-process done 1 0
> 0 8.66e4 0.00
> dpdk-process any wait 0 0
> 32 3.49e6 0.00
> drop active 1 1
> 0 3.19e3 1.00
> error-drop active 1 1
> 0 5.17e3 1.00
> fib-walk any wait 0 0
> 48 1.09e4 0.00
> flow-report-process any wait 0 0
> 1 6.19e3 0.00
> flowprobe-timer-process any wait 0 0
> 1 7.33e3 0.00
> gbp-scanner event wait 0 0
> 1 1.79e4 0.00
> igmp-timer-process event wait 0 0
> 1 8.09e3 0.00
> ikev2-manager-process any wait 0 0
> 95 6.15e3 0.00
> ioam-export-process any wait 0 0
> 1 8.14e3 0.00
> ip-neighbor-scan-process any wait 0 0
> 2 8.52e3 0.00
> ip-route-resolver-process any wait 0 0
> 1 1.64e4 0.00
> ip4-glean active 1 1
> 0 4.86e4 1.00
> ip4-lookup active 3 3
> 0 2.11e4 1.00
> ip4-reassembly-expire-walk any wait 0 0
> 10 9.13e3 0.00
> ip4-rewrite active 2 2
> 0 2.51e4 1.00
> ip6-icmp-neighbor-discovery-ev any wait 0 0
> 95 5.36e3 0.00
> ip6-reassembly-expire-walk any wait 0 0
> 10 1.02e4 0.00
> l2fib-mac-age-scanner-process event wait 0 0
> 1 1.12e4 0.00
> lldp-process event wait 0 0
> 1 8.02e6 0.00
> memif-process event wait 0 0
> 1 2.21e4 0.00
> nat-det-expire-walk done 1 0
> 0 2.42e3 0.00
> nat-ha-process event wait 0 0
> 1 1.63e4 0.00
> nat64-expire-walk event wait 0 0
> 1 1.44e4 0.00
> nsh-md2-ioam-export-process any wait 0 0
> 1 3.86e4 0.00
> perfmon-periodic-process event wait 0 0
> 1 1.15e8 0.00
> rd-cp-process any wait 0 0
> 1 2.75e4 0.00
> send-dhcp6-client-message-proc any wait 0 0
> 1 6.52e3 0.00
> send-dhcp6-pd-client-message-p any wait 0 0
> 1 6.29e3 0.00
> send-rs-process any wait 0 0
> 1 6.07e3 0.00
> session-queue-process any wait 0 0
> 88 3.15e7 0.00
> startup-config-process done 1 0
> 1 7.44e3 0.00
> statseg-collector-process time wait 0 0
> 10 4.64e4 0.00
> udp-ping-process any wait 0 0
> 1 1.78e4 0.00
> unix-cli-stdin active 0 0
> 23 1.57e8 0.00
> unix-epoll-input polling 234102 0
> 0 8.81e5 0.00
> vhost-user-process any wait 0 0
> 1 3.18e4 0.00
> vhost-user-send-interrupt-proc any wait 0 0
> 1 6.59e3 0.00
> vpe-link-state-process event wait 0 0
> 2 5.77e3 0.00
> vxlan-gpe-ioam-export-process any wait 0 0
> 1 7.51e3 0.00
> wildcard-ip4-arp-publisher-pro event wait 0 0
> 1 2.52e4 0.00
> ---------------
> Thread 1 vpp_wk_0 (lcore 4)
> Time 94.7, average vectors/node 29.61, last 128 main loops 0.00 per node 0.00
> vector rates in 4.0237e4, out 3.0603e4, drop 6.2283e-1, punt 0.0000e0
> Name State Calls Vectors
> Suspends Clocks Vectors/Call
> GigabitEthernet0/3/0-output active 105001 2899214
> 0 3.65e2 27.61
> GigabitEthernet0/3/0-tx active 105001 2898993
> 0 2.02e2 27.61
> arp-input active 1 1
> 0 1.56e4 1.00
> arp-reply active 1 1
> 0 4.51e4 1.00
> dpdk-input polling 567157066 912390
> 0 5.93e4 0.00
> drop active 47 59
> 0 2.29e3 1.26
> error-drop active 47 59
> 0 3.20e3 1.26
> ethernet-input active 28266 912390
> 0 3.81e1 32.28
> ip4-drop active 1 1
> 0 1.12e4 1.00
> ip4-input active 28236 912332
> 0 5.31e1 32.31
> ip4-local active 28235 912331
> 0 5.61e1 32.31
> ip4-lookup active 120215 3811545
> 0 3.74e1 31.71
> ip4-mfib-forward-lookup active 1 1
> 0 3.89e4 1.00
> ip4-mfib-forward-rpf active 1 1
> 0 2.90e3 1.00
> ip4-rewrite active 105001 2899214
> 0 3.07e1 27.61
> llc-input active 45 57
> 0 1.62e3 1.27
> session-queue polling 447161920 2899214
> 0 3.79e4 0.00
> tcp4-established active 28233 912329
> 0 2.52e2 32.31
> tcp4-input active 28235 912331
> 0 9.77e1 32.31
> tcp4-output active 105001 2899214
> 0 1.21e2 27.61
> tcp4-syn-sent active 2 2
> 0 9.87e5 1.00
> unix-epoll-input polling 553333 0
> 0 1.57e3 0.00
> ---------------
> Thread 2 vpp_wk_1 (lcore 5)
> Time 94.7, average vectors/node 0.00, last 128 main loops 0.00 per node 0.00
> vector rates in 0.0000e0, out 0.0000e0, drop 0.0000e0, punt 0.0000e0
> Name State Calls Vectors
> Suspends Clocks Vectors/Call
> session-queue polling 717178839 0
> 0 2.08e2 0.00
> unix-epoll-input polling 720515 0
> 0 2.09e4 0.00
> ---------------
> Thread 3 vpp_wk_2 (lcore 6)
> Time 94.7, average vectors/node 0.00, last 128 main loops 0.00 per node 0.00
> vector rates in 0.0000e0, out 0.0000e0, drop 0.0000e0, punt 0.0000e0
> Name State Calls Vectors
> Suspends Clocks Vectors/Call
> session-queue polling 733366473 0
> 0 2.02e2 0.00
> unix-epoll-input polling 741268 0
> 0 2.04e4 0.00
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
>
> View/Reply Online (#14884): https://lists.fd.io/g/vpp-dev/message/14884
> Mute This Topic: https://lists.fd.io/mt/65863639/675152
> Group Owner: vpp-dev+ow...@lists.fd.io
> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [fcoras.li...@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14885): https://lists.fd.io/g/vpp-dev/message/14885
Mute This Topic: https://lists.fd.io/mt/65863639/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
