Hi Dom,
Great to see progress! More inline.
> On Dec 6, 2019, at 10:21 AM, dch...@akouto.com wrote:
>
> Hi Florin,
>
> Some progress, at least with the built-in echo app, thank you for all the
> suggestions so far! By adjusting the fifo-size and testing in half-duplex I
> was able to get close to 5 Gbps between the two openstack instances using the
> built-in test echo app:
>
> vpp# test echo clients gbytes 1 no-return fifo-size 1000000 uri
> tcp://10.0.0.156/5555
FC: The cli for the echo apps is a bit confusing. Whatever you pass above is
left shifted by 10 (multiplied by 1024) so that’s why I suggested to use 4096
(~4MB). You can also use larger values, but above you are asking for ~1GB :-)
> 1 three-way handshakes in .26 seconds 3.86/s
> Test started at 745.163085
> Test finished at 746.937343
> 1073741824 bytes (1024 mbytes, 1 gbytes) in 1.77 seconds
> 605177784.33 bytes/second half-duplex
> 4.8414 gbit/second half-duplex
>
> I need to get closer to 10 Gbps but at least there is good proof that the
> issue is related to configuration / tuning. So, I switched back to iperf
> testing with VCL, and I'm back to 600 Mbps, even though I can confirm that
> the fifo sizes match what is configured in vcl.conf (note that in this test
> run I changed that to 8 Mb each for rx and tx from the previous 16, but
> results are the same when I use 16 Mb). I'm obviously missing something in
> the configuration but I can't imagine what that might be. Below is my exact
> startup.conf, vcl.conf and output from show session from this iperf run to
> give the full picture, hopefully something jumps out as missing in my
> configuration. Thank you for your patience and support with this, much
> appreciated!
FC: Not entirely sure what the issue is but some things can be improved. More
lower.
>
> [root@vpp-test-1 centos]# cat vcl.conf
> vcl {
> rx-fifo-size 8000000
> tx-fifo-size 8000000
> app-scope-local
> app-scope-global
> api-socket-name /tmp/vpp-api.sock
> }
FC: This looks okay.
>
> [root@vpp-test-1 centos]# cat /etc/vpp/startup.conf
> unix {
> nodaemon
> log /var/log/vpp/vpp.log
> full-coredump
> cli-listen /run/vpp/cli.sock
> gid vpp
> interactive
> }
> dpdk {
> dev 0000:00:03.0{
> num-rx-desc 65535
> num-tx-desc 65535
FC: Not sure about this. I don’t have any experience with vhost interfaces, but
for XL710s I typically use 256 descriptors. It might be too low if you start
noticing lots of rx/tx drops with “show int”.
> }
> }
> session { evt_qs_memfd_seg }
> socksvr { socket-name /tmp/vpp-api.sock }
> api-trace {
> on
> }
> api-segment {
> gid vpp
> }
> cpu {
> main-core 7
> corelist-workers 4-6
> workers 3
FC: For starters, could you try this out with only 1 worker, since you’re
testing with 1 connection.
Also, did you try pinning iperf with taskset to a worker on the same numa like
your vpp workers, in case you have multiple numas? Check with lscpu your cpu
into numa distribution.
You may want to pin iperf even if you have only one numa, just to be sure it
won’t be scheduled by mistake on the cores vpp is using.
> }
> buffers {
> ## Increase number of buffers allocated, needed only in scenarios with
> ## large number of interfaces and worker threads. Value is per numa
> node.
> ## Default is 16384 (8192 if running unpriviledged)
> buffers-per-numa 128000
FC: For simple testing I only use 16k, but this value actually depends on the
number of rx/tx descriptors you have configured.
>
> ## Size of buffer data area
> ## Default is 2048
> default data-size 8192
FC: Are you trying to use jumbo buffers? You need to add to the tcp stanza,
i.e., tcp { mtu <max value in bytes> }. But for starters don’t modify the
buffer size, just to get an idea of where performance is without this.
Afterwards, as Jerome suggested, you may want to try tso by enabling it for
tcp, i.e., tcp { tso } in startup.conf and enabling tso for the nic by adding
“tso on” to the nic’s dpdk stanza (if the nic actually supports it). You don’t
need to change the buffer size for that.
> }
>
> vpp# sh session verbose 2
> Thread 0: no sessions
> [1:0][T] 10.0.0.152:41737->10.0.0.156:5201 ESTABLISHED
> index: 0 flags: timers:
> snd_una 124 snd_nxt 124 snd_una_max 124 rcv_nxt 5 rcv_las 5
> snd_wnd 7999488 rcv_wnd 7999488 rcv_wscale 10 snd_wl1 4 snd_wl2 124
> flight size 0 out space 4413 rcv_wnd_av 7999488 tsval_recent 12893009
> tsecr 10757431 tsecr_last_ack 10757431 tsval_recent_age 1995 snd_mss 1428
> rto 200 rto_boff 0 srtt 3 us 3.887 rttvar 2 rtt_ts 0.0000 rtt_seq 124
> cong: none algo newreno cwnd 4413 ssthresh 4194304 bytes_acked 0
> cc space 4413 prev_cwnd 0 prev_ssthresh 0 rtx_bytes 0
> snd_congestion 1736877166 dupack 0 limited_transmit 1736877166
> sboard: sacked_bytes 0 last_sacked_bytes 0 lost_bytes 0
> last_bytes_delivered 0 high_sacked 1736877166 snd_una_adv 0
> cur_rxt_hole 4294967295 high_rxt 1736877166 rescue_rxt 1736877166
> stats: in segs 7 dsegs 4 bytes 4 dupacks 0
> out segs 7 dsegs 2 bytes 123 dupacks 0
> fr 0 tr 0 rxt segs 0 bytes 0 duration 2.484
> err wnd data below 0 above 0 ack below 0 above 0
> pacer: bucket 42459 tokens/period .685 last_update 61908201
> Rx fifo: cursize 0 nitems 7999999 has_event 0
> head 4 tail 4 segment manager 3
> vpp session 0 thread 1 app session 0 thread 0
> ooo pool 0 active elts newest 4294967295
> Tx fifo: cursize 0 nitems 7999999 has_event 0
> head 123 tail 123 segment manager 3
> vpp session 0 thread 1 app session 0 thread 0
> ooo pool 0 active elts newest 4294967295
> [1:1][T] 10.0.0.152:53460->10.0.0.156:5201 ESTABLISHED
> index: 1 flags: PSH pending timers: RETRANSMIT
> snd_una 160482962 snd_nxt 160735718 snd_una_max 160735718 rcv_nxt 1 rcv_las 1
> snd_wnd 7999488 rcv_wnd 7999488 rcv_wscale 10 snd_wl1 1 snd_wl2 160482962
> flight size 252756 out space 714 rcv_wnd_av 7999488 tsval_recent 12895476
> tsecr 10759907 tsecr_last_ack 10759907 tsval_recent_age 4294966825 snd_mss
> 1428
> rto 200 rto_boff 0 srtt 1 us 3.418 rttvar 2 rtt_ts 42.0588 rtt_seq 160485818
> cong: none algo newreno cwnd 253470 ssthresh 187782 bytes_acked 2856
FC: You’re using newreno congestion control algorithm, add tcp { cc-algo cubic
} to your startup.conf to change it to cubic.
> cc space 714 prev_cwnd 382704 prev_ssthresh 187068 rtx_bytes 0
> snd_congestion 150237062 dupack 0 limited_transmit 817908495
> sboard: sacked_bytes 0 last_sacked_bytes 0 lost_bytes 0
> last_bytes_delivered 0 high_sacked 150242774 snd_una_adv 0
> cur_rxt_hole 4294967295 high_rxt 150235634 rescue_rxt 149855785
> stats: in segs 84958 dsegs 0 bytes 0 dupacks 1237
> out segs 112747 dsegs 112746 bytes 160999897 dupacks 0
> fr 5 tr 0 rxt segs 185 bytes 264180 duration 2.473
FC: I assume this is the sender. It seems you entered fast retransmit 5 times
in 2.4 seconds and you received 1237 duplicate acks. It’s a sign that something
is losing packets. Do you see any rx/tx errors if you check with “sh int”?
Alternatively, the vswitch might be losing packets.
Regards,
Florin
> err wnd data below 0 above 0 ack below 0 above 0
> pacer: bucket 22180207 tokens/period 117.979 last_update 61e173e5
> Rx fifo: cursize 0 nitems 7999999 has_event 0
> head 0 tail 0 segment manager 3
> vpp session 1 thread 1 app session 1 thread 0
> ooo pool 0 active elts newest 0
> Tx fifo: cursize 7999999 nitems 7999999 has_event 1
> head 482961 tail 482960 segment manager 3
> vpp session 1 thread 1 app session 1 thread 0
> ooo pool 0 active elts newest 4294967295
> Thread 1: active sessions 2
> Thread 2: no sessions
> Thread 3: no sessions
> vpp#
> Regards,
> Dom -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
>
> View/Reply Online (#14827): https://lists.fd.io/g/vpp-dev/message/14827
> Mute This Topic: https://lists.fd.io/mt/65863639/675152
> Group Owner: vpp-dev+ow...@lists.fd.io
> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [fcoras.li...@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14828): https://lists.fd.io/g/vpp-dev/message/14828
Mute This Topic: https://lists.fd.io/mt/65863639/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
