Are you using VPP native virtio or DPDK virtio ?
Jerome
De : <vpp-dev@lists.fd.io> au nom de "dch...@akouto.com" <dch...@akouto.com>
Date : mercredi 4 décembre 2019 à 16:29
À : "vpp-dev@lists.fd.io" <vpp-dev@lists.fd.io>
Objet : Re: [vpp-dev] VPP / tcp_echo performance
Hi,
Thank you Florin and Jerome for your time, very much appreciated.
· For VCL configuration, FIFO sizes are 16 MB
· "show session verbose 2" does not indicate any retransmissions. Here
are the numbers during a test run where approx. 9 GB were transferred (the
difference in values between client and server is just because it took me a few
seconds to issue the command on the client side as you can see from the
duration):
SERVER SIDE:
stats: in segs 5989307 dsegs 5989306 bytes 8544661342 dupacks 0
out segs 3942513 dsegs 0 bytes 0 dupacks 0
fr 0 tr 0 rxt segs 0 bytes 0 duration 106.489
err wnd data below 0 above 0 ack below 0 above 0
CLIENT SIDE:
stats: in segs 4207793 dsegs 0 bytes 0 dupacks 0
out segs 6407444 dsegs 6407443 bytes 9141373892 dupacks 0
fr 0 tr 0 rxt segs 0 bytes 0 duration 114.113
err wnd data below 0 above 0 ack below 0 above 0
· sh int does not seem to indicate any issue. There are occasional
drops but I enabled tracing and checked those out, they are LLC BPDU's, I'm not
sure where those are coming from but I suspect they are from linuxbridge in the
compute host where the VMs are running.
· @Jerome: Before I use the dpdk-devbind command to make the interfaces
available to VPP, they use virtio drivers. When assigned to VPP they use
uio_pci_generic.
I'm not sure if any other stats might be useful so I'm just pasting a bunch of
stats & information from the client & server instances below, I know it's a
lot, just putting it here in case there is something useful in there. Thanks
again for taking the time to follow-up with me and for the suggestions, I
really do appreciate it very much!
Regards,
Dom
#############################################################################
# Interface uses virtio-pci when the iperf3 test is run using regular Linux
# networking.
#############################################################################
[root@vpp-test-1 centos]# dpdk-devbind --status
Network devices using kernel driver
===================================
0000:00:03.0 'Virtio network device 1000' if=eth0 drv=virtio-pci
unused=virtio_pci *Active*
0000:00:04.0 'Virtio network device 1000' if=eth1 drv=virtio-pci
unused=virtio_pci *Active*
#############################################################################
# Interface uses uio_pci_generic when set up for VPP
#############################################################################
[root@vpp-test-1 centos]# dpdk-devbind --status
Network devices using DPDK-compatible driver
============================================
0000:00:03.0 'Virtio network device 1000' drv=uio_pci_generic unused=virtio_pci
Network devices using kernel driver
===================================
0000:00:04.0 'Virtio network device 1000' if=eth1 drv=virtio-pci
unused=virtio_pci,uio_pci_generic *Active*
vpp# sh hardware-interfaces
Name Idx Link Hardware
GigabitEthernet0/3/0 1 up GigabitEthernet0/3/0
Link speed: 10 Gbps
Ethernet address fa:16:3e:10:5e:4b
Red Hat Virtio
carrier up full duplex mtu 9206
flags: admin-up pmd maybe-multiseg
rx: queues 1 (max 1), desc 256 (min 0 max 65535 align 1)
tx: queues 1 (max 1), desc 256 (min 0 max 65535 align 1)
pci: device 1af4:1000 subsystem 1af4:0001 address 0000:00:03.00 numa 0
max rx packet len: 9728
promiscuous: unicast off all-multicast on
vlan offload: strip off filter off qinq off
rx offload avail: vlan-strip udp-cksum tcp-cksum tcp-lro vlan-filter
jumbo-frame
rx offload active: jumbo-frame
tx offload avail: vlan-insert udp-cksum tcp-cksum tcp-tso multi-segs
tx offload active: multi-segs
rss avail: none
rss active: none
tx burst function: virtio_xmit_pkts
rx burst function: virtio_recv_mergeable_pkts
rx frames ok 467
rx bytes ok 27992
extended stats:
rx good packets 467
rx good bytes 27992
rx q0packets 467
rx q0bytes 27992
rx q0 good packets 467
rx q0 good bytes 27992
rx q0 multicast packets 465
rx q0 broadcast packets 2
rx q0 undersize packets 467
#############################################################################
# Dropped packets are LLC BPDUs, not sure but probably a linuxbridge thing
#############################################################################
vpp# show trace
------------------- Start of thread 0 vpp_main -------------------
No packets in trace buffer
------------------- Start of thread 1 vpp_wk_0 -------------------
Packet 1
00:08:35:202159: dpdk-input
GigabitEthernet0/3/0 rx queue 0
buffer 0xfee2f4: current data 0, length 60, buffer-pool 0, ref-count 1,
totlen-nifb 0, trace handle 0x1000000
ext-hdr-valid
l4-cksum-computed l4-cksum-correct
PKT MBUF: port 0, nb_segs 1, pkt_len 60
buf_len 4224, data_len 60, ol_flags 0x0, data_off 128, phys_addr 0xffb8bd80
packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0x0 fdir.hi 0x0 fdir.lo 0x0
0x0027: f8:b1:56:7d:61:58 -> 01:80:c2:00:00:00
00:08:35:202184: ethernet-input
frame: flags 0x1, hw-if-index 1, sw-if-index 1
0x0027: f8:b1:56:7d:61:58 -> 01:80:c2:00:00:00
00:08:35:202190: llc-input
LLC bpdu -> bpdu
00:08:35:202205: error-drop
rx:GigabitEthernet0/3/0
00:08:35:202207: drop
llc-input: unknown llc ssap/dsap
#############################################################################
# Stats before test
#############################################################################
###### SERVER
vpp# sh int
Name Idx State MTU (L3/IP4/IP6/MPLS)
Counter Count
GigabitEthernet0/3/0 1 up 9000/0/0/0 rx packets
12
rx bytes
720
drops
12
vpp# show session verbose 2
Thread 0: no sessions
Thread 1: no sessions
Thread 2: no sessions
Thread 3: no sessions
Thread 4: no sessions
Thread 5: no sessions
### CLIENT
vpp# sh int
Name Idx State MTU (L3/IP4/IP6/MPLS)
Counter Count
GigabitEthernet0/3/0 1 up 9000/0/0/0 rx packets
18
rx bytes
1080
drops
18
local0 0 down 0/0/0/0
vpp# show session verbose 2
Thread 0: no sessions
Thread 1: no sessions
Thread 2: no sessions
Thread 3: no sessions
Thread 4: no sessions
Thread 5: no sessions
#############################################################################
# Stats AFTER test
#############################################################################
###### SERVER
vpp# sh int
Name Idx State MTU (L3/IP4/IP6/MPLS)
Counter Count
GigabitEthernet0/3/0 1 up 9000/0/0/0 rx packets
5960412
rx bytes
8896698636
tx packets
3923484
tx bytes
258949928
drops
77
ip4
5960335
local0 0 down 0/0/0/0
vpp# show session verbose 2
[0:0][T] 10.0.0.153:5555->0.0.0.0:0 LISTEN
Thread 0: active sessions 1
[1:0][T] 10.0.0.153:5555->10.0.0.151:61592 ESTABLISHED
index: 0 flags: timers:
snd_una 1 snd_nxt 1 snd_una_max 1 rcv_nxt 4249694047 rcv_las 4249694047
snd_wnd 64512 rcv_wnd 63488 rcv_wscale 10 snd_wl1 4249692619 snd_wl2 1
flight size 0 out space 4290 rcv_wnd_av 63488 tsval_recent 87602812
tsecr 87652914 tsecr_last_ack 87652914 tsval_recent_age 4294967074 snd_mss 1428
rto 200 rto_boff 0 srtt 1 us 1.196 rttvar 0 rtt_ts 0.0000 rtt_seq 1135896793
cong: none algo cubic cwnd 4290 ssthresh 4194304 bytes_acked 0
cc space 4290 prev_cwnd 0 prev_ssthresh 0 rtx_bytes 0
snd_congestion 1135896793 dupack 0 limited_transmit 1135896793
sboard: sacked_bytes 0 last_sacked_bytes 0 lost_bytes 0
last_bytes_delivered 0 high_sacked 1135896793 snd_una_adv 0
cur_rxt_hole 4294967295 high_rxt 1135896793 rescue_rxt 1135896793
stats: in segs 5989307 dsegs 5989306 bytes 8544661342 dupacks 0
out segs 3942513 dsegs 0 bytes 0 dupacks 0
fr 0 tr 0 rxt segs 0 bytes 0 duration 106.489
err wnd data below 0 above 0 ack below 0 above 0
pacer: bucket 13532532 tokens/period .061 last_update d88e7922
Rx fifo: cursize 0 nitems 65534 has_event 0
head 11437 tail 11437 segment manager 2
vpp session 0 thread 1 app session 199999 thread 0
ooo pool 0 active elts newest 4294967295
Tx fifo: cursize 0 nitems 65534 has_event 0
head 0 tail 0 segment manager 2
vpp session 0 thread 1 app session 199999 thread 0
ooo pool 0 active elts newest 0
Thread 1: active sessions 1
Thread 2: no sessions
Thread 3: no sessions
Thread 4: no sessions
Thread 5: no sessions
###### CLIENT
vpp# sh int
Name Idx State MTU (L3/IP4/IP6/MPLS)
Counter Count
GigabitEthernet0/3/0 1 up 9000/0/0/0 rx packets
4195035
rx bytes
276871796
tx packets
6385403
tx bytes
9531349660
drops
84
ip4
4194951
local0 0 down 0/0/0/0 drops
1
vpp# show session verbose 2
Thread 0: no sessions
[1:0][T] 10.0.0.151:61592->10.0.0.153:5555 ESTABLISHED
index: 0 flags: timers: RETRANSMIT
snd_una 551376469 snd_nxt 551439301 snd_una_max 551439301 rcv_nxt 1 rcv_las 1
snd_wnd 63488 rcv_wnd 64512 rcv_wscale 10 snd_wl1 1 snd_wl2 551376469
flight size 62832 out space 656 rcv_wnd_av 64512 tsval_recent 87660537
tsecr 87610434 tsecr_last_ack 87610434 tsval_recent_age 4294967183 snd_mss 1428
rto 200 rto_boff 0 srtt 1 us .708 rttvar 1 rtt_ts 167.5240 rtt_seq 551393605
cong: none algo newreno cwnd 4195240 ssthresh 4194304 bytes_acked 1428
cc space 656 prev_cwnd 0 prev_ssthresh 0 rtx_bytes 0
snd_congestion 3019902672 dupack 0 limited_transmit 3019897507
sboard: sacked_bytes 0 last_sacked_bytes 0 lost_bytes 0
last_bytes_delivered 0 high_sacked 3019897507 snd_una_adv 0
cur_rxt_hole 4294967295 high_rxt 3019897507 rescue_rxt 3019897507
stats: in segs 4207793 dsegs 0 bytes 0 dupacks 0
out segs 6407444 dsegs 6407443 bytes 9141373892 dupacks 0
fr 0 tr 0 rxt segs 0 bytes 0 duration 114.113
err wnd data below 0 above 0 ack below 0 above 0
pacer: bucket 3321031074 tokens/period 2756.792 last_update d31de7db
Rx fifo: cursize 0 nitems 65534 has_event 0
head 0 tail 0 segment manager 2
vpp session 0 thread 1 app session 0 thread 0
ooo pool 0 active elts newest 0
Tx fifo: cursize 65534 nitems 65534 has_event 1
head 30515 tail 30514 segment manager 2
vpp session 0 thread 1 app session 0 thread 0
ooo pool 0 active elts newest 4294967295
Thread 1: active sessions 1
Thread 2: no sessions
Thread 3: no sessions
Thread 4: no sessions
Thread 5: no sessions
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14797): https://lists.fd.io/g/vpp-dev/message/14797
Mute This Topic: https://lists.fd.io/mt/65863639/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
