Hi Dom, [traveling so a quick reply]
For some reason, your rx/tx fifos (see nitems), and implicitly the snd and rcv wnd, are 64kB in your logs lower. Is this the tcp echo or iperf result? Regards, Florin > On Dec 4, 2019, at 7:29 AM, dch...@akouto.com wrote: > > Hi, > > Thank you Florin and Jerome for your time, very much appreciated. > > For VCL configuration, FIFO sizes are 16 MB > "show session verbose 2" does not indicate any retransmissions. Here are the > numbers during a test run where approx. 9 GB were transferred (the difference > in values between client and server is just because it took me a few seconds > to issue the command on the client side as you can see from the duration): > SERVER SIDE: > stats: in segs 5989307 dsegs 5989306 bytes 8544661342 dupacks 0 > out segs 3942513 dsegs 0 bytes 0 dupacks 0 > fr 0 tr 0 rxt segs 0 bytes 0 duration 106.489 > err wnd data below 0 above 0 ack below 0 above 0 > CLIENT SIDE: > stats: in segs 4207793 dsegs 0 bytes 0 dupacks 0 > out segs 6407444 dsegs 6407443 bytes 9141373892 dupacks 0 > fr 0 tr 0 rxt segs 0 bytes 0 duration 114.113 > err wnd data below 0 above 0 ack below 0 above 0 > sh int does not seem to indicate any issue. There are occasional drops but I > enabled tracing and checked those out, they are LLC BPDU's, I'm not sure > where those are coming from but I suspect they are from linuxbridge in the > compute host where the VMs are running. > @Jerome: Before I use the dpdk-devbind command to make the interfaces > available to VPP, they use virtio drivers. When assigned to VPP they use > uio_pci_generic. > > I'm not sure if any other stats might be useful so I'm just pasting a bunch > of stats & information from the client & server instances below, I know it's > a lot, just putting it here in case there is something useful in there. > Thanks again for taking the time to follow-up with me and for the > suggestions, I really do appreciate it very much! > > Regards, > Dom > > ############################################################################# > # Interface uses virtio-pci when the iperf3 test is run using regular Linux > # networking. > ############################################################################# > [root@vpp-test-1 centos]# dpdk-devbind --status > > Network devices using kernel driver > =================================== > 0000:00:03.0 'Virtio network device 1000' if=eth0 drv=virtio-pci > unused=virtio_pci *Active* > 0000:00:04.0 'Virtio network device 1000' if=eth1 drv=virtio-pci > unused=virtio_pci *Active* > > ############################################################################# > # Interface uses uio_pci_generic when set up for VPP > ############################################################################# > > [root@vpp-test-1 centos]# dpdk-devbind --status > > Network devices using DPDK-compatible driver > ============================================ > 0000:00:03.0 'Virtio network device 1000' drv=uio_pci_generic > unused=virtio_pci > > Network devices using kernel driver > =================================== > 0000:00:04.0 'Virtio network device 1000' if=eth1 drv=virtio-pci > unused=virtio_pci,uio_pci_generic *Active* > > > vpp# sh hardware-interfaces > Name Idx Link Hardware > GigabitEthernet0/3/0 1 up GigabitEthernet0/3/0 > Link speed: 10 Gbps > Ethernet address fa:16:3e:10:5e:4b > Red Hat Virtio > carrier up full duplex mtu 9206 > flags: admin-up pmd maybe-multiseg > rx: queues 1 (max 1), desc 256 (min 0 max 65535 align 1) > tx: queues 1 (max 1), desc 256 (min 0 max 65535 align 1) > pci: device 1af4:1000 subsystem 1af4:0001 address 0000:00:03.00 numa 0 > max rx packet len: 9728 > promiscuous: unicast off all-multicast on > vlan offload: strip off filter off qinq off > rx offload avail: vlan-strip udp-cksum tcp-cksum tcp-lro vlan-filter > jumbo-frame > rx offload active: jumbo-frame > tx offload avail: vlan-insert udp-cksum tcp-cksum tcp-tso multi-segs > tx offload active: multi-segs > rss avail: none > rss active: none > tx burst function: virtio_xmit_pkts > rx burst function: virtio_recv_mergeable_pkts > > rx frames ok 467 > rx bytes ok 27992 > extended stats: > rx good packets 467 > rx good bytes 27992 > rx q0packets 467 > rx q0bytes 27992 > rx q0 good packets 467 > rx q0 good bytes 27992 > rx q0 multicast packets 465 > rx q0 broadcast packets 2 > rx q0 undersize packets 467 > > > ############################################################################# > # Dropped packets are LLC BPDUs, not sure but probably a linuxbridge thing > ############################################################################# > vpp# show trace > ------------------- Start of thread 0 vpp_main ------------------- > No packets in trace buffer > ------------------- Start of thread 1 vpp_wk_0 ------------------- > Packet 1 > > 00:08:35:202159: dpdk-input > GigabitEthernet0/3/0 rx queue 0 > buffer 0xfee2f4: current data 0, length 60, buffer-pool 0, ref-count 1, > totlen-nifb 0, trace handle 0x1000000 > ext-hdr-valid > l4-cksum-computed l4-cksum-correct > PKT MBUF: port 0, nb_segs 1, pkt_len 60 > buf_len 4224, data_len 60, ol_flags 0x0, data_off 128, phys_addr > 0xffb8bd80 > packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 > rss 0x0 fdir.hi 0x0 fdir.lo 0x0 > 0x0027: f8:b1:56:7d:61:58 -> 01:80:c2:00:00:00 > 00:08:35:202184: ethernet-input > frame: flags 0x1, hw-if-index 1, sw-if-index 1 > 0x0027: f8:b1:56:7d:61:58 -> 01:80:c2:00:00:00 > 00:08:35:202190: llc-input > LLC bpdu -> bpdu > 00:08:35:202205: error-drop > rx:GigabitEthernet0/3/0 > 00:08:35:202207: drop > llc-input: unknown llc ssap/dsap > > ############################################################################# > # Stats before test > ############################################################################# > ###### SERVER > vpp# sh int > Name Idx State MTU (L3/IP4/IP6/MPLS) > Counter Count > GigabitEthernet0/3/0 1 up 9000/0/0/0 rx > packets 12 > rx bytes > 720 > drops > 12 > > vpp# show session verbose 2 > Thread 0: no sessions > Thread 1: no sessions > Thread 2: no sessions > Thread 3: no sessions > Thread 4: no sessions > Thread 5: no sessions > > ### CLIENT > vpp# sh int > Name Idx State MTU (L3/IP4/IP6/MPLS) > Counter Count > GigabitEthernet0/3/0 1 up 9000/0/0/0 rx > packets 18 > rx bytes > 1080 > drops > 18 > local0 0 down 0/0/0/0 > > vpp# show session verbose 2 > Thread 0: no sessions > Thread 1: no sessions > Thread 2: no sessions > Thread 3: no sessions > Thread 4: no sessions > Thread 5: no sessions > > ############################################################################# > # Stats AFTER test > ############################################################################# > ###### SERVER > vpp# sh int > Name Idx State MTU (L3/IP4/IP6/MPLS) > Counter Count > GigabitEthernet0/3/0 1 up 9000/0/0/0 rx > packets 5960412 > rx bytes > 8896698636 > tx > packets 3923484 > tx bytes > 258949928 > drops > 77 > ip4 > 5960335 > local0 0 down 0/0/0/0 > > vpp# show session verbose 2 > [0:0][T] 10.0.0.153:5555->0.0.0.0:0 LISTEN > Thread 0: active sessions 1 > [1:0][T] 10.0.0.153:5555->10.0.0.151:61592 ESTABLISHED > index: 0 flags: timers: > snd_una 1 snd_nxt 1 snd_una_max 1 rcv_nxt 4249694047 rcv_las 4249694047 > snd_wnd 64512 rcv_wnd 63488 rcv_wscale 10 snd_wl1 4249692619 snd_wl2 1 > flight size 0 out space 4290 rcv_wnd_av 63488 tsval_recent 87602812 > tsecr 87652914 tsecr_last_ack 87652914 tsval_recent_age 4294967074 snd_mss > 1428 > rto 200 rto_boff 0 srtt 1 us 1.196 rttvar 0 rtt_ts 0.0000 rtt_seq 1135896793 > cong: none algo cubic cwnd 4290 ssthresh 4194304 bytes_acked 0 > cc space 4290 prev_cwnd 0 prev_ssthresh 0 rtx_bytes 0 > snd_congestion 1135896793 dupack 0 limited_transmit 1135896793 > sboard: sacked_bytes 0 last_sacked_bytes 0 lost_bytes 0 > last_bytes_delivered 0 high_sacked 1135896793 snd_una_adv 0 > cur_rxt_hole 4294967295 high_rxt 1135896793 rescue_rxt 1135896793 > stats: in segs 5989307 dsegs 5989306 bytes 8544661342 dupacks 0 > out segs 3942513 dsegs 0 bytes 0 dupacks 0 > fr 0 tr 0 rxt segs 0 bytes 0 duration 106.489 > err wnd data below 0 above 0 ack below 0 above 0 > pacer: bucket 13532532 tokens/period .061 last_update d88e7922 > Rx fifo: cursize 0 nitems 65534 has_event 0 > head 11437 tail 11437 segment manager 2 > vpp session 0 thread 1 app session 199999 thread 0 > ooo pool 0 active elts newest 4294967295 > Tx fifo: cursize 0 nitems 65534 has_event 0 > head 0 tail 0 segment manager 2 > vpp session 0 thread 1 app session 199999 thread 0 > ooo pool 0 active elts newest 0 > Thread 1: active sessions 1 > Thread 2: no sessions > Thread 3: no sessions > Thread 4: no sessions > Thread 5: no sessions > > ###### CLIENT > vpp# sh int > Name Idx State MTU (L3/IP4/IP6/MPLS) > Counter Count > GigabitEthernet0/3/0 1 up 9000/0/0/0 rx > packets 4195035 > rx bytes > 276871796 > tx > packets 6385403 > tx bytes > 9531349660 > drops > 84 > ip4 > 4194951 > local0 0 down 0/0/0/0 drops > 1 > > vpp# show session verbose 2 > Thread 0: no sessions > [1:0][T] 10.0.0.151:61592->10.0.0.153:5555 ESTABLISHED > index: 0 flags: timers: RETRANSMIT > snd_una 551376469 snd_nxt 551439301 snd_una_max 551439301 rcv_nxt 1 rcv_las 1 > snd_wnd 63488 rcv_wnd 64512 rcv_wscale 10 snd_wl1 1 snd_wl2 551376469 > flight size 62832 out space 656 rcv_wnd_av 64512 tsval_recent 87660537 > tsecr 87610434 tsecr_last_ack 87610434 tsval_recent_age 4294967183 snd_mss > 1428 > rto 200 rto_boff 0 srtt 1 us .708 rttvar 1 rtt_ts 167.5240 rtt_seq 551393605 > cong: none algo newreno cwnd 4195240 ssthresh 4194304 bytes_acked 1428 > cc space 656 prev_cwnd 0 prev_ssthresh 0 rtx_bytes 0 > snd_congestion 3019902672 dupack 0 limited_transmit 3019897507 > sboard: sacked_bytes 0 last_sacked_bytes 0 lost_bytes 0 > last_bytes_delivered 0 high_sacked 3019897507 snd_una_adv 0 > cur_rxt_hole 4294967295 high_rxt 3019897507 rescue_rxt 3019897507 > stats: in segs 4207793 dsegs 0 bytes 0 dupacks 0 > out segs 6407444 dsegs 6407443 bytes 9141373892 dupacks 0 > fr 0 tr 0 rxt segs 0 bytes 0 duration 114.113 > err wnd data below 0 above 0 ack below 0 above 0 > pacer: bucket 3321031074 tokens/period 2756.792 last_update d31de7db > Rx fifo: cursize 0 nitems 65534 has_event 0 > head 0 tail 0 segment manager 2 > vpp session 0 thread 1 app session 0 thread 0 > ooo pool 0 active elts newest 0 > Tx fifo: cursize 65534 nitems 65534 has_event 1 > head 30515 tail 30514 segment manager 2 > vpp session 0 thread 1 app session 0 thread 0 > ooo pool 0 active elts newest 4294967295 > Thread 1: active sessions 1 > Thread 2: no sessions > Thread 3: no sessions > Thread 4: no sessions > Thread 5: no sessions > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#14794): https://lists.fd.io/g/vpp-dev/message/14794 > Mute This Topic: https://lists.fd.io/mt/65863639/675152 > Group Owner: vpp-dev+ow...@lists.fd.io > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [fcoras.li...@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14802): https://lists.fd.io/g/vpp-dev/message/14802 Mute This Topic: https://lists.fd.io/mt/65863639/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
