Hi, Thank you Florin and Jerome for your time, very much appreciated.
* For VCL configuration, FIFO sizes are 16 MB * "show session verbose 2" does not indicate any retransmissions. Here are the numbers during a test run where approx. 9 GB were transferred (the difference in values between client and server is just because it took me a few seconds to issue the command on the client side as you can see from the duration): *SERVER SIDE:* stats: in segs 5989307 dsegs 5989306 bytes 8544661342 dupacks 0 out segs 3942513 dsegs 0 bytes 0 dupacks 0 fr 0 tr 0 rxt segs 0 bytes 0 duration 106.489 err wnd data below 0 above 0 ack below 0 above 0 *CLIENT SIDE:* stats: in segs 4207793 dsegs 0 bytes 0 dupacks 0 out segs 6407444 dsegs 6407443 bytes 9141373892 dupacks 0 fr 0 tr 0 rxt segs 0 bytes 0 duration 114.113 err wnd data below 0 above 0 ack below 0 above 0 * sh int does not seem to indicate any issue. There are occasional drops but I enabled tracing and checked those out, they are LLC BPDU's, I'm not sure where those are coming from but I suspect they are from linuxbridge in the compute host where the VMs are running. * *@Jerome* : Before I use the dpdk-devbind command to make the interfaces available to VPP, they use virtio drivers. When assigned to VPP they use uio_pci_generic. I'm not sure if any other stats might be useful so I'm just pasting a bunch of stats & information from the client & server instances below, I know it's a lot, just putting it here in case there is something useful in there. Thanks again for taking the time to follow-up with me and for the suggestions, I really do appreciate it very much! Regards, Dom *#############################################################################* *# Interface uses virtio-pci when the iperf3 test is run using regular Linux* *# networking.* *#############################################################################* *[root@vpp-test-1 centos]# dpdk-devbind --status* Network devices using kernel driver =================================== 0000:00:03.0 'Virtio network device 1000' if=eth0 drv=virtio-pci unused=virtio_pci *Active* 0000:00:04.0 'Virtio network device 1000' if=eth1 drv=virtio-pci unused=virtio_pci *Active* *#############################################################################* *# Interface uses uio_pci_generic when set up for VPP* *#############################################################################* *[root@vpp-test-1 centos]# dpdk-devbind --status* Network devices using DPDK-compatible driver ============================================ 0000:00:03.0 'Virtio network device 1000' drv=uio_pci_generic unused=virtio_pci Network devices using kernel driver =================================== 0000:00:04.0 'Virtio network device 1000' if=eth1 drv=virtio-pci unused=virtio_pci,uio_pci_generic *Active* *vpp# sh hardware-interfaces* Name Idx Link Hardware GigabitEthernet0/3/0 1 up GigabitEthernet0/3/0 Link speed: 10 Gbps Ethernet address fa:16:3e:10:5e:4b Red Hat Virtio carrier up full duplex mtu 9206 flags: admin-up pmd maybe-multiseg rx: queues 1 (max 1), desc 256 (min 0 max 65535 align 1) tx: queues 1 (max 1), desc 256 (min 0 max 65535 align 1) pci: device 1af4:1000 subsystem 1af4:0001 address 0000:00:03.00 numa 0 max rx packet len: 9728 promiscuous: unicast off all-multicast on vlan offload: strip off filter off qinq off rx offload avail: vlan-strip udp-cksum tcp-cksum tcp-lro vlan-filter jumbo-frame rx offload active: jumbo-frame tx offload avail: vlan-insert udp-cksum tcp-cksum tcp-tso multi-segs tx offload active: multi-segs rss avail: none rss active: none tx burst function: virtio_xmit_pkts rx burst function: virtio_recv_mergeable_pkts rx frames ok 467 rx bytes ok 27992 extended stats: rx good packets 467 rx good bytes 27992 rx q0packets 467 rx q0bytes 27992 rx q0 good packets 467 rx q0 good bytes 27992 rx q0 multicast packets 465 rx q0 broadcast packets 2 rx q0 undersize packets 467 *#############################################################################* *# Dropped packets are LLC BPDUs, not sure but probably a linuxbridge thing* *#############################################################################* *vpp# show trace* ------------------- Start of thread 0 vpp_main ------------------- No packets in trace buffer ------------------- Start of thread 1 vpp_wk_0 ------------------- Packet 1 00:08:35:202159: dpdk-input GigabitEthernet0/3/0 rx queue 0 buffer 0xfee2f4: current data 0, length 60, buffer-pool 0, ref-count 1, totlen-nifb 0, trace handle 0x1000000 ext-hdr-valid l4-cksum-computed l4-cksum-correct PKT MBUF: port 0, nb_segs 1, pkt_len 60 buf_len 4224, data_len 60, ol_flags 0x0, data_off 128, phys_addr 0xffb8bd80 packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 rss 0x0 fdir.hi 0x0 fdir.lo 0x0 0x0027: f8:b1:56:7d:61:58 -> 01:80:c2:00:00:00 00:08:35:202184: ethernet-input frame: flags 0x1, hw-if-index 1, sw-if-index 1 0x0027: f8:b1:56:7d:61:58 -> 01:80:c2:00:00:00 00:08:35:202190: llc-input LLC bpdu -> bpdu 00:08:35:202205: error-drop rx:GigabitEthernet0/3/0 00:08:35:202207: drop llc-input: unknown llc ssap/dsap *#############################################################################* *# Stats before test* *#############################################################################* *###### SERVER* *vpp# sh int* Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count GigabitEthernet0/3/0 1 up 9000/0/0/0 rx packets 12 rx bytes 720 drops 12 *vpp# show session verbose 2* Thread 0: no sessions Thread 1: no sessions Thread 2: no sessions Thread 3: no sessions Thread 4: no sessions Thread 5: no sessions *### CLIENT* *vpp# sh int* Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count GigabitEthernet0/3/0 1 up 9000/0/0/0 rx packets 18 rx bytes 1080 drops 18 local0 0 down 0/0/0/0 vpp# show session verbose 2 Thread 0: no sessions Thread 1: no sessions Thread 2: no sessions Thread 3: no sessions Thread 4: no sessions Thread 5: no sessions *#############################################################################* *# Stats AFTER test* *#############################################################################* *###### SERVER* *vpp# sh int* Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count GigabitEthernet0/3/0 1 up 9000/0/0/0 rx packets 5960412 rx bytes 8896698636 tx packets 3923484 tx bytes 258949928 drops 77 ip4 5960335 local0 0 down 0/0/0/0 * vpp# show session verbose 2* [0:0][T] 10.0.0.153:5555->0.0.0.0:0 LISTEN Thread 0: active sessions 1 [1:0][T] 10.0.0.153:5555->10.0.0.151:61592 ESTABLISHED index: 0 flags: timers: snd_una 1 snd_nxt 1 snd_una_max 1 rcv_nxt 4249694047 rcv_las 4249694047 snd_wnd 64512 rcv_wnd 63488 rcv_wscale 10 snd_wl1 4249692619 snd_wl2 1 flight size 0 out space 4290 rcv_wnd_av 63488 tsval_recent 87602812 tsecr 87652914 tsecr_last_ack 87652914 tsval_recent_age 4294967074 snd_mss 1428 rto 200 rto_boff 0 srtt 1 us 1.196 rttvar 0 rtt_ts 0.0000 rtt_seq 1135896793 cong: none algo cubic cwnd 4290 ssthresh 4194304 bytes_acked 0 cc space 4290 prev_cwnd 0 prev_ssthresh 0 rtx_bytes 0 snd_congestion 1135896793 dupack 0 limited_transmit 1135896793 sboard: sacked_bytes 0 last_sacked_bytes 0 lost_bytes 0 last_bytes_delivered 0 high_sacked 1135896793 snd_una_adv 0 cur_rxt_hole 4294967295 high_rxt 1135896793 rescue_rxt 1135896793 stats: in segs 5989307 dsegs 5989306 bytes 8544661342 dupacks 0 out segs 3942513 dsegs 0 bytes 0 dupacks 0 fr 0 tr 0 rxt segs 0 bytes 0 duration 106.489 err wnd data below 0 above 0 ack below 0 above 0 pacer: bucket 13532532 tokens/period .061 last_update d88e7922 Rx fifo: cursize 0 nitems 65534 has_event 0 head 11437 tail 11437 segment manager 2 vpp session 0 thread 1 app session 199999 thread 0 ooo pool 0 active elts newest 4294967295 Tx fifo: cursize 0 nitems 65534 has_event 0 head 0 tail 0 segment manager 2 vpp session 0 thread 1 app session 199999 thread 0 ooo pool 0 active elts newest 0 Thread 1: active sessions 1 Thread 2: no sessions Thread 3: no sessions Thread 4: no sessions Thread 5: no sessions *###### CLIENT* *vpp# sh int* Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count GigabitEthernet0/3/0 1 up 9000/0/0/0 rx packets 4195035 rx bytes 276871796 tx packets 6385403 tx bytes 9531349660 drops 84 ip4 4194951 local0 0 down 0/0/0/0 drops 1 *vpp# show session verbose 2* Thread 0: no sessions [1:0][T] 10.0.0.151:61592->10.0.0.153:5555 ESTABLISHED index: 0 flags: timers: RETRANSMIT snd_una 551376469 snd_nxt 551439301 snd_una_max 551439301 rcv_nxt 1 rcv_las 1 snd_wnd 63488 rcv_wnd 64512 rcv_wscale 10 snd_wl1 1 snd_wl2 551376469 flight size 62832 out space 656 rcv_wnd_av 64512 tsval_recent 87660537 tsecr 87610434 tsecr_last_ack 87610434 tsval_recent_age 4294967183 snd_mss 1428 rto 200 rto_boff 0 srtt 1 us .708 rttvar 1 rtt_ts 167.5240 rtt_seq 551393605 cong: none algo newreno cwnd 4195240 ssthresh 4194304 bytes_acked 1428 cc space 656 prev_cwnd 0 prev_ssthresh 0 rtx_bytes 0 snd_congestion 3019902672 dupack 0 limited_transmit 3019897507 sboard: sacked_bytes 0 last_sacked_bytes 0 lost_bytes 0 last_bytes_delivered 0 high_sacked 3019897507 snd_una_adv 0 cur_rxt_hole 4294967295 high_rxt 3019897507 rescue_rxt 3019897507 stats: in segs 4207793 dsegs 0 bytes 0 dupacks 0 out segs 6407444 dsegs 6407443 bytes 9141373892 dupacks 0 fr 0 tr 0 rxt segs 0 bytes 0 duration 114.113 err wnd data below 0 above 0 ack below 0 above 0 pacer: bucket 3321031074 tokens/period 2756.792 last_update d31de7db Rx fifo: cursize 0 nitems 65534 has_event 0 head 0 tail 0 segment manager 2 vpp session 0 thread 1 app session 0 thread 0 ooo pool 0 active elts newest 0 Tx fifo: cursize 65534 nitems 65534 has_event 1 head 30515 tail 30514 segment manager 2 vpp session 0 thread 1 app session 0 thread 0 ooo pool 0 active elts newest 4294967295 Thread 1: active sessions 1 Thread 2: no sessions Thread 3: no sessions Thread 4: no sessions Thread 5: no sessions
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14794): https://lists.fd.io/g/vpp-dev/message/14794 Mute This Topic: https://lists.fd.io/mt/65863639/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
