On 18.10.2023. 15:35, Mischa wrote: > Hi All, > > Just upgraded a couple of machines to 7.4. smooth as always!! > > I am however seeing issues with IPv4, slowness or no throughput at all. > The machines I have upgraded are using an Intel X540-T network card and > is connected on 10G. > > ix0 at pci5 dev 0 function 0 "Intel X540T" rev 0x01, msix, 16 queues, > address b8:ca:3a:62:ee:40 > ix1 at pci5 dev 0 function 1 "Intel X540T" rev 0x01, msix, 16 queues, > address b8:ca:3a:62:ee:42 > > root@n2:~ # sysctl kern.version > kern.version=OpenBSD 7.4 (GENERIC.MP) #1397: Tue Oct 10 09:02:37 MDT 2023 > dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP > > There are a bunch of VMs running on top of it, as soon as I want to > fetch something with ftp, for example, I don't get anything over IPv4, > with IPv6 everything is normal. > > mischa@www2:~ $ ftp -4 > https://mirror.openbsd.amsterdam/pub/OpenBSD/7.4/amd64/install74.iso > Trying 46.23.88.18... > Requesting > https://mirror.openbsd.amsterdam/pub/OpenBSD/7.4/amd64/install74.iso > 0% | | 512 KB - stalled > -^C > > A trace on mirror / n2: > > n2:~ # tcpdump -i vport880 host 46.23.88.32 > tcpdump: listening on vport880, link-type EN10MB > 15:16:08.730274 www2.high5.nl.1828 > n2.high5.nl.https: S > 2182224746:2182224746(0) win 16384 <mss 1460,nop,nop,sackOK,nop,wscale > 6,nop,nop,timestamp 2899683458 0> (DF) > 15:16:08.730297 arp who-has www2.high5.nl tell n2.high5.nl > 15:16:08.731535 arp reply www2.high5.nl is-at fe:51:bb:1e:12:11 > 15:16:08.731540 n2.high5.nl.https > www2.high5.nl.1828: S > 633749938:633749938(0) ack 2182224747 win 16384 <mss > 1460,nop,nop,sackOK,nop,wscale 6,nop,nop,timestamp 3129955106 > 2899683458> (DF) > 15:16:08.732017 www2.high5.nl.1828 > n2.high5.nl.https: . ack 1 win 256 > <nop,nop,timestamp 2899683458 3129955106> (DF) > 15:16:08.785752 www2.high5.nl.1828 > n2.high5.nl.https: P 1:312(311) ack > 1 win 256 <nop,nop,timestamp 2899683510 3129955106> (DF) > 15:16:08.786092 n2.high5.nl.https > www2.high5.nl.1828: P 1:128(127) ack > 312 win 271 <nop,nop,timestamp 3129955166 2899683510> (DF) > 15:16:08.786376 n2.high5.nl.https > www2.high5.nl.1828: P 128:134(6) ack > 312 win 271 <nop,nop,timestamp 3129955166 2899683510> (DF) > 15:16:08.786396 n2.high5.nl.https > www2.high5.nl.1828: P 134:166(32) > ack 312 win 271 <nop,nop,timestamp 3129955166 2899683510> (DF) > 15:16:08.786455 n2.high5.nl.https > www2.high5.nl.1828: . 166:1614(1448) > ack 312 win 271 <nop,nop,timestamp 3129955166 2899683510> (DF) > 15:16:08.786457 n2.high5.nl.https > www2.high5.nl.1828: . > 1614:3062(1448) ack 312 win 271 <nop,nop,timestamp 3129955166 > 2899683510> (DF) > 15:16:08.786460 n2.high5.nl.https > www2.high5.nl.1828: P 3062:3803(741) > ack 312 win 271 <nop,nop,timestamp 3129955166 2899683510> (DF) > 15:16:08.786943 www2.high5.nl.1828 > n2.high5.nl.https: . ack 134 win > 255 <nop,nop,timestamp 2899683510 3129955166> (DF) > 15:16:08.796534 n2.high5.nl.https > www2.high5.nl.1828: P 3803:4345(542) > ack 312 win 271 <nop,nop,timestamp 3129955176 2899683510> (DF) > 15:16:08.796577 n2.high5.nl.https > www2.high5.nl.1828: P 4345:4403(58) > ack 312 win 271 <nop,nop,timestamp 3129955176 2899683510> (DF) > 15:16:08.797518 www2.high5.nl.1828 > n2.high5.nl.https: . ack 166 win > 256 <nop,nop,timestamp 2899683526 3129955166,nop,nop,sack 1 {3803:4345} >> (DF) > 15:16:08.797522 www2.high5.nl.1828 > n2.high5.nl.https: . ack 166 win > 256 <nop,nop,timestamp 2899683526 3129955166,nop,nop,sack 1 {3803:4403} >> (DF) > 15:16:09.790297 n2.high5.nl.https > www2.high5.nl.1828: . 166:1614(1448) > ack 312 win 271 <nop,nop,timestamp 3129956176 2899683526> (DF) > 15:16:09.790902 www2.high5.nl.1828 > n2.high5.nl.https: . ack 1614 win > 233 <nop,nop,timestamp 2899684519 3129956176,nop,nop,sack 1 {3803:4403} >> (DF) > 15:16:09.790917 n2.high5.nl.https > www2.high5.nl.1828: . > 1614:3062(1448) ack 312 win 271 <nop,nop,timestamp 3129956176 > 2899684519> (DF) > 15:16:09.790923 n2.high5.nl.https > www2.high5.nl.1828: P > 3062:4403(1341) ack 312 win 271 <nop,nop,timestamp 3129956176 > 2899684519> (DF) > 15:16:10.790299 n2.high5.nl.https > www2.high5.nl.1828: . > 1614:3062(1448) ack 312 win 271 <nop,nop,timestamp 3129957176 > 2899684519> (DF) > 15:16:10.791204 www2.high5.nl.1828 > n2.high5.nl.https: . ack 3062 win > 233 <nop,nop,timestamp 2899685520 3129957176,nop,nop,sack 1 {3803:4403} >> (DF) > 15:16:10.791223 n2.high5.nl.https > www2.high5.nl.1828: P > 3062:4403(1341) ack 312 win 271 <nop,nop,timestamp 3129957176 > 2899685520> (DF) > 15:16:10.791692 www2.high5.nl.1828 > n2.high5.nl.https: . ack 4403 win > 235 <nop,nop,timestamp 2899685520 3129957176> (DF) > 15:16:10.802647 www2.high5.nl.1828 > n2.high5.nl.https: P 312:318(6) ack > 4403 win 256 <nop,nop,timestamp 2899685528 3129957176> (DF) > 15:16:11.000297 n2.high5.nl.https > www2.high5.nl.1828: . ack 318 win > 271 <nop,nop,timestamp 3129957386 2899685528> (DF) > 15:16:11.001162 www2.high5.nl.1828 > n2.high5.nl.https: P 318:527(209) > ack 4403 win 256 <nop,nop,timestamp 2899685730 3129957386> (DF) > 15:16:11.001860 n2.high5.nl.https > www2.high5.nl.1828: P 4403:5059(656) > ack 527 win 271 <nop,nop,timestamp 3129957386 2899685730> (DF) > 15:16:11.001989 n2.high5.nl.https > www2.high5.nl.1828: . > 5059:6507(1448) ack 527 win 271 <nop,nop,timestamp 3129957386 > 2899685730> (DF) > 15:16:11.001992 n2.high5.nl.https > www2.high5.nl.1828: . > 6507:7955(1448) ack 527 win 271 <nop,nop,timestamp 3129957386 > 2899685730> (DF) > 15:16:11.195431 www2.high5.nl.1828 > n2.high5.nl.https: . ack 5059 win > 256 <nop,nop,timestamp 2899685924 3129957386> (DF) > 15:16:11.195447 n2.high5.nl.https > www2.high5.nl.1828: . > 7955:9403(1448) ack 527 win 271 <nop,nop,timestamp 3129957576 > 2899685924> (DF) > > > Running a trace on www2 I am seeing: > > www2:~ # tcpdump -i vio0 host 46.23.88.18 > tcpdump: listening on vio0, link-type EN10MB > 15:16:08.729974 www2.high5.nl.1828 > n2.high5.nl.https: S > 2182224746:2182224746(0) win 16384 <mss 1460,nop,nop,sackOK,nop,wscale > 6,nop,nop,timestamp 2899683458 0> (DF) > 15:16:08.731114 arp who-has www2.high5.nl tell n2.high5.nl > 15:16:08.731229 arp reply www2.high5.nl is-at fe:51:bb:1e:12:11 > 15:16:08.731631 n2.high5.nl.https > www2.high5.nl.1828: S > 633749938:633749938(0) ack 2182224747 win 16384 <mss > 1460,nop,nop,sackOK,nop,wscale 6,nop,nop,timestamp 3129955106 > 2899683458> (DF) > 15:16:08.731732 www2.high5.nl.1828 > n2.high5.nl.https: . ack 1 win 256 > <nop,nop,timestamp 2899683458 3129955106> (DF) > 15:16:08.785371 www2.high5.nl.1828 > n2.high5.nl.https: P 1:312(311) ack > 1 win 256 <nop,nop,timestamp 2899683510 3129955106> (DF) > 15:16:08.786219 n2.high5.nl.https > www2.high5.nl.1828: P 1:128(127) ack > 312 win 271 <nop,nop,timestamp 3129955166 2899683510> (DF) > 15:16:08.786522 n2.high5.nl.https > www2.high5.nl.1828: P 128:134(6) ack > 312 win 271 <nop,nop,timestamp 3129955166 2899683510> (DF) > 15:16:08.786523 n2.high5.nl.https > www2.high5.nl.1828: P 134:166(32) > ack 312 win 271 <nop,nop,timestamp 3129955166 2899683510> (DF) > 15:16:08.786525 truncated-ip - 1665 bytes missing!n2.high5.nl.https > > www2.high5.nl.1828: P 166:2138(1972) ack 312 win 271 <nop,nop,timestamp > 3129955166 2899683510> (DF) > 15:16:08.786628 www2.high5.nl.1828 > n2.high5.nl.https: . ack 134 win > 255 <nop,nop,timestamp 2899683510 3129955166> (DF) > 15:16:08.797085 n2.high5.nl.https > www2.high5.nl.1828: P 3803:4345(542) > ack 312 win 271 <nop,nop,timestamp 3129955176 2899683510> (DF) > 15:16:08.797086 n2.high5.nl.https > www2.high5.nl.1828: P 4345:4403(58) > ack 312 win 271 <nop,nop,timestamp 3129955176 2899683510> (DF) > 15:16:08.797207 www2.high5.nl.1828 > n2.high5.nl.https: . ack 166 win > 256 <nop,nop,timestamp 2899683526 3129955166,nop,nop,sack 1 {3803:4345} >> (DF) > 15:16:08.797208 www2.high5.nl.1828 > n2.high5.nl.https: . ack 166 win > 256 <nop,nop,timestamp 2899683526 3129955166,nop,nop,sack 1 {3803:4403} >> (DF) > 15:16:09.790498 n2.high5.nl.https > www2.high5.nl.1828: . 166:1614(1448) > ack 312 win 271 <nop,nop,timestamp 3129956176 2899683526> (DF) > 15:16:09.790611 www2.high5.nl.1828 > n2.high5.nl.https: . ack 1614 win > 233 <nop,nop,timestamp 2899684519 3129956176,nop,nop,sack 1 {3803:4403} >> (DF) > 15:16:09.791036 truncated-ip - 817 bytes missing!n2.high5.nl.https > > www2.high5.nl.1828: P 1614:3586(1972) ack 312 win 271 <nop,nop,timestamp > 3129956176 2899684519> (DF) > 15:16:10.790743 n2.high5.nl.https > www2.high5.nl.1828: . > 1614:3062(1448) ack 312 win 271 <nop,nop,timestamp 3129957176 > 2899684519> (DF) > 15:16:10.790872 www2.high5.nl.1828 > n2.high5.nl.https: . ack 3062 win > 233 <nop,nop,timestamp 2899685520 3129957176,nop,nop,sack 1 {3803:4403} >> (DF) > 15:16:10.791287 n2.high5.nl.https > www2.high5.nl.1828: P > 3062:4403(1341) ack 312 win 271 <nop,nop,timestamp 3129957176 > 2899685520> (DF) > 15:16:10.791393 www2.high5.nl.1828 > n2.high5.nl.https: . ack 4403 win > 235 <nop,nop,timestamp 2899685520 3129957176> (DF) > 15:16:10.802259 www2.high5.nl.1828 > n2.high5.nl.https: P 312:318(6) ack > 4403 win 256 <nop,nop,timestamp 2899685528 3129957176> (DF) > 15:16:11.000715 n2.high5.nl.https > www2.high5.nl.1828: . ack 318 win > 271 <nop,nop,timestamp 3129957386 2899685528> (DF) > 15:16:11.000830 www2.high5.nl.1828 > n2.high5.nl.https: P 318:527(209) > ack 4403 win 256 <nop,nop,timestamp 2899685730 3129957386> (DF) > 15:16:11.002040 n2.high5.nl.https > www2.high5.nl.1828: P 4403:5059(656) > ack 527 win 271 <nop,nop,timestamp 3129957386 2899685730> (DF) > 15:16:11.002042 truncated-ip - 924 bytes missing!n2.high5.nl.https > > www2.high5.nl.1828: . 5059:7031(1972) ack 527 win 271 <nop,nop,timestamp > 3129957386 2899685730> (DF) > 15:16:11.195146 www2.high5.nl.1828 > n2.high5.nl.https: . ack 5059 win > 256 <nop,nop,timestamp 2899685924 3129957386> (DF) > 15:16:11.195567 n2.high5.nl.https > www2.high5.nl.1828: . > 7955:9403(1448) ack 527 win 271 <nop,nop,timestamp 3129957576 > 2899685924> (DF) > > > From "What is new" I noticed: > * Enable LRO per default in network drivers. LRO allows to receive > aggregated packets larger than the MTU. Receiving TCP streams becomes > much faster. Currently only ix(4) and lo(4) devices support LRO, and > ix(4) is limited to IPv4 and hardware newer than the old 82598 model. > > Could this have something to do with it? > > Mischa >
is it better when you disable ifconfig ix0 -tcplro ifconfig ix1 -tcplro and maybe you would need net.inet.tcp.tso=1
