On Tue, Sep 16, 2025 at 03:25:21PM +0200, Rafael Sadowski wrote:
> On Tue Sep 16, 2025 at 03:18:28PM +0200, Jan Klemkow wrote:
> > On Tue, Sep 16, 2025 at 03:01:33PM +0200, Rafael Sadowski wrote:
> > > WireGuard shows severe performance degradation (95% bandwidth
> > > loss) on Intel 10Gb interfaces compared to direct connections,
> > > with significant packet loss patterns.
> > >
> > > Performance Comparison:
> > >
> > > ServerA (Chicago) - Intel 10Gb interface (ix0)
> > > ServerB (Atlanta) - Intel 10Gb interface (ix3)
> > >
> > > - Direct connection (iperf): 66.8 Mbps
> > > - WireGuard tunnel (iperf): 3.3 Mbps
> > > - Performance loss: 95%
> > >
> > > The physical Intel interface (ix3) shows 149426 output failures:
> > >
> > > ix3 1500 <Link> f8:f2:1e:3c:9c:09 195418012 0 144748154
> > > 149426 0
> > >
> > > suggesting hardware/driver level problems that worsen with
> > > WireGuard traffic processing?
> > >
> > > Are there known compatibility issues between ix(4) driver and
> > > WireGuard packet processing?
> > >
> > > Could the bridge configuration (veb0 + vport0) be contributing to
> > > the packet loss patterns?
> >
> > Yes. You will lose that kind of performance over bridge(4) and veb(4)
> > because, they don't use segmentation offloading nor parallel processing
> > of packets.
> >
> > The wg(4) device may have a similar missing performance features.
> >
> > > Any guidance on debugging approaches or known workarounds would be
> > > greatly appreciated. I'm happy to provide additional data.
> >
> > Could you provide a netstat -s stats, before and after you've done you
> > measurement? So, we can see if there are also any error or drop counter
> > involved.
>
> Before iperf:
> ...
>
> After iperf via wg0
> ...
All counter are looking fine.
Nothing special for me.
--- before.log Tue Sep 16 15:39:17 2025
+++ after.log Tue Sep 16 15:39:31 2025
@@ -1,5 +1,5 @@
ip:
- 261476447 total packets received
+ 261480916 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
@@ -12,12 +12,12 @@
0 malformed fragments dropped
0 fragments dropped after timeout
0 packets reassembled ok
- 257441439 packets for this host
+ 257445892 packets for this host
2298 packets for unknown/unsupported protocol
789407 packets forwarded
0 packets not forwardable
0 redirects sent
- 186666566 packets sent from this host
+ 186670884 packets sent from this host
192 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
@@ -28,20 +28,20 @@
0 packets with ip length > max ip packet size
0 tunneling packets that can't find gif
0 datagrams with bad address in header
- 108844283 input datagrams software-checksummed
- 176706792 output datagrams software-checksummed
+ 108845951 input datagrams software-checksummed
+ 176712264 output datagrams software-checksummed
0 multicast packets which we don't join
- 270340630 route cache hit
- 130030816 route cache miss
+ 270347510 route cache hit
+ 130032853 route cache miss
0 packets received on wrong interface
0 input packets dropped due to no bufs, etc.
icmp:
- 8420 calls to icmp_error
+ 8425 calls to icmp_error
0 errors not generated because old message was icmp
0 errors not generated because of rate limitation
Output packet histogram:
echo reply: 72
- destination unreachable: 8420
+ destination unreachable: 8425
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
@@ -76,62 +76,62 @@
0 protocol family mismatches
0 attempts to use tunnel with unspecified endpoint(s)
tcp:
- 94948757 packets sent
- 28710544 data packets (16624602340 bytes)
- 102472 data packets (42596644 bytes) retransmitted
+ 94950448 packets sent
+ 28711966 data packets (16627717749 bytes)
+ 102473 data packets (42596712 bytes) retransmitted
83 fast retransmitted packets
- 50664046 ack-only packets (65368737 delayed)
+ 50664244 ack-only packets (65369286 delayed)
0 URG only packets
0 window probe packets
- 13931200 window update packets
- 1781795 control packets
- 99184392 packets software-checksummed
- 740063 output TSO packets software chopped
+ 13931256 window update packets
+ 1781810 control packets
+ 99187484 packets software-checksummed
+ 740745 output TSO packets software chopped
230 output TSO packets hardware processed
- 3606746 output TSO packets generated
+ 3608829 output TSO packets generated
0 output TSO packets dropped
- 135967341 packets received
- 30001001 acks (for 16562209986 bytes)
- 3055906 duplicate acks
+ 135969868 packets received
+ 30003080 acks (for 16565325460 bytes)
+ 3055950 duplicate acks
0 acks for unsent data
0 acks for old data
- 108194698 packets (102313362560 bytes) received in-sequence
+ 108195340 packets (102313602925 bytes) received in-sequence
864795 completely duplicate packets (1115421580 bytes)
3374 old duplicate packets
7558 packets with some duplicate data (4908373 bytes duplicated)
- 3209405 out-of-order packets (1208918804 bytes)
+ 3209452 out-of-order packets (1208918804 bytes)
7 packets (0 bytes) of data after window
0 window probes
- 98237 window update packets
+ 98254 window update packets
19064 packets received after close
27194 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
0 discarded for missing IPsec protection
0 discarded due to memory shortage
- 97670476 packets software-checksummed
+ 97672032 packets software-checksummed
0 bad/missing md5 checksums
0 good md5 checksums
0 input LRO packets passed through pseudo device
0 input LRO generated packets from hardware
0 input LRO coalesced packets by network device
0 input bad LRO packets dropped
- 508590 connection requests
- 2822613 connection accepts
- 3330550 connections established (including accepts)
- 3337566 connections closed (including 117159 drops)
+ 508596 connection requests
+ 2822664 connection accepts
+ 3330607 connections established (including accepts)
+ 3337621 connections closed (including 117160 drops)
0 connections drained
9 embryonic connections dropped
- 30493863 segments updated rtt (of 21788757 attempts)
- 336765 retransmit timeouts
+ 30495948 segments updated rtt (of 21789989 attempts)
+ 336766 retransmit timeouts
25453 connections dropped by rexmit timeout
0 persist timeouts
9352 keepalive timeouts
2969 keepalive probes sent
2 connections dropped by keepalive
- 595239 correct ACK header predictions
- 96201454 correct data packet header predictions
- 6274586 PCB cache misses
+ 595277 correct ACK header predictions
+ 96201746 correct data packet header predictions
+ 6274689 PCB cache misses
15555 dropped due to no socket
0 ECN connections accepted
0 ECE packets received
@@ -141,13 +141,13 @@
0 ECE packets sent
0 CWR packets sent
cwr by fastrecovery: 8493
- cwr by timeout: 336765
+ cwr by timeout: 336766
cwr by ecn: 0
23571 bad connection attempts
0 SYN packets dropped due to queue or memory full
- 3191317 SYN cache entries added
+ 3191368 SYN cache entries added
0 hash collisions
- 2822613 completed
+ 2822664 completed
0 aborted (no space to build PCB)
329303 timed out
0 dropped due to overflow
@@ -161,28 +161,28 @@
293 hash bucket array size in current SYN cache
0 entries in current SYN cache, limit is 10255
0 longest bucket length in current SYN cache, limit is 105
- 8683 uses of current SYN cache left
+ 8632 uses of current SYN cache left
8410 SACK recovery episodes
11383 segment rexmits in SACK recovery episodes
15770037 byte rexmits in SACK recovery episodes
- 113995 SACK options received
+ 113999 SACK options received
834022 SACK options sent
83 SACK options dropped
udp:
- 121473937 datagrams received
+ 121475863 datagrams received
0 with incomplete header
0 with bad data length field
0 with bad checksum
0 with no checksum
- 9468707 input packets software-checksummed
- 76724393 output packets software-checksummed
- 8420 dropped due to no socket
+ 9468811 input packets software-checksummed
+ 76726768 output packets software-checksummed
+ 8425 dropped due to no socket
0 broadcast/multicast datagrams dropped due to no socket
0 dropped due to missing IPsec protection
0 dropped due to full socket buffers
- 121465517 delivered
- 90072470 datagrams output
- 114216491 missed PCB cache
+ 121467438 delivered
+ 90075091 datagrams output
+ 114218296 missed PCB cache
ipsec:
0 input IPsec packets
0 output IPsec packets
@@ -399,3 +399,4 @@
0 messages dropped due to full socket buffers
0 delivered
0 datagrams output
