Public bug reported:
Package: network-manager-config-connectivity-ubuntu
NetworkManager: 1.54.3-2ubuntu3
Ubuntu: 26.04 LTS
WHAT HAPPENS
NetworkManager's connectivity check always sends "Connection: close".
connectivity-check.ubuntu.com answers that with an abortive close: the 204
response, then FIN, then RST, sent back to back.
On a network path that reorders packets, that RST can arrive ahead of the
response body. An RST is not data - it holds no sequence space and never
enters the reassembly queue - so when the kernel accepts it, it tears down
the connection and discards whatever is buffered, including a complete and
valid 204 that had already arrived. libcurl then reports
(56) Recv failure: Connection reset by peer
and NetworkManager scores the interface as having limited connectivity.
The server is not doing anything invalid, and the response itself is
perfectly well-formed. But the abortive close leaves no margin: any
reordering between the body and the RST turns a successful request into a
failed one. Every other connectivity endpoint I tested closes gracefully
with FIN alone and is unaffected by the same reordering.
WHY IT MATTERS
A "limited" verdict adds +20000 to that device's default-route metric. On a
multi-homed host whose interface metrics differ by less than that - the
default wired 100 / WiFi 600 pairing, for instance - each failed probe moves
the default route to the backup interface and the next successful probe
moves it back. On my host this produced roughly 830 IPv4 and 740 IPv6
default-route changes per day, with no actual connectivity fault anywhere.
Changing only the probe endpoint took it to zero.
The reordering on my path comes from powerline ethernet adapters, which is
my environment's problem and not Ubuntu's. The point of this report is
narrower: the endpoint's abortive close is what converts a tolerable path
condition into a failed connectivity check, and it is the only endpoint
tested that does so.
HOW TO REPRODUCE
From a host on a path that reorders (or any lossy path), compare the same
request with and without the header NetworkManager sends:
# NM's actual request shape - fails
for i in $(seq 60); do
curl -4 -sS -m 5 -H 'Connection: close' -o /dev/null \
http://connectivity-check.ubuntu.com/ || echo FAIL
done
# identical but keepalive - never fails
for i in $(seq 60); do
curl -4 -sS -m 5 -o /dev/null \
http://connectivity-check.ubuntu.com/ || echo FAIL
done
Results on my host, wired interface:
with Connection: close 40 ok, 20 failed (all curl 56)
default keepalive 60 ok, 0 failed
Note that plain curl does NOT reproduce this by default, because curl uses
keepalive and NetworkManager does not. The explicit header is required.
ENDPOINT COMPARISON
40 requests each, same host, same wired interface, all carrying
"Connection: close":
connectivity-check.ubuntu.com 14 failed
nmcheck.gnome.org 0 failed
connectivitycheck.gstatic.com 0 failed
network-test.debian.org 0 failed
Same request over two different paths from the same host, run back to back,
counting both TCP streams delivered with FIN/RST ahead of the body and
outright failures:
wired (via powerline) 36 of 40 streams reordered, 19 failed
WiFi (direct to AP) 1 of 40 streams reordered, 1 failed
PACKET CAPTURE
A failing exchange. Note the three server packets share a capture timestamp
and their IP IDs are 0x7b (body), 0x7c (FIN), 0x7d (RST) - so the server
sent body-first, and they were delivered body-last:
10.0.0.63.49660 > 91.189.91.97.80: Flags [P.], seq 1:88, length 87
GET / HTTP/1.1
Host: connectivity-check.ubuntu.com
Accept: */*
Connection: close
91.189.91.97.80 > 10.0.0.63.49660: Flags [F.], seq 80, ack 88 <- IP ID 0x7c
91.189.91.97.80 > 10.0.0.63.49660: Flags [R.], seq 81, ack 88 <- IP ID 0x7d
91.189.91.97.80 > 10.0.0.63.49660: Flags [P.], seq 1:80, length 79 <- IP ID
0x7b
HTTP/1.1 204 No Content
x-networkmanager-status: online
connection: close
10.0.0.63.49660 > 91.189.91.97.80: Flags [.], ack 1, sack 1 {80:81}
91.189.91.97.80 > 10.0.0.63.49660: Flags [R], seq 701268504, win 0
The client's ack 1 with sack {80:81} shows the stack had not taken the
payload - it acknowledges nothing and selectively acknowledges only the FIN.
The connection then ends reset, with a valid 204 discarded.
EXPECTED
connectivity-check.ubuntu.com closes gracefully after its response - FIN,
without the trailing RST - as nmcheck.gnome.org, connectivitycheck.gstatic.com
and network-test.debian.org all do. That removes the race entirely and costs
nothing, since NetworkManager has already asked for the connection to close.
WORKAROUND
Override the endpoint in /etc/NetworkManager/conf.d/, at a name sorting after
20-connectivity-ubuntu.conf. nmcheck.gnome.org carries no
X-NetworkManager-Status header, so the verdict has to come from the body:
[connectivity]
uri=http://nmcheck.gnome.org/check_network_status.txt
response=NetworkManager is online
** Affects: network-manager (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2163449
Title:
connectivity-check.ubuntu.com sends RST immediately after FIN, so its
204 is discarded on any path that reorders packets
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/2163449/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
