casaroli opened a new pull request, #22025:
URL: https://github.com/apache/nuttx/pull/22025

   ## Summary
   
   This PR fixes the end of TCP connections in `net/tcp`: `shutdown()`, 
`close()` and the connections that stay after `close()`. Each commit fixes one 
bug. All are in common network code: the same in flat, protected and kernel 
builds, on every architecture.
   
   1. **`net/tcp: Send the FIN of shutdown(SHUT_WR) after the queued data.`** 
`tcp_shutdown()` sent the FIN at the next poll and disabled the send callback, 
also while data still waited in the write queue (for example behind a zero 
window). The peer read end-of-file too early and lost the data, or got it after 
the FIN. `tcp_appsend()` sets `tx_unacked` to 1 for the FIN, so the two ends 
then disagree on the sequence numbers: over the loopback device the two closing 
connections sent ACKs to each other for ever. Now the shutdown callback waits, 
as `close()` does, until the write queue is empty and all data is acknowledged; 
it also runs on `TCP_ACKDATA` and frees itself on a lost connection. It no 
longer sets `d_len` to 0 (an ACK can carry data for the reader), and a second 
`shutdown()` uses the callback that still waits. A `close()` after 
`shutdown(SHUT_WR)` has its close callback after the shutdown callback: it 
ended its wait at any `TCP_TXCLOSE`, also the one that the shutdown callbac
 k sets to send its FIN, and so no one freed the connection after TIME_WAIT. 
Now the close callback ends its wait at `TCP_TXCLOSE` only in `TCP_CLOSED` (the 
ACK of our FIN in LAST_ACK, the one place where `tcp_input()` gives this event).
   2. **`net/tcp: Reset a closed connection with unread data at once.`** RFC 
1122, 4.2.2.13: a `close()` with unread data sends a RST. 
`tcp_close_eventhandler()` checked for unread data only after its own data was 
sent and acknowledged. Two programs that each wait for the other to read then 
kept both connections, with their read and write buffers, for ever: over 
loopback the IOBs ran out and every later TCP `send()` blocked. Now the check 
comes first.
   3. **`net/tcp: Send a RST, not a FIN, at an abortive close.`** `close()` 
with `SO_LINGER` {1, 0} left the RST to `tcp_timer()` at the next poll. An 
event before that poll (an ACK) went to the close handler, which sent a FIN 
(the peer read end-of-file) or let the send callback send queued data. Now the 
close handler resets at its first event, as for unread data.
   4. **`net/tcp: Look at the state of a closed connection under its locks.`** 
For a reset (commits 2 and 3, and on master for unread data), `tcp_appsend()` 
sets `TCP_CLOSED` after the close handler queued `tcp_close_work()`. On SMP the 
work ran on another CPU, saw the old state and freed nothing; the connection 
stayed in the active list as `TCP_CLOSED` with no reference. Now the work takes 
the device and connection locks, which the poll holds until `tcp_appsend()` 
returns.
   5. **`net/tcp: Free the send callback and the monitor of a closed 
connection.`** A connection in TIME_WAIT kept three devif callbacks for 
`CONFIG_NET_TCP_WAIT_TIMEOUT` (120 s): the close callback, the disabled send 
callback and the monitor. With the default fixed pool of 16 callbacks, four 
such connections and two open sockets used all of them, and the next `close()` 
freed its connection at once with no FIN and lost its queued data. Now the send 
callback is freed when the FIN is queued (close and shutdown callbacks) and 
`close()` stops the monitor first, so the close callback can use its structure.
   6. **`net/tcp: Free a closing connection when close() finds no callback.`** 
When the callback allocation of `close()` still fails, `close()` now frees the 
oldest connection that no socket uses and that only waits for the end of its 
close (`tcp_reclaim()`), and tries once more. This is the fall-back that 
`tcp_alloc()` already had. `tcp_reclaim()` also fixes two faults of that 
fall-back: it held the connection list lock while `tcp_free()` took the device 
lock (the poll takes the device lock, then the list lock: a deadlock with one 
device, from `socket()`), and while `tcp_free()` waited in `work_cancel_sync()` 
for the timer work, which takes the list lock. And it could take a connection 
that `tcp_close_work()` or `close()` was freeing at the same time. Now it takes 
the device lock first (with the lock of another device held, from a SYN, it 
only tries it), takes only a connection that still has its close callback or a 
closed one whose close work it can take from the work queue, remove
 s the callbacks of that connection and marks it closed under the device lock, 
and calls `tcp_free()` with no lock held.
   7. **`net/tcp: Close a closing connection when its device goes down.`** On 
`NETDEV_DOWN` the close callback freed itself and queued `tcp_close_work()`, 
which frees only a connection in `TCP_CLOSED`. A connection in TIME_WAIT (or 
FIN_WAIT_1/2, CLOSING, LAST_ACK) kept its state with no callback, became 
`TCP_CLOSED` when its timer ended and stayed in the active list. Now the close 
callback sets `TCP_CLOSED` on `NETDEV_DOWN`: no socket refers to the connection 
and the device sends nothing more (`tcp_appsend()` does the same for a 
`NETDEV_DOWN` result).
   8. **`Documentation/os/networking: Describe how a closed TCP connection 
ends.`** The TCP state machine page: the FIN after the queued data, the RST 
cases, the close callback, the device that goes down, `tcp_reclaim()`.
   
   ## Impact
   
   TCP only (`net/tcp/tcp_close.c`, `tcp_shutdown.c`, `tcp_conn.c`, `tcp.h`). 
No API, ABI or Kconfig change.
   
   - `shutdown(SHUT_WR)` and `close()` deliver the queued data before the FIN.
   - `close()` with unread data, or with `SO_LINGER` {1, 0}, sends a RST at 
once (Linux does the same). The peer gets a reset instead of waiting.
   - A connection in TIME_WAIT holds one devif callback instead of three, and 
no closed connection stays in the active list after a reset on SMP, after 
`shutdown()` + `close()`, or after its device went down. With 
`CONFIG_NET_TCP_ALLOC_CONNS` these were kernel heap leaks (about 520 bytes 
each).
   - `tcp_alloc()` frees a connection in `TCP_CLOSED` only while its close work 
is still queued (it takes the work from the queue). A closed connection that 
`tcp_close_work()` or `close()` frees at that moment is left to them.
   - Size: see the build table below.
   
   ## Testing
   
   qemu-armv8a under HVF (Apple M-series), TCP over the loopback device: 
`NET_IPv4`, `NET_TCP`, `NET_TCPBACKLOG`, `NET_LOOPBACK`, `NET_SOCKOPTS`, 
`NET_SOLINGER`, `NET_TCP_WRITE_BUFFERS`, `NET_TCP_KEEPALIVE`, `NET_STATISTICS` 
(for `/proc/net/tcp`), 64 IOBs of 1534 bytes, `NET_RECV_BUFSIZE` 32768. 
"dynamic": `NET_TCP_ALLOC_CONNS=1`, `NET_ALLOC_DEVIF_CALLBACKS=1`; "default": 8 
connections and 16 devif callbacks. knsh: kernel build, the programs from 
`/system/bin`; nsh: flat build. Master is fc8d354f9a1, the base of this PR.
   
   Test programs (standalone, run from NSH):
   
   - `tcpshutdata`: 20 rounds: the writer fills the window and its write queue 
with non-blocking `send()`, then `shutdown(SHUT_WR)`; the reader must get every 
byte, then EOF. Then `send()` after `shutdown(SHUT_WR)`, `shutdown()` + 
`close()` at once, `SO_LINGER` {1, 0} with data queued, "closeunread" (5 pairs 
close with unread data both ways, then 4 MiB over a new connection, free IOBs 
before and after), "bulk" (3 x 8 MiB) and 4 checks of how a reset is reported.
   - `tcpeof`: 1 MiB, then `close()`; a forked reader counts the bytes up to 
EOF; 10 times, in 16 KiB `send()`s (p) and in one `send()` (s).
   - `leakloop`: 3 batches of 150 closes with 1 KiB unread (tcpunread), of 150 
closes with `SO_LINGER` {1, 0} (tcplinger0), and of 100 rounds of both and a 
normal close (tcp); after each batch it waits until TIME_WAIT ended and 
compares `/proc/net/tcp`, the kernel heap and the IOBs.
   - `twcount`: 100 connections end in TIME_WAIT; the kernel heap per 
connection, and again after 130 s.
   - `cmocka_net_tcp` with the 5 new cases of apache/nuttx-apps#MMMM.
   
   qemu-armv8a:knsh (kernel), 1 CPU, dynamic pools:
   
   | Test | master | this PR |
   |---|---|---|
   | tcpshutdata: 20 rounds of `shutdown(SHUT_WR)` with data queued | 0 of 20: 
each reader gets 32768 of 83132 bytes, then EOF | 20 of 20 |
   | tcpshutdata: `send()` after `shutdown(SHUT_WR)`; `shutdown()` + `close()` 
at once | both FAIL (the reader gets EOF early) | both PASS |
   | tcpshutdata closeunread: free IOBs after the 5 closes (of 64) | 30 | 64 |
   | `/proc/iobinfo` after tcpshutdata: free IOBs | 30 | 64 |
   | tcpeof p, s (10 x 1 MiB) | 10 of 10, 10 of 10 | 10 of 10, 10 of 10 |
   | leakloop tcpunread, tcplinger0, tcp | PASS, +0 connections | PASS, +0 
connections |
   | cmocka_net_tcp | (not run) | 11 passed, `test_tcp_connect_ipv4` failed 
(also on master) |
   | twcount: kernel heap per connection in TIME_WAIT (100 connections; 130 s 
later all back) | +613.4 B | +485.3 B, +522.2 B (2 runs) |
   | ostest | status 0 | status 0 |
   
   The other 5 tcpshutdata checks fail before and after: 4 test how a reset is 
reported to the reader (ECONNRESET), which this PR does not change, and 
`SO_LINGER` {1, 0} with data queued expects ECONNRESET too (with this PR the 
reader gets the data sent before the RST, then end-of-file).
   
   qemu-armv8a:knsh, 4 CPUs (`CONFIG_SMP`), dynamic pools:
   
   | Test | master | this PR |
   |---|---|---|
   | tcpshutdata | 3 of 29 | 23 of 29 (25 of 29 in an earlier run) |
   | tcpeof p, s | 10 of 10; 8 of 10 (2 short `send()`s) | 10 of 10; 9 of 10 (1 
short `send()`) |
   | after tcpeof | the shell does not come back: QEMU at 381 % CPU (4 vCPUs) 
after 8 minutes, stopped | leakloop, cmocka and ostest run |
   | leakloop tcpunread (3 x 150) | 2, 0, 1 connections left per batch in 
`TCP_CLOSED` with no reference (`st 00 ref 0` in `/proc/net/tcp`) | +0 |
   | leakloop tcplinger0, tcp | tcplinger0 +0 (tcp not run) | +0, +0 |
   | cmocka_net_tcp | not run | 11 passed, `test_tcp_connect_ipv4` failed (also 
on master) |
   | ostest | not run | status 0 |
   
   With this PR the 2 extra tcpshutdata failures on 4 CPUs are slow transfers 
("closeunread" 4 MiB and one "bulk" 8 MiB did not end in 20 s): the closed 
connections gave back all IOBs (64 of 64), and the receive window of the new 
connection stayed small; this PR does not change the receive window. A short 
`send()` is a 1 MiB `send()` that returned less after a 20 s stall (our copy of 
tcpeof sets `SO_SNDTIMEO` 20 s); in every iteration the reader got exactly the 
bytes sent, then EOF.
   
   qemu-armv8a:knsh, 1 CPU, default pools (8 connections, 16 callbacks): 
tcpshutdata 3 of 29 on master, 25 of 29 with this PR (the same 4 reset checks 
fail); tcpeof p and s 10 of 10 on both; twcount makes its 100 connections 
through the 8 connection structures on both (`tcp_alloc()` frees the oldest one 
in TIME_WAIT); ostest status 0 on both.
   
   cmocka_net_tcp with the new cases, default pools (8 connections, 16 
callbacks), 1 CPU:
   
   | Configuration | master | this PR |
   |---|---|---|
   | qemu-armv8a:knsh, 3 runs in one boot | runs 1 and 2: the 5 new cases fail; 
run 3: 3 of them | 11 passed in each run |
   | qemu-armv8a:nsh (flat) | the 5 new cases fail | 11 passed |
   
   `test_tcp_connect_ipv4` fails in every run, on master too: its loopback 
server refuses the connection.
   
   With dynamic connections and the default fixed pool of 16 callbacks 
(`NET_TCP_ALLOC_CONNS=1`, `NET_ALLOC_DEVIF_CALLBACKS=0`), 3 runs of 
cmocka_net_tcp in one boot: on master 4 of the 5 new cases fail in each run 
(the reader gets 32768 of 83132 bytes and no EOF, then `shutdown()` gives 
`EBUSY`; `unread` passes because `close()` finds no callback and frees both 
connections at once). With this PR the first run passes (11 passed); the next 
runs fail at once with `EBUSY` from `connect()` and `shutdown()`: 14 
connections in TIME_WAIT hold 14 of the 16 callbacks, and only `close()` and 
`tcp_alloc()` free such a connection when the pool is empty. See "Not in this 
PR" below.
   
   rv-virt:knsh64 (QEMU TCG, riscv64): the test program faults at its start in 
`lib_stdoutstream()`, before any socket call, on master and with this PR alike, 
so there is no runtime result for riscv64; it builds (below).
   
   Builds in the NuttX CI image (`EXTRAFLAGS="-Wno-cpp -Werror"`), master and 
this PR: all OK, 0 warnings. Text size (bytes) master -> this PR:
   
   | Configuration | Build mode | master | this PR |
   |---|---|---|---|
   | qemu-armv8a:netnsh (no write buffers) | flat | 547544 | 548192 (+648) |
   | qemu-armv8a:netnsh + `NET_TCP_WRITE_BUFFERS`, `NET_SOLINGER` | flat | 
550560 | 551280 (+720) |
   | the same, CMake | flat | 551344 | 552064 (+720) |
   | qemu-armv8a:netnsh_smp + `NET_TCP_WRITE_BUFFERS`, `NET_SOLINGER` | flat, 
SMP | 526056 | 526776 (+720) |
   | rv-virt:pnsh64 + TCP, write buffers, `NET_SOLINGER` | protected | 183712 | 
184240 (+528) |
   | rv-virt:pnsh64 + TCP, no write buffers | protected | 181264 | 181728 
(+464) |
   | rv-virt:pnsh64 + TCP, write buffers, no `NET_SOCKOPTS` | protected | 
181680 | 182456 (+776) |
   | rv-virt:knetnsh64 | kernel | 227879 | 228409 (+530) |
   | rv-virt:knetnsh64 + write buffers, `NET_SOLINGER`, dynamic pools | kernel 
| 230299 | 230893 (+594) |
   | sim:tcpblaster (write buffers, `NET_SOLINGER`) | sim | 993650 | 995330 
(+1680) |
   | sim:tcpblaster without write buffers and `NET_SOLINGER` | sim | 984922 | 
986434 (+1512) |
   | lm3s6432-s2e:nsh (96 KiB flash) | flat | 63344 | 63828 (+484), flash 65.84 
% |
   
   rv-virt:knetnsh64 with CMake also builds (with an earlier version of commit 
6). In qemu-armv8a:netnsh `NET_SOCKOPTS` cannot be turned off (an enabled 
program selects it), so the build without it is rv-virt:pnsh64.
   
   `tools/checkpatch.sh -c -u -m -g` passes. Documentation: commit 8 (TCP state 
machine page); no Kconfig option changes.
   
   ## Not in this PR
   
   - Only `close()` and `tcp_alloc()` free a closing connection when a pool is 
empty. `connect()`, `send()`, `recv()`, `shutdown()`, `sendfile()` and `poll()` 
still fail with `EBUSY` when no devif callback is free. With dynamic 
connections and a fixed callback pool, each connection in TIME_WAIT holds its 
close callback for `CONFIG_NET_TCP_WAIT_TIMEOUT`.
   - When `close()` finds no callback even after `tcp_reclaim()`, it still 
frees the connection with no FIN and no RST: without a callback there is no way 
to send a segment from `close()`. With commit 5 this needs another task to take 
the callback of the monitor between the two calls.
   - `tcp_free()` waits in `work_cancel_sync()` for the timer work, which takes 
the device lock (`tcp_timer_expiry()`). Some callers of `tcp_free()` hold that 
device lock (`tcp_input()`, `tcp_timer()`); `tcp_unlisten()` calls it with the 
connection list lock held. This PR does not add such a call: `tcp_reclaim()` 
calls `tcp_free()` with no lock held (from a SYN, with the lock of that device 
held, as `tcp_input()` does today).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to