casaroli opened a new pull request, #22077: URL: https://github.com/apache/nuttx/pull/22077
## Summary This PR fixes four defects of TCP `accept()` and `connect()` when an event of the peer comes near the end of the call. Each commit fixes one defect; the last commit is the documentation. 1. **`net/tcp: Keep a connection whose peer closed before the monitor started.`** `tcp_start_monitor()` refused every connection that was not in `TCP_ESTABLISHED` or `TCP_SYN_RCVD`. A peer that closes its half (FIN, `TCP_CLOSE_WAIT`) before the monitor starts made `connect()` fail with `ENOTCONN` after the connection was made (a server that accepts and closes at once), and `accept()` fail for a client that closed first. Now `TCP_CLOSE_WAIT` starts the monitor and sets `SHUT_RD`, as the monitor does for the `TCP_RXCLOSE` event of the FIN. The check of the state and the start of the monitor are also one step under the device lock: with SMP, a FIN handled on another CPU between the two reached no monitor, and the next read waited for ever. 2. **`net/tcp: Lock the listener in tcp_accept_connection().`** `psock_tcp_accept()` looks at the backlog and installs its accept callback with the listener's lock held. `tcp_accept_connection()` (the input path) looked at the callback without that lock, and took the lock only to add the connection to the backlog. A connection whose handshake completed between the two went to the backlog, and `accept()` slept with a connection waiting (the peer, already connected, waited for data). Now the input path takes the listener's lock before it looks at the callback (the device lock first, as in the other paths). 3. **`net/tcp: Mark a TCP socket connected with the device locked.`** `psock_accept()` and `psock_connect()` set `_SF_CONNECTED` (and cleared `_SF_CLOSED`) after the TCP call returned, without the device lock. The monitor that the TCP call started changes the same flags for an event of the peer, with the device locked. A RST, a time-out or a device that went down in between was undone: the socket looked connected again, and a `send()` queued data on a closed connection or waited for ever. Now `tcp_start_monitor()` marks the socket with the device locked. A new socket capability, `SOCKCAP_CONNSTATE`, tells `psock_accept()` and `psock_connect()` that the address family sets these flags itself; only TCP gives it. The other address families do not change. 4. **`net/tcp: Report a reset that comes before accept() or connect() returns.`** A RST after the handshake, before `accept()` or `connect()` returned, gave `ENOTCONN`. Now, as on Linux, `accept()` gives the socket and the reset is reported as for a reset after `accept()` (apache/nuttx#22029): the data that came before the RST, then `ECONNRESET` once, then end-of-file. `connect()` fails with `ECONNRESET`, and the socket is not connected. 5. **`Documentation/os/networking: Describe a close or reset before accept().`** The TCP state machine page, and `ECONNRESET` in the errors of `connect()`. The changes are in common network code: the same in flat, protected and kernel builds, on every architecture. ## Behaviour What the calls give when the peer closes or resets the connection after the handshake, before the call returns (the cases of apache/nuttx-apps#MMMM; Linux 7.2 checked with a host program): | Event before the call returns | master | this PR (and Linux) | |---|---|---| | client sends 5 bytes and closes (FIN) before `accept()` | `accept()` -1 `ENOTCONN` | socket; `recv()` 5 bytes, then 0 | | client closes (FIN) before `accept()` | -1 `ENOTCONN` | socket; `recv()` 0 | | client shuts down its writing side before `accept()` | -1 `ENOTCONN` | socket; `recv()` 0; `send()` reaches the client | | server accepts and closes (FIN) before `connect()` returns | `connect()` -1 `ENOTCONN` | 0; `recv()` 0 | | client resets (RST) before `accept()` returns | `accept()` -1 `ENOTCONN` | socket; `recv()` -1 `ECONNRESET`, then 0; `send()` -1 `EPIPE` | | client sends 5 bytes and resets before `accept()` returns | socket; `recv()` 5 bytes, `SO_ERROR` `ECONNRESET`, then `recv()` gives `EAGAIN` with `MSG_DONTWAIT`: the socket looks connected | socket; `recv()` 5 bytes, `SO_ERROR` `ECONNRESET`, then `recv()` 0 | | server accepts and resets before `connect()` returns | `connect()` -1 `ENOTCONN` | -1 `ECONNRESET`; `recv()` -1 `ENOTCONN`; `SO_ERROR` 0 | | RST just after `accept()` started the monitor (window widened by a test-only 50 ms delay) | `send()` succeeds and `recv()` gives `EAGAIN` on the reset connection (90 of 100 rounds) | `send()` -1 `ECONNRESET`, `recv()` 0 (100 of 100) | A connection that the peer resets while it waits in the backlog of the listener is freed, as before: `accept()` waits for the next one (Linux gives the socket; this PR does not change it). ## Impact TCP, plus the flag handling of `psock_accept()` and `psock_connect()`. No user API, ABI or Kconfig change. `include/nuttx/net/net.h` (kernel only) gets `SOCKCAP_CONNSTATE`; only `inet_sockcaps()` returns it, for `SOCK_STREAM` with the TCP stack, so `psock_accept()` and `psock_connect()` do exactly what they did for every other address family and socket type (`local`, `usrsock`, `rpmsg`, `vsock`, UDP, `netlink`, `bluetooth`, `ieee802154`; `netdev_ioctl()` `FIONBIO` tests only `SOCKCAP_NONBLOCKING`). For TCP the flags end the same on success; `_SF_NONBLOCK` of `accept4(SOCK_NONBLOCK)` is set by `inet_accept()` before the monitor starts. The errors that change are in the table. In-tree users of `accept()`/`connect()` (nuttx and nuttx-apps, grep of `ENOTCONN` and of the in-kernel `psock_accept()`/`psock_connect()` users): none tests `ENOTCONN` from `accept()` or `connect()`. `fs/nfs` already reconnects on `ECONNRESET` (apache/nuttx#22029). `drivers/video/vnc`, `drivers/usrsock/usrsock_rpmsg_server.c`, `drivers/serial/netdev_console.c`, `drivers/tee/optee_socket.c`, `fs/v9fs/socket_9p.c` and `inet_socketpair()` treat any negative result as a failure. Locks: commit 1 takes `conn_dev_lock()` in `tcp_start_monitor()`; the connect path already holds it (recursive mutexes). Commit 2 takes the listener's lock after the device lock in the input path; the input path already took it there for the backlog, so the order is the same as on master. Commit 3 only moves the flag writes into the section that holds the device lock. Size: +88 to +344 bytes of text on arm64 and riscv64, about +520 on sim (the build table below). ## Testing qemu-armv8a under HVF (Apple M-series), TCP over the loopback device: `NET_IPv4`, `NET_TCP`, `NET_TCPBACKLOG`, `NET_LOOPBACK`, `NET_SOCKOPTS`, `NET_SOLINGER`, `NET_TCP_WRITE_BUFFERS`, `NET_TCP_KEEPALIVE`, 64 IOBs of 1534 bytes; the default pools (8 connections, 16 devif callbacks) unless "dynamic" (`NET_TCP_ALLOC_CONNS=1`, `NET_ALLOC_DEVIF_CALLBACKS=1`). knsh: kernel build, the programs from `/system/bin`; nsh: flat build. The tests ran on master 84abb4756d3; the branch is now on 1194bc8c4ff, which changes only `net/rpmsg` and `net/net_initialize.c` in `net/` (the 4-CPU master and rv-virt master runs used 47ddca21973, the same TCP code). Tests: - `cmocka_net_tcp` with the 8 new cases of apache/nuttx-apps#MMMM (`testing/nettest/tcp/test_tcp_closewait.c`), 2 runs per boot. - `rstrace` (a standalone program): 100 rounds of a client that resets the connection 0 to 9 ms after its `connect()` returned, to an `accept()` that waits; then `send()` and `recv()` on the accepted socket must report the reset. On Linux 7.2: 100 of 100. - `acceptrace` (a standalone program): 1000 rounds of a thread that accepts and writes 4 bytes while main connects and reads them (the race of commit 2). - ostest. | Configuration | master | this PR | |---|---|---| | knsh, 1 CPU: the 8 new cmocka cases | 8 fail in each run | 8 pass in each run | | knsh, 1 CPU: rstrace / acceptrace / ostest | 90 of 100 (10 `accept()` `ENOTCONN`) / PASS / 0 | 100 of 100 / PASS / 0 | | knsh, 1 CPU, dynamic pools | not run | 8 pass; rstrace 100/100; acceptrace PASS; ostest 0 | | nsh (flat), 1 CPU: the 8 new cases / ostest | 8 fail / 0 | 8 pass / 0 | | knsh, 1 CPU, test-only 50 ms delay after `si_accept()` in `psock_accept()`: rstrace | `send()` succeeds after the RST in 90 of 100 rounds (10 `ENOTCONN`) | 100 of 100 | | knsh, 4 CPUs (`CONFIG_SMP`): the 8 new cases | assertion `tcp_conn.c:820` in the first run | 8 pass in each run (then see below) | | knsh, 4 CPUs, with apache/nuttx#22025: cmocka / rstrace | #22025 alone: 7 of 8 fail | 8 pass in each run; rstrace 100/100 | | knsh, 4 CPUs, dynamic pools, with apache/nuttx#22025: cmocka / acceptrace / rstrace / ostest | #22025 alone: 7 of 8 fail; acceptrace PASS | 8 pass; acceptrace PASS; rstrace 100/100; ostest 0 | | rv-virt:knsh64 (riscv64, QEMU TCG), dynamic pools: rstrace / ostest | 100 of 100 / 0 (the timing of TCG does not show the defect) | 100 of 100 / 0 | On 4 CPUs without apache/nuttx#22025, master stops at an assertion of the close path in the first `cmocka_net_tcp` run (`tcp_free()` called by the close work for a connection with a reference, `tcp_conn.c:820`, in the older case `test_tcp_listen_again`). With this PR alone the 8 new cases pass in each run before the same defect comes later: the same assertion in `acceptrace`, or a task that waits for ever on the lock of a free connection slot (gdb). apache/nuttx#22025 fixes that defect of the close path: with it, 4 `cmocka_net_tcp` runs pass, and with dynamic pools also `acceptrace`, `rstrace` and ostest. With the default pools (8 connections) on 4 CPUs, `acceptrace` and an older case can run out of connections (`connect()` `ENOMEM`, `socket()` fails), with #22025 alone too. `test_tcp_connect_ipv4` (an older case) fails in every run, on master too: its loopback server refuses the connection, and the group teardown then fails. With apache/nuttx#22025, #21900 and #22069: `git merge-tree` gives no conflict with any of them. The branch is on master and needs none of them. Builds in the NuttX CI image (`EXTRAFLAGS="-Wno-cpp -Werror"`), master 84abb4756d3 and this PR (the same code on 84abb4756d3): all 32 OK, 0 warnings. Text size (bytes): | Configuration | Build mode | master | this PR | |---|---|---|---| | qemu-armv8a:netnsh | flat | 549408 | 549728 (+320) | | qemu-armv8a:netnsh + `NET_TCP_WRITE_BUFFERS`, `NET_SOLINGER` | flat | 552380 | 552700 (+320) | | the same, CMake | flat | 553164 | 553484 (+320) | | qemu-armv8a:netnsh_smp + `NET_TCP_WRITE_BUFFERS`, `NET_SOLINGER` | flat, SMP | 528000 | 528320 (+320) | | rv-virt:pnsh64 + TCP, write buffers, `NET_SOLINGER` | protected | 184672 | 185016 (+344) | | rv-virt:pnsh64 + TCP, no write buffers | protected | 182512 | 182600 (+88) | | rv-virt:pnsh64 + TCP, write buffers, no `NET_SOCKOPTS` | protected | 182880 | 182968 (+88) | | rv-virt:knetnsh64 | kernel | 229123 | 229357 (+234) | | the same, CMake | kernel | 229139 | 229373 (+234) | | rv-virt:knetnsh64 + write buffers, `NET_SOLINGER`, dynamic pools | kernel | 231501 | 231735 (+234) | | sim:tcpblaster | sim | 997810 | 998330 (+520) | | sim:tcpblaster without write buffers and `NET_SOLINGER` | sim | 989122 | 989634 (+512) | | sim:usrsocktest (`usrsock` sockets, generic `accept()`/`connect()`) | sim | 658453 | 658549 (+96) | | sim:rpserver (`rpmsg` sockets) | sim | 1144002 | 1144542 (+540) | | lm3s6432-s2e:nsh (96 KiB flash) | flat | 64764 | 65008 (+244), flash 67.16 % | In qemu-armv8a:netnsh `NET_SOCKOPTS` cannot be turned off (an enabled program selects it), so the build without it is rv-virt:pnsh64. `tools/checkpatch.sh -c -u -m -g` passes. Documentation: commit 5. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
