casaroli opened a new pull request, #22077:
URL: https://github.com/apache/nuttx/pull/22077

   ## Summary
   
   This PR fixes four defects of TCP `accept()` and `connect()` when an event 
of the peer comes near the end of the call. Each commit fixes one defect; the 
last commit is the documentation.
   
   1. **`net/tcp: Keep a connection whose peer closed before the monitor 
started.`** `tcp_start_monitor()` refused every connection that was not in 
`TCP_ESTABLISHED` or `TCP_SYN_RCVD`. A peer that closes its half (FIN, 
`TCP_CLOSE_WAIT`) before the monitor starts made `connect()` fail with 
`ENOTCONN` after the connection was made (a server that accepts and closes at 
once), and `accept()` fail for a client that closed first. Now `TCP_CLOSE_WAIT` 
starts the monitor and sets `SHUT_RD`, as the monitor does for the 
`TCP_RXCLOSE` event of the FIN. The check of the state and the start of the 
monitor are also one step under the device lock: with SMP, a FIN handled on 
another CPU between the two reached no monitor, and the next read waited for 
ever.
   2. **`net/tcp: Lock the listener in tcp_accept_connection().`** 
`psock_tcp_accept()` looks at the backlog and installs its accept callback with 
the listener's lock held. `tcp_accept_connection()` (the input path) looked at 
the callback without that lock, and took the lock only to add the connection to 
the backlog. A connection whose handshake completed between the two went to the 
backlog, and `accept()` slept with a connection waiting (the peer, already 
connected, waited for data). Now the input path takes the listener's lock 
before it looks at the callback (the device lock first, as in the other paths).
   3. **`net/tcp: Mark a TCP socket connected with the device locked.`** 
`psock_accept()` and `psock_connect()` set `_SF_CONNECTED` (and cleared 
`_SF_CLOSED`) after the TCP call returned, without the device lock. The monitor 
that the TCP call started changes the same flags for an event of the peer, with 
the device locked. A RST, a time-out or a device that went down in between was 
undone: the socket looked connected again, and a `send()` queued data on a 
closed connection or waited for ever. Now `tcp_start_monitor()` marks the 
socket with the device locked. A new socket capability, `SOCKCAP_CONNSTATE`, 
tells `psock_accept()` and `psock_connect()` that the address family sets these 
flags itself; only TCP gives it. The other address families do not change.
   4. **`net/tcp: Report a reset that comes before accept() or connect() 
returns.`** A RST after the handshake, before `accept()` or `connect()` 
returned, gave `ENOTCONN`. Now, as on Linux, `accept()` gives the socket and 
the reset is reported as for a reset after `accept()` (apache/nuttx#22029): the 
data that came before the RST, then `ECONNRESET` once, then end-of-file. 
`connect()` fails with `ECONNRESET`, and the socket is not connected.
   5. **`Documentation/os/networking: Describe a close or reset before 
accept().`** The TCP state machine page, and `ECONNRESET` in the errors of 
`connect()`.
   
   The changes are in common network code: the same in flat, protected and 
kernel builds, on every architecture.
   
   ## Behaviour
   
   What the calls give when the peer closes or resets the connection after the 
handshake, before the call returns (the cases of apache/nuttx-apps#MMMM; Linux 
7.2 checked with a host program):
   
   | Event before the call returns | master | this PR (and Linux) |
   |---|---|---|
   | client sends 5 bytes and closes (FIN) before `accept()` | `accept()` -1 
`ENOTCONN` | socket; `recv()` 5 bytes, then 0 |
   | client closes (FIN) before `accept()` | -1 `ENOTCONN` | socket; `recv()` 0 
|
   | client shuts down its writing side before `accept()` | -1 `ENOTCONN` | 
socket; `recv()` 0; `send()` reaches the client |
   | server accepts and closes (FIN) before `connect()` returns | `connect()` 
-1 `ENOTCONN` | 0; `recv()` 0 |
   | client resets (RST) before `accept()` returns | `accept()` -1 `ENOTCONN` | 
socket; `recv()` -1 `ECONNRESET`, then 0; `send()` -1 `EPIPE` |
   | client sends 5 bytes and resets before `accept()` returns | socket; 
`recv()` 5 bytes, `SO_ERROR` `ECONNRESET`, then `recv()` gives `EAGAIN` with 
`MSG_DONTWAIT`: the socket looks connected | socket; `recv()` 5 bytes, 
`SO_ERROR` `ECONNRESET`, then `recv()` 0 |
   | server accepts and resets before `connect()` returns | `connect()` -1 
`ENOTCONN` | -1 `ECONNRESET`; `recv()` -1 `ENOTCONN`; `SO_ERROR` 0 |
   | RST just after `accept()` started the monitor (window widened by a 
test-only 50 ms delay) | `send()` succeeds and `recv()` gives `EAGAIN` on the 
reset connection (90 of 100 rounds) | `send()` -1 `ECONNRESET`, `recv()` 0 (100 
of 100) |
   
   A connection that the peer resets while it waits in the backlog of the 
listener is freed, as before: `accept()` waits for the next one (Linux gives 
the socket; this PR does not change it).
   
   ## Impact
   
   TCP, plus the flag handling of `psock_accept()` and `psock_connect()`. No 
user API, ABI or Kconfig change. `include/nuttx/net/net.h` (kernel only) gets 
`SOCKCAP_CONNSTATE`; only `inet_sockcaps()` returns it, for `SOCK_STREAM` with 
the TCP stack, so `psock_accept()` and `psock_connect()` do exactly what they 
did for every other address family and socket type (`local`, `usrsock`, 
`rpmsg`, `vsock`, UDP, `netlink`, `bluetooth`, `ieee802154`; `netdev_ioctl()` 
`FIONBIO` tests only `SOCKCAP_NONBLOCKING`). For TCP the flags end the same on 
success; `_SF_NONBLOCK` of `accept4(SOCK_NONBLOCK)` is set by `inet_accept()` 
before the monitor starts.
   
   The errors that change are in the table. In-tree users of 
`accept()`/`connect()` (nuttx and nuttx-apps, grep of `ENOTCONN` and of the 
in-kernel `psock_accept()`/`psock_connect()` users): none tests `ENOTCONN` from 
`accept()` or `connect()`. `fs/nfs` already reconnects on `ECONNRESET` 
(apache/nuttx#22029). `drivers/video/vnc`, 
`drivers/usrsock/usrsock_rpmsg_server.c`, `drivers/serial/netdev_console.c`, 
`drivers/tee/optee_socket.c`, `fs/v9fs/socket_9p.c` and `inet_socketpair()` 
treat any negative result as a failure.
   
   Locks: commit 1 takes `conn_dev_lock()` in `tcp_start_monitor()`; the 
connect path already holds it (recursive mutexes). Commit 2 takes the 
listener's lock after the device lock in the input path; the input path already 
took it there for the backlog, so the order is the same as on master. Commit 3 
only moves the flag writes into the section that holds the device lock.
   
   Size: +88 to +344 bytes of text on arm64 and riscv64, about +520 on sim (the 
build table below).
   
   ## Testing
   
   qemu-armv8a under HVF (Apple M-series), TCP over the loopback device: 
`NET_IPv4`, `NET_TCP`, `NET_TCPBACKLOG`, `NET_LOOPBACK`, `NET_SOCKOPTS`, 
`NET_SOLINGER`, `NET_TCP_WRITE_BUFFERS`, `NET_TCP_KEEPALIVE`, 64 IOBs of 1534 
bytes; the default pools (8 connections, 16 devif callbacks) unless "dynamic" 
(`NET_TCP_ALLOC_CONNS=1`, `NET_ALLOC_DEVIF_CALLBACKS=1`). knsh: kernel build, 
the programs from `/system/bin`; nsh: flat build. The tests ran on master 
84abb4756d3; the branch is now on 1194bc8c4ff, which changes only `net/rpmsg` 
and `net/net_initialize.c` in `net/` (the 4-CPU master and rv-virt master runs 
used 47ddca21973, the same TCP code).
   
   Tests:
   
   - `cmocka_net_tcp` with the 8 new cases of apache/nuttx-apps#MMMM 
(`testing/nettest/tcp/test_tcp_closewait.c`), 2 runs per boot.
   - `rstrace` (a standalone program): 100 rounds of a client that resets the 
connection 0 to 9 ms after its `connect()` returned, to an `accept()` that 
waits; then `send()` and `recv()` on the accepted socket must report the reset. 
On Linux 7.2: 100 of 100.
   - `acceptrace` (a standalone program): 1000 rounds of a thread that accepts 
and writes 4 bytes while main connects and reads them (the race of commit 2).
   - ostest.
   
   | Configuration | master | this PR |
   |---|---|---|
   | knsh, 1 CPU: the 8 new cmocka cases | 8 fail in each run | 8 pass in each 
run |
   | knsh, 1 CPU: rstrace / acceptrace / ostest | 90 of 100 (10 `accept()` 
`ENOTCONN`) / PASS / 0 | 100 of 100 / PASS / 0 |
   | knsh, 1 CPU, dynamic pools | not run | 8 pass; rstrace 100/100; acceptrace 
PASS; ostest 0 |
   | nsh (flat), 1 CPU: the 8 new cases / ostest | 8 fail / 0 | 8 pass / 0 |
   | knsh, 1 CPU, test-only 50 ms delay after `si_accept()` in 
`psock_accept()`: rstrace | `send()` succeeds after the RST in 90 of 100 rounds 
(10 `ENOTCONN`) | 100 of 100 |
   | knsh, 4 CPUs (`CONFIG_SMP`): the 8 new cases | assertion `tcp_conn.c:820` 
in the first run | 8 pass in each run (then see below) |
   | knsh, 4 CPUs, with apache/nuttx#22025: cmocka / rstrace | #22025 alone: 7 
of 8 fail | 8 pass in each run; rstrace 100/100 |
   | knsh, 4 CPUs, dynamic pools, with apache/nuttx#22025: cmocka / acceptrace 
/ rstrace / ostest | #22025 alone: 7 of 8 fail; acceptrace PASS | 8 pass; 
acceptrace PASS; rstrace 100/100; ostest 0 |
   | rv-virt:knsh64 (riscv64, QEMU TCG), dynamic pools: rstrace / ostest | 100 
of 100 / 0 (the timing of TCG does not show the defect) | 100 of 100 / 0 |
   
   On 4 CPUs without apache/nuttx#22025, master stops at an assertion of the 
close path in the first `cmocka_net_tcp` run (`tcp_free()` called by the close 
work for a connection with a reference, `tcp_conn.c:820`, in the older case 
`test_tcp_listen_again`). With this PR alone the 8 new cases pass in each run 
before the same defect comes later: the same assertion in `acceptrace`, or a 
task that waits for ever on the lock of a free connection slot (gdb). 
apache/nuttx#22025 fixes that defect of the close path: with it, 4 
`cmocka_net_tcp` runs pass, and with dynamic pools also `acceptrace`, `rstrace` 
and ostest. With the default pools (8 connections) on 4 CPUs, `acceptrace` and 
an older case can run out of connections (`connect()` `ENOMEM`, `socket()` 
fails), with #22025 alone too.
   
   `test_tcp_connect_ipv4` (an older case) fails in every run, on master too: 
its loopback server refuses the connection, and the group teardown then fails.
   
   With apache/nuttx#22025, #21900 and #22069: `git merge-tree` gives no 
conflict with any of them. The branch is on master and needs none of them.
   
   Builds in the NuttX CI image (`EXTRAFLAGS="-Wno-cpp -Werror"`), master 
84abb4756d3 and this PR (the same code on 84abb4756d3): all 32 OK, 0 warnings. 
Text size (bytes):
   
   | Configuration | Build mode | master | this PR |
   |---|---|---|---|
   | qemu-armv8a:netnsh | flat | 549408 | 549728 (+320) |
   | qemu-armv8a:netnsh + `NET_TCP_WRITE_BUFFERS`, `NET_SOLINGER` | flat | 
552380 | 552700 (+320) |
   | the same, CMake | flat | 553164 | 553484 (+320) |
   | qemu-armv8a:netnsh_smp + `NET_TCP_WRITE_BUFFERS`, `NET_SOLINGER` | flat, 
SMP | 528000 | 528320 (+320) |
   | rv-virt:pnsh64 + TCP, write buffers, `NET_SOLINGER` | protected | 184672 | 
185016 (+344) |
   | rv-virt:pnsh64 + TCP, no write buffers | protected | 182512 | 182600 (+88) 
|
   | rv-virt:pnsh64 + TCP, write buffers, no `NET_SOCKOPTS` | protected | 
182880 | 182968 (+88) |
   | rv-virt:knetnsh64 | kernel | 229123 | 229357 (+234) |
   | the same, CMake | kernel | 229139 | 229373 (+234) |
   | rv-virt:knetnsh64 + write buffers, `NET_SOLINGER`, dynamic pools | kernel 
| 231501 | 231735 (+234) |
   | sim:tcpblaster | sim | 997810 | 998330 (+520) |
   | sim:tcpblaster without write buffers and `NET_SOLINGER` | sim | 989122 | 
989634 (+512) |
   | sim:usrsocktest (`usrsock` sockets, generic `accept()`/`connect()`) | sim 
| 658453 | 658549 (+96) |
   | sim:rpserver (`rpmsg` sockets) | sim | 1144002 | 1144542 (+540) |
   | lm3s6432-s2e:nsh (96 KiB flash) | flat | 64764 | 65008 (+244), flash 67.16 
% |
   
   In qemu-armv8a:netnsh `NET_SOCKOPTS` cannot be turned off (an enabled 
program selects it), so the build without it is rv-virt:pnsh64.
   
   `tools/checkpatch.sh -c -u -m -g` passes. Documentation: commit 5.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] net/tcp: Fix accept() an... casaroli (via GitHub) via commits

Reply via email to