casaroli opened a new pull request, #22078: URL: https://github.com/apache/nuttx/pull/22078
## Summary Three defects of TCP retransmission, each in its own commit, and the documentation. 1. **`net/tcp: Restart the retransmission timer only as RFC 6298 says.`** `tcp_fill_header()` restarted the retransmission timer for every outgoing segment, also for an ACK without data, and `tcp_input()` restarted it for every ACK that was not an old one, also for a duplicate ACK (with an RTT sample from it and a reset of `conn->nrtx`). While the peer sends data, each of its segments carries such an ACK and gets an ACK from us, so the timer did not expire until the peer stopped. RFC 6298, section 5: the timer starts when data is sent and restarts when an ACK acknowledges new data. Now only a segment that takes sequence space (data, SYN, FIN) restarts it when it is sent, and an ACK that is not beyond `snd_wl2` (the highest ACK so far) gives no RTT sample and leaves the timer, `sndseq` and `nrtx` alone (it still sets `TCP_ACKDATA`). 2. **`net/tcp: Do not count a segment with data as a duplicate ACK.`** RFC 5681, section 2: a duplicate ACK carries no data. A peer that sends data while our segment is in flight repeats the ACK number in each segment. `psock_send_eventhandler()` and NewReno's `tcp_cc_recv_ack()` counted those segments, so every third one started a fast retransmission. The fast retransmission for a segment with data also corrupted the stream: it notified the driver while the device held the received segment, the driver can poll at once in `netdev_txnotify_dev()`, and after that poll the code put back the buffer and the lengths but not `d_sndlen`. When the poll stopped right after a sent segment, the reply to the received segment sent those bytes as new data, and then the real data under the same sequence numbers (an SFTP upload over SSH failed with a bad MAC). Now only a segment without data is a duplicate ACK, and that code is removed. 3. **`net/tcp: Double the retransmission timeout at the first expiry too.`** `tcp_timer()` computed `conn->rto = TCP_RTO << nrtx` before it counted the expiry, so the timeouts of one segment were 1.5 s, 1.5 s, 3 s, 6 s (with the default `NET_TCP_RTO`). RFC 6298, section 5.5. Now 1.5 s, 3 s, 6 s. `TCP_MAXRTX`, `TCP_MAXSYNRTX` and the largest shift do not change; the time until `TCP_MAXRTX` gives up grows from about 96 s to about 118 s. 4. **`Documentation/os/networking: Describe the retransmission timer.`** The TCP state machine page. ## Impact TCP only (`net/tcp`, one call in `net/sixlowpan/sixlowpan_tcpsend.c`). No API, ABI or Kconfig change. `tcp_fill_header()` and `tcp_cc_recv_ack()` get one more argument; both are internal (`net/tcp/tcp.h`), and every caller is changed (`tcp_send.c`, `sixlowpan_tcpsend.c`, `tcp_input.c`). Behaviour on the wire: fewer retransmissions (no fast retransmission from segments with data, no ACK that keeps the timer from expiring) and the second timeout of a segment twice the first. A segment with data that repeats the ACK number no longer triggers a fast retransmission with `NET_TCP_FAST_RETRANSMIT` or NewReno; three duplicate ACKs without data still do. Build modes and options: common network code, the same in flat, protected and kernel builds and on every architecture. With and without `NET_TCP_WRITE_BUFFERS`, `NET_TCP_FAST_RETRANSMIT`, `NET_TCP_CC_NEWRENO`, `NET_TCP_SELECTIVE_ACK`, `NET_TCPURGDATA`, `NET_SENDFILE`, `NET_TCP_FIXED_RTO`: see the builds below. SMP: the changed code runs under the device and connection locks, as before. ## Testing qemu-armv8a:knsh (kernel build) under HVF (Apple M-series), TCP over the loopback device and a virtio-net NIC that QEMU loops back to itself (`-netdev dgram` to its own port), 64 IOBs of 1534 bytes, write buffers, dynamic connections and callbacks. qemu-armv8a:nsh (flat build) with the default pools. Master is 84abb4756d3 (one 4-CPU master boot: 1194bc8c4ff, no network change between them); the tests are standalone programs run from `/system/bin`. No riscv64 or x86_64 run here. The runs and the sizes below are of an earlier version of this PR that also had a commit for the receive window update (RFC 1122, section 4.2.3.3); that commit is not in this PR. - `dupackdata`: a scripted TCP peer on a packet socket of the looped-back NIC, in the same guest. Per round: the handshake with a one-segment window; the server writes 32 KiB; the peer sends 40 data segments 0.1 s apart that repeat the old ACK (as if our first segment were lost), then ACKs everything. PASS needs 1 or 2 timeout retransmissions of the first segment in those 4 s, none within 50 ms of a data segment of the peer (a fast retransmission), the stream byte-exact and no sequence number with two payloads. `--backoff`: 60 segments over 6 s; the second timeout must be at least 1.6 times the first. - `tcpbulk bulk`: 3 x 8 MiB over 127.0.0.1, 20 s each. `tcpbulk zwin`: 3 x 4 MiB where the reader starts after 1 s (the window fills, the sender's write buffers take the free IOBs, then the window must open again). - `tcpeof 1024 10 p|s`: 1 MiB in 16 KiB `send()`s or in one `send()`, then `close()`; the reader must get every byte and then end-of-file, 10 times. - `cmocka_net_tcp` (nuttx-apps) and `ostest`. | Test | master | this PR | |---|---|---| | knsh 1 CPU: `dupackdata` 5 rounds, twice | FAIL in round 0: 7 fast retransmissions of the first segment in 4 s | 10 of 10 rounds pass: 1 timeout retransmission, no fast one, stream byte-exact | | knsh 1 CPU: `dupackdata --backoff` 3 rounds (2 boots) | FAIL in round 0: 7 fast retransmissions | 6 of 6 rounds pass | | knsh 4 CPUs (`CONFIG_SMP`): `dupackdata` 5 rounds twice, `--backoff` 3 rounds (2 boots) | FAIL in round 0: 7 fast retransmissions | 16 of 16 rounds pass, no fast retransmission | | knsh 1 CPU: `tcpbulk bulk`, `tcpbulk zwin` x3, `tcpeof` p and s | pass | pass | | knsh 1 and 4 CPUs, nsh: `cmocka_net_tcp` | 6 of 7 (`test_tcp_connect_ipv4` fails) | the same | | knsh 1 and 4 CPUs, nsh: `ostest` | status 0 | status 0 | TCP over 127.0.0.1 with 4 CPUs: a reader sometimes gets `ECONNRESET` or a transfer stalls, on master and with this PR at about the same rate. 4 CPUs, 5 boots each: `tcpeof` 6 of 130 iterations fail on master, 5 of 120 with this PR; `tcpbulk bulk` 6 of 10 runs fail on master, 4 of 8 with this PR; `tcpbulk zwin` 3 of 11 and 2 of 9; 2 and 1 boots stop in a hang after a transfer. With `CONFIG_DEBUG_NET_WARN` the reset is `psock_send_eventhandler: WARNING: Expiring wrb ... nrtx=8`: a write buffer that the loopback device dropped (no free IOB at the receiver) reaches `TCP_MAXRTX` retransmissions and the connection is reset. The count of a write buffer is not reset when an ACK takes part of it. That is not changed here. `test_tcp_connect_ipv4` of `cmocka_net_tcp` fails on master and with this PR alike (its loopback server refuses the connection). Builds in the NuttX CI image (`EXTRAFLAGS="-Wno-cpp -Werror"`), master and this PR: all OK, 0 warnings. Text size (bytes): | Configuration | Build mode | master | this PR | |---|---|---|---| | qemu-armv8a:netnsh (no write buffers) | flat | 549408 | 549464 (+56) | | qemu-armv8a:netnsh + `NET_TCP_WRITE_BUFFERS`, `NET_SOLINGER` | flat | 552380 | 552380 (+0) | | the same, CMake | flat | 553164 | 553164 (+0) | | qemu-armv8a:netnsh_smp + write buffers | flat, SMP | 528000 | 528000 (+0) | | qemu-armv8a:netnsh + write buffers + `NET_TCP_CC_NEWRENO` | flat | 553056 | 553068 (+12) | | qemu-armv8a:netnsh + `NET_TCP_CC_NEWRENO`, no write buffers | flat | 550068 | 550136 (+68) | | qemu-armv8a:netnsh + write buffers, `NET_TCP_FAST_RETRANSMIT` off | flat | 552124 | 552184 (+60) | | qemu-armv8a:netnsh + write buffers, NewReno, out-of-order, SACK, urgent data, sendfile, delayed ACK | flat | 557188 | 557208 (+20) | | the same without write buffers and NewReno, `NET_TCP_FIXED_RTO` | flat | 552740 | 552804 (+64) | | rv-virt:pnsh64 + TCP, write buffers | protected | 184672 | 184864 (+192) | | rv-virt:pnsh64 + TCP, no write buffers | protected | 182512 | 182512 (+0) | | rv-virt:knetnsh64 | kernel | 229123 | 229161 (+38) | | rv-virt:knetnsh64 + write buffers, dynamic pools | kernel | 231501 | 231479 (-22) | | lc823450-xgevk:krndis | protected | 193228 | 193284 (+56) | | sim:tcpblaster | sim | 997810 | 997698 (-112) | | sim:tcpblaster without write buffers | sim | 989122 | 989266 (+144) | | sim:sixlowpan | sim, 6LoWPAN | 637756 | 637676 (-80) | | samv71-xult:mrf24j40-starhub | flat, 6LoWPAN | 244864 | 244832 (-32) | | lm3s6432-s2e:nsh (96 KiB flash) | flat | 64764 | 64788 (+24) | `tools/checkpatch.sh -c -u -m -g` passes. Documentation: commit 4. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
