BlueGradientHorizon opened a new issue, #20619:
URL: https://github.com/apache/nuttx/issues/20619
### Description / Steps to reproduce the issue
Two `netdev` RX worker threads wedge each other under full-duplex
load through NAT44, killing all RX on both interfaces permanently
(no panic, only external reset recovers).
Setup (ESP32-C3 SuperMini, `esp32c3-supermini` board):
- `ESPRESSIF_WIFI_STATION_SOFTAP=y` (AP+STA): `wlan0` = STA uplink
(DHCP client, gets 192.168.1.241), `wlan1` = open AP `espproxy`
(static 10.0.0.1/24, `dhcpd` serves 10.0.0.2+).
- `CONFIG_NET_IPFORWARD=y`, `CONFIG_NET_NAT=y` (NAT44 full cone),
`iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE`.
- Relevant net tuning: `CONFIG_NET_ETH_PKTSIZE=1514`,
`CONFIG_IOB_NBUFFERS=72`, `CONFIG_NET_TCP_WINDOW_SCALE=y`,
`CONFIG_NET_TCP_SELECTIVE_ACK=y`, `CONFIG_NET_TCP_CC_NEWRENO=y`.
Reproduce:
1. Associate a phone/laptop with the open `espproxy` AP (gets 10.0.0.x).
2. Run a speedtest (simultaneous up+down TCP through the NAT).
3. After ~10-15 s Wi-Fi "drops": re-association hangs on DHCP
(no OFFER). The board itself is alive on the USB console, no panic.
Observed state after the wedge (`ps`):
```text
6 0 0 100 RR Kthread - Waiting Mutex:10 ... netdev-wlan0
10 0 0 100 RR Kthread - Waiting Mutex:6 ... netdev-wlan1
```
`dhcpd`, `telnetd`, `wifi` tasks all look normal. `ifconfig` hangs and
never returns. Ping fails on BOTH interfaces (STA .241 and AP 10.0.0.1).
`free` shows `used` +9K vs idle with a `maxused` spike (146K vs 82K
at idle). Heap timeline sampled every 4 s during the test:
T+0-8 idle (used 77664), T+12-20 climb 80592/82872/109936,
T+24 both workers wedged, heap frozen at 83928.
Why this is a cross-deadlock (ABBA), not exhaustion:
- In `ps` output the `Mutex:N` suffix is the holder **TID**
(`NXSEM_MHOLDER`, see `sched/sched/sched_get_stateinfo.c`).
- TID 6 is `netdev-wlan0` itself and TID 10 is `netdev-wlan1` itself,
so each RX worker holds the mutex the other one is waiting for.
- IOB exhaustion would show semaphore waits, not mutex waits, and
would recover when the flood stops; this never recovers.
Trigger condition is duplex: half-duplex iperf (either direction,
including a 40 Mbit/s UDP blast) never wedges the board; simultaneous
up+down (speedtest) wedges it in 10-15 s every time.
What I already ruled out:
- The forward path (`net/ipforward/ipv4_forward.c`) takes locks in one
consistent order (`d_lock` of ingress dev, then global `g_nat_lock`);
no second device lock is taken there, so the classic ABBA is not
visible statically — the peer lock must be taken on a path I have
not identified (TX-notify/poll on the egress device is a candidate).
- #5973 (`TCP_WRITE_BUFFERS` + IOB deadlock) is a different mechanism
(semaphore waits; fix #8130 from 2023 is in this tree) and our
display shows mutex waits with cross TIDs.
- No existing issue covers duplex NAT-forward ABBA between two netdev
workers (searched `deadlock`, `NAT`, `ipforward`, `softap forward`).
Related context: #15446 (lock-order detection feature request),
#17358 (IOB throttle under RX flood), #20519 (L2 bridge; cross-device
locking under active design discussion there).
I can provide `System.map`/opening a JTAG session on the wedged board
(`halt` + netdev worker stacks + mutex contents) if that helps
pinpoint the exact mutex pair — just say what to dump.
### On which OS does this issue occur?
[OS: Linux]
### What is the version of your OS?
Ubuntu 26.04.1 LTS (x86_64, build host; target console via USB CDC)
### NuttX Version
master 4bc1ab2fd7 (2026-10-12)
### Issue Architecture
[Arch: risc-v]
### Issue Area
[Area: Networking]
### Host information
OS_VERSION[] = "Ubuntu 26.04.1 LTS Linux user-VirtualBox
7.0.0-38-generic #38-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 4 09:10:14 UTC
2026 x86_64"
Toolchain: riscv32-esp-elf-gcc (crosstool-NG esp-16.1.0_20260609) 16.1.0
Esptool: 5.5.0 (pipx install esptool)
Target: ESP32-C3 QFN32 rev v0.4, 4MB XMC flash, MAC d8:3b:da:34:b0:84
Board: esp32c3-supermini, defconfig: see Description
### Verification
- [x] I have verified before submitting the report.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
