Hi Fotis, Nice, seems like you are narrowing down the numbers of variables, it will make easier to let us to replicate the issue when you find it.
You also can use coloration and canaries to help with memory issues. BR, Alan On 8/12/22, Fotis Panagiotopoulos <f.j.pa...@gmail.com> wrote: > Hi Alan, > > I am trying hard to reproduce the issue reliably, but I haven't been able > to do so yet. > > I noticed that when I disable CONFIG_NET_TCP_WRITE_BUFFERS, the problem > does not disappear, rather it changes form. > Now I occasionally get a failed assertion in wdog/wd_cancel.c line 95. > > I have to mention that everything in my system is commented out. > Currently the only thing working is the network thread that opens the TCP > connection, nothing else. > I have disabled all of my usage of the workers, all signals etc. > I verify that when the fault occurs, this thread is not interrupted by > anything (using Segger SystemView). > It looks like a scheduling issue is unlikely. > > I also increased the stacks more, and I added padding to the very few > malloc's that I use. > > --- > > At this moment I observe something very interesting. > I am calling netlib_ifdown(), which causes the attached stack trace. > > So: > 1. netdev_ifdown() calls devif_dev_event() with the argument pvconn set > explicitly to NULL. > 2. devif_dev_event() eventually calls tcp_close_eventhandler() > 3. tcp_close_eventhandler() assumes that conn is NOT NULL. Which causes the > crash. > > This is wrong, but I don't have the understanding of it yet. > Shall there be a check for a NULL conn? > Or maybe tcp_close_eventhandler() is wrong to be in the cb's list in the > first place? > Or tcp_close_eventhandler() should be tolerant to a NULL conn argument? > > > > > > On Thu, Aug 11, 2022 at 12:05 PM Alan Carvalho de Assis <acas...@gmail.com> > wrote: > >> Hi Fotis, >> >> Are you in sync with mainline? >> >> If you can create a host application to induce the issue will be >> easier for us to test. >> >> BR, >> >> Alan >> >> On 8/9/22, Fotis Panagiotopoulos <f.j.pa...@gmail.com> wrote: >> > Hello, >> > >> > still trying to make the network work reliably. >> > After fixing another issue of my application, I hit another problem. >> > >> > The following sequence causes NuttX to crash: >> > >> > 1. My application is creating a TCP socket and communicates with a >> server. >> > 2. At one point the server stops responding (unrelated to NuttX / >> > network >> > issue). >> > 3. The application detects the timeout, and calls close() on the >> > socket. >> > 4. A new socket is created, and it is connected to the server. >> > 5. At this point, the server decides to send a FIN message for the >> previous >> > connection. >> > 6. I get a failed assertion in devif_callback.c at line 85. >> > >> > Note that I haven't managed to manually reproduce this issue. >> > No matter what I do manually, everything seems to be working correctly. >> > I just have to wait for it to happen. >> > It seems that it is only triggered if a FIN arrives **after** a SYN. >> > >> > I am sure that this is only happening with CONFIG_NET_TCP_WRITE_BUFFERS >> > enabled. >> > I have no problems without buffering. >> > >> > The assertion seems right to fire. >> > When a FIN is received for a closed connection, the same callback is >> free'd >> > both by tcp_lost_connection() and later on by tcp_close_eventhandler(). >> > All these are happening within the same execution of tcp_input(). >> > >> > Any ideas? >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > On Tue, Jul 26, 2022 at 3:44 PM Sebastien Lorquet >> > <sebast...@lorquet.fr> >> > wrote: >> > >> >> Hi, >> >> >> >> good find but >> >> >> >> -I dont think any usual application tinkers with PHY regs during its >> >> lifetime except the ethernet monitor >> >> >> >> -the fix is certainly a lock somewhere but global or fine grained I >> >> dont >> >> know. >> >> >> >> Not all calls need to be locked, eg the one that returns the PHY >> >> address. Probaby not needed by default, but a PHY access lock would >> >> prevent any issue you describe. >> >> >> >> I will wait for people with more expertise about this. >> >> >> >> Just a note, dont forget that not all PHY have an interrupt, the one >> >> on >> >> the nucleo stm32h743zi[2] board does not have one. >> >> >> >> Sebastien >> >> >> >> Le 26/07/2022 à 11:05, Fotis Panagiotopoulos a écrit : >> >> > Hello, >> >> > >> >> > I have eventually found 2 issues regarding networking in my >> >> > application. >> >> > I would like to discuss the first one. >> >> > >> >> > >> >> > My code contains something like this: >> >> > >> >> > int sd = socket(AF_INET, SOCK_DGRAM, 0); >> >> > >> >> > struct ifreq ifr; >> >> > memset(&ifr, 0, sizeof(struct ifreq)); >> >> > strncpy(ifr.ifr_name, CONFIG_NETIF_DEV_NAME, IFNAMSIZ); >> >> > ifr.ifr_mii_phy_id = CONFIG_STM32_PHYADDR; >> >> > ifr.ifr_mii_reg_num = MII_LAN8720_SECR; >> >> > ifr.ifr_mii_val_out = 0; >> >> > ioctl(sd, SIOCGMIIREG, (unsigned long)&ifr); >> >> > >> >> > // Do stuff with ifr.ifr_mii_val_out. >> >> > >> >> > close(sd); >> >> > >> >> > I realized that this type of ioctl will directly access the >> >> > hardware, >> >> > without any locking. >> >> > That is, if any other task needs to use the PHY in any other way, it >> >> > will >> >> > eventually corrupt its register data. >> >> > >> >> > >> >> > Two questions on this: >> >> > 1. Is there any good reason for this? >> >> > 2. What is the best way to fix it? Shall I add a driver level lock, >> >> > or >> >> > should net_lock() be used in any higher layer? >> >> > >> >> > >> >> > >> >> > On Tue, Jul 19, 2022 at 10:30 PM Fotis Panagiotopoulos < >> >> f.j.pa...@gmail.com> >> >> > wrote: >> >> > >> >> >> Hello, >> >> >> >> >> >>> We have deployed hundreds of boards with stm32f427 and ethernet, >> they >> >> >>> have all been working reliably for months without stopping, we >> >> >>> know >> >> >>> it >> >> >>> because they critically depend on network functionality and we >> >> >>> have >> >> >>> reports if a card becomes unreachable. None has so far outside of >> >> >>> dedicated tests. >> >> >>> So I believe that there is no obvious hard bug in these drivers. >> >> >> Good to hear that! >> >> >> Although, I may be using a feature or protocol that you are not. >> >> >> Of course, I don't believe that NuttX is broken per se, but a minor >> >> >> bug >> >> >> may lurk somewhere... >> >> >> >> >> >> >> >> >>> I have seen that when I enable the network debugging features, it >> >> >>> seems >> >> >> to >> >> >>> hit an assertion failure before getting to nsh prompt at startup. >> >> >>> This >> >> >> was >> >> >>> on a quite recent master. I haven't had a chance to diagnose this >> >> >> further. >> >> >>> Have you tried enabling these and if so, do they work? >> >> >> If you refer to CONFIG_DEBUG_NET, then yes I have enabled it and it >> >> works. >> >> >> I have some devices under test, waiting to reproduce the issue to >> >> >> see >> >> >> if >> >> >> this option provides any useful information. >> >> >> >> >> >> >> >> >>> Also, out of curiosity, have you tried running ostest on your >> >> >>> board? >> >> >> I just tried. >> >> >> It passed all the tests. >> >> >> >> >> >> On Tue, Jul 19, 2022 at 4:44 PM Sebastien Lorquet >> >> >> <sebast...@lorquet.fr >> >> > >> >> >> wrote: >> >> >> >> >> >>> Hi, >> >> >>> >> >> >>> We have deployed hundreds of boards with stm32f427 and ethernet, >> they >> >> >>> have all been working reliably for months without stopping, we >> >> >>> know >> >> >>> it >> >> >>> because they critically depend on network functionality and we >> >> >>> have >> >> >>> reports if a card becomes unreachable. None has so far outside of >> >> >>> dedicated tests. >> >> >>> >> >> >>> So I believe that there is no obvious hard bug in these drivers. >> >> >>> >> >> >>> Most certainly a build option on your particular config. debug is >> >> >>> a >> >> >>> possible issue, thread problems is another possibility. >> >> >>> >> >> >>> Sebastien >> >> >>> >> >> >>> >> >> >>> On 7/19/22 13:47, Fotis Panagiotopoulos wrote: >> >> >>>> Hello! >> >> >>>> >> >> >>>> I am using Ethernet on an STM32F427 target, but I am facing some >> >> issues. >> >> >>>> >> >> >>>> Initially the device works correctly. After some hours of >> continuous >> >> >>>> operation I completely lose all network communications. >> >> >>>> Trying to troubleshoot the issue, I enabled assertions and >> >> >>>> various >> >> other >> >> >>>> debug features. >> >> >>>> >> >> >>>> Again the device works correctly for some hours, and then I get a >> >> failed >> >> >>>> assertion at stm32_eth.c, line 1372: >> >> >>>> >> >> >>>> DEBUGASSERT(dev->d_len == 0 && dev->d_buf == NULL); >> >> >>>> >> >> >>>> No other errors are reported (e.g. stack overflows etc). >> >> >>>> >> >> >>>> >> >> >>>> I have observed that this issue usually manifests itself when >> >> >>>> there >> >> >>>> is >> >> >>>> insufficient stack on a task. >> >> >>>> But in my case, all tasks have oversized stacks. Typically they >> >> >>>> do >> >> >>>> not >> >> >>>> exceed 50% utilization. >> >> >>>> I have plenty of room available in the heap too (> 100kB). >> >> >>>> >> >> >>>> Regarding the rest of the firmware, I cannot see any other >> >> misbehaviour >> >> >>> or >> >> >>>> problem. >> >> >>>> I haven't ever seen any other unexplained problem, assertion >> >> >>>> fail, >> >> >>>> hard-fault etc. >> >> >>>> The application code passes all of our tests. >> >> >>>> In fact, even when this issue happens, although I lose network >> >> >>>> connectivity, the rest of the system works perfectly. >> >> >>>> >> >> >>>> Please note that I have checked the contents of dev->d_len and >> >> >>> dev->d_buf, >> >> >>>> and they seem to contain valid data. >> >> >>>> The address lies within the normal address space of the MCU, and >> the >> >> >>> size >> >> >>>> is sane. >> >> >>>> So it doesn't look like any kind of memory corruption. >> >> >>>> >> >> >>>> >> >> >>>> At this point I believe that this is an actual bug either on the >> >> >>>> STM32 >> >> >>> MAC >> >> >>>> driver, or at the TCP/IP stack itself. >> >> >>>> I had a look at the driver code, but I didn't see anything >> >> >>>> suspicious. >> >> >>>> >> >> >>>> >> >> >>>> Has anyone observed the same issue before? >> >> >>>> Can it be affected in any way with my configuration? >> >> >>>> Or maybe, do you have any recommendations on what to test next? >> >> >>>> >> >> >>>> >> >> >>>> Thank you! >> >> >>>> >> >> >> > >> >
