Re: debugging user app crashes
On Thu, Jan 11, 2024 at 10:35 AM Gregory Nutt wrote: > Everything you need to know is in that referenced wiki page: > > https://cwiki.apache.org/confluence/display/NUTTX/Debugging+ELF+Loadable+Modules > > I don't know anything about QEMU or RISC-V. But assuming that NSH is > your init application... > > * NSH is not in memory when you start the OS; it resides in your init > file. The OS will automatically start NSH as the init application > (if properly configured). > * Since there is no gdbserver or ptrace, there is not elegant way to > break into the program. Some kludgey ways to do that are described > in the referenced wiki page. > * Then just follow the instructions in the wiki page: > o Start the debugger > o Load the symbol table of the NSH application. You probably > don't need as load offset as described in the wiki page. > o Set a breakpoint at nsh_parse, and > o Continue > > I have nothing to add to that. Note to self: Check whether above-linked wiki page has been migrated to Documentation in the repository and, if not, migrate it... I likely won't be able to work on that until next week, so if someone else wants to do it, feel free. :-) Cheers Nathan
Re: debugging user app crashes
Everything you need to know is in that referenced wiki page: https://cwiki.apache.org/confluence/display/NUTTX/Debugging+ELF+Loadable+Modules I don't know anything about QEMU or RISC-V. But assuming that NSH is your init application... * NSH is not in memory when you start the OS; it resides in your init file. The OS will automatically start NSH as the init application (if properly configured). * Since there is no gdbserver or ptrace, there is not elegant way to break into the program. Some kludgey ways to do that are described in the referenced wiki page. * Then just follow the instructions in the wiki page: o Start the debugger o Load the symbol table of the NSH application. You probably don't need as load offset as described in the wiki page. o Set a breakpoint at nsh_parse, and o Continue I have nothing to add to that. On 1/10/2024 7:36 PM, yfliu2008 wrote: Spudaneco, Can you share more details about how to break in "nsh_parse()" with "rv-virt/knsh64"? Regards, yf Original From:"spudaneco"
Re:Re: debugging user app crashes
You should be able to set breakpoints properly, but only after the correct MMU mapping is in place. Original message From: yfliu2008 Date: 1/9/24 8:06 PM (GMT-06:00) To: dev Subject: Re:Re: debugging user app crashes Yes, in kernel mode each user space program has similar but isolated virtual address spaces. So it seems that we can't set break points to them like FLAT or PROTECTED build.Not sure if we need some sort of debug agent in KERNEL build so that it can help debugging user space tasks. When use of kernel build araises, this need will increase I guess.Regards,yf Original From:"Alan C. Assis"< acas...@gmail.com ;Date:2024/1/10 9:47To:"dev"< dev@nuttx.apache.org ;Subject:Re: debugging user app crashesYes! For kernel mode each program has its own main() implementation, notthe definition.BR,AlanOn Tue, Jan 9, 2024 at 10:34 PM Gregory Nutt wrote: On 1/9/2024 7:28 PM, Alan C. Assis wrote: Hi Yf, Just look at your System.map and you will see that all those main() became _main. It isn't possible to have a unique main() symbol, otherwise we will have a duplicated symbol, also even if the compiler accepted that, how could NuttX know you want to call the main from Hello instead of the main from Alarm? BR, Alan This is not true in the kernel build mode. All applications run at exactly the same virtual address and the entry point is main() for all of them. In the kernel build, all applications are loadable modules. None of the symbols in the loadable modules will appear in System.map since each application module is separately compiled and linked.
Re: debugging user app crashes
On 1/9/2024 8:06 PM, yfliu2008 wrote: Yes, in kernel mode each user space program has similar but isolated virtual address spaces. So it seems that we can't set break points to them like FLAT or PROTECTED build. Not sure if we need some sort of debug agent in KERNEL build so that it can help debugging user space tasks. When use of kernel build araises, this need will increase I guess. Regards, yf People have periodically suggested implementing ptrace for NuttX. For example, https://github.com/apache/nuttx/issues/2028
Re: debugging user app crashes
Yes! For kernel mode each program has its own main() implementation, not the definition. BR, Alan On Tue, Jan 9, 2024 at 10:34 PM Gregory Nutt wrote: > > On 1/9/2024 7:28 PM, Alan C. Assis wrote: > > Hi Yf, > > Just look at your System.map and you will see that all those main() > became > > _main. > > > > It isn't possible to have a unique main() symbol, otherwise we will have > a > > duplicated symbol, also even if the compiler accepted that, how could > NuttX > > know you want to call the main from Hello instead of the main from Alarm? > > > > BR, > > > > Alan > This is not true in the kernel build mode. All applications run at > exactly the same virtual address and the entry point is main() for all > of them. In the kernel build, all applications are loadable modules. > None of the symbols in the loadable modules will appear in System.map > since each application module is separately compiled and linked.
Re: debugging user app crashes
On 1/9/2024 7:28 PM, Alan C. Assis wrote: Hi Yf, Just look at your System.map and you will see that all those main() became _main. It isn't possible to have a unique main() symbol, otherwise we will have a duplicated symbol, also even if the compiler accepted that, how could NuttX know you want to call the main from Hello instead of the main from Alarm? BR, Alan This is not true in the kernel build mode. All applications run at exactly the same virtual address and the entry point is main() for all of them. In the kernel build, all applications are loadable modules. None of the symbols in the loadable modules will appear in System.map since each application module is separately compiled and linked.
Re: debugging user app crashes
This is old and does not specifically address kernel mode, but I think most of the debug tips do apply in kernel mode too. The load address is a constant for kernel ELF modules so at least that part doesn't apply. But the rest probably does. On 1/9/2024 7:20 PM, yfliu2008 wrote: Alan, Thanks a lot for your warm replies. These apps are built with kernel mode export libraries, their entrances are all "main" as they each run in their user address spaces. So when gdb connected with QEMU, command "break qemu_rv_start" can work but "break main" simply got function not defined even though I did "load ../apps/bin/init". It seems to me that current gdb session is bound to the kernel ELF, not to the apps ELF. Regards, yf Original From:"Alan C. Assis"< acas...@gmail.com ; Date:2024/1/10 8:11 To:"dev"< dev@nuttx.apache.org ; Subject:Re: debugging user app crashes Hi Yf Liu, I don't know if this minidump server works on 64-bit and with QEMU, but about the other question how to setup breakpoint in the NSH user space program, just set the break point to the nsh_main if you want to debug the NSH itself or set it to program_main to put the breakpoint to some user program called from NSH. For example: setting it to hello_main will cause the GDB to stop at hello_main when you type "nsh hello" and press enter. BR, Alan On Mon, Jan 8, 2024 at 9:13 PM yfliu2008 wrote: Dear community, I have NSH command crashes within qemu-system-riscv64 using a testing rv-virt/knsh configuration, I wonder what tools can I use to find the crash stack? I have feed console crash.log to minidumpserver.py as per this guide: https://nuttx.apache.org/docs/12.3.0/guides/minidumpserver.htm,; but it seems the tool might not support 64bit yet? I've tried gdb connection with qemu-system-riscv64 as per this guide: https://nuttx.apache.org/docs/12.3.0/guides/gdbwithpython.html, but I don't know how to set a break point in the user space NSH program. Can anyone give me a pointer? Regards, yf
Re: Re: debugging user app crashes
Hi Yf, Just look at your System.map and you will see that all those main() became _main. It isn't possible to have a unique main() symbol, otherwise we will have a duplicated symbol, also even if the compiler accepted that, how could NuttX know you want to call the main from Hello instead of the main from Alarm? BR, Alan On Tue, Jan 9, 2024 at 10:20 PM yfliu2008 wrote: > Alan, > > > Thanks a lot for your warm replies. > > > These apps are built with kernel mode export libraries, their entrances > are all "main" as they each run in their user address spaces. > > > So when gdb connected with QEMU, command "break qemu_rv_start" > can work but "break main" simply got function not defined even though I did > "load ../apps/bin/init". It seems to me that current gdb session is bound > to the kernel ELF, not to the apps ELF. > > > > > Regards, > yf > > > > > > Original > > > > From:"Alan C. Assis"< acas...@gmail.com ; > > Date:2024/1/10 8:11 > > To:"dev"< dev@nuttx.apache.org ; > > Subject:Re: debugging user app crashes > > > Hi Yf Liu, > I don't know if this minidump server works on 64-bit and with QEMU, but > about the other question how to setup breakpoint in the NSH user space > program, just set the break point to the nsh_main if you want to debug the > NSH itself or set it to program_main to put the breakpoint to some user > program called from NSH. For example: setting it to hello_main will cause > the GDB to stop at hello_main when you type "nsh hello" and press > enter. > > BR, > > Alan > > On Mon, Jan 8, 2024 at 9:13 PM yfliu2008 wrote: > > Dear community, > > > > > I have NSH command crashes within qemu-system-riscv64 using a testing > rv-virt/knsh configuration, I wonder what tools can I use to find the > crash > stack? > > > > > I have feed console crash.log to minidumpserver.py as per this > guide: > https://nuttx.apache.org/docs/12.3.0/guides/minidumpserver.htm,; > but > it seems the tool might not support 64bit yet? > > > > > I've tried gdb connection with qemu-system-riscv64 as per this > guide: https://nuttx.apache.org/docs/12.3.0/guides/gdbwithpython.html > , > but I don't know how to set a break point in the user space NSH > program. > > > > > Can anyone give me a pointer? > > > > > Regards, > > yf
Re: debugging user app crashes
Hi Yf Liu, I don't know if this minidump server works on 64-bit and with QEMU, but about the other question how to setup breakpoint in the NSH user space program, just set the break point to the nsh_main if you want to debug the NSH itself or set it to program_main to put the breakpoint to some user program called from NSH. For example: setting it to hello_main will cause the GDB to stop at hello_main when you type "nsh> hello" and press enter. BR, Alan On Mon, Jan 8, 2024 at 9:13 PM yfliu2008 wrote: > Dear community, > > > > > I have NSH command crashes within qemu-system-riscv64 using a testing > rv-virt/knsh configuration, I wonder what tools can I use to find the crash > stack? > > > > > I have feed console crash.log to minidumpserver.py as per this guide: > https://nuttx.apache.org/docs/12.3.0/guides/minidumpserver.htm,; but > it seems the tool might not support 64bit yet? > > > > > I've tried gdb connection with qemu-system-riscv64 as per this > guide: https://nuttx.apache.org/docs/12.3.0/guides/gdbwithpython.html, > but I don't know how to set a break point in the user space NSH program. > > > > > Can anyone give me a pointer? > > > > > Regards, > > yf
