Re: Symbol debugging support for kernel modules in crash dumps

2020-05-02 Thread Christos Zoulas 
In article <20200501233413.291a717f...@rebar.astron.com>,
Christos Zoulas  wrote:
>
>Hi,
>
>I just added symbol debugging support for modules in kernel dumps.
>Things are not perfect because of what I call "current thread
>confusion" in the kvm target, but as you see in the following
>session it works just fine if you follow the right steps. First of
>all you need a build from HEAD that has the capability to build
>.debug files for kernel modules.  Once that's done, you are all
>set; see how it works (comments prefixed by )

I fixed the "current thread confusion" by setting the current
thread and reloading the symbol file in gdb, this works as
expected:

$ gdb netbsd.gdb
(gdb) target kvm netbsd.666.core
(gdb) source /usr/src/sys/gdbscripts/modload
(gdb) modload
(gdb) where

christos

Re: Symbol debugging support for kernel modules in crash dumps

2020-05-01 Thread Paul Goyette 

Very cool!

On Fri, 1 May 2020, Christos Zoulas wrote:



Hi,

I just added symbol debugging support for modules in kernel dumps.
Things are not perfect because of what I call "current thread
confusion" in the kvm target, but as you see in the following
session it works just fine if you follow the right steps. First of
all you need a build from HEAD that has the capability to build
.debug files for kernel modules.  Once that's done, you are all
set; see how it works (comments prefixed by )

Enjoy,

christos

$ gdb /usr/src/sys/arch/amd64/compile/QUASAR/netbsd.gdb
GNU gdb (GDB) 8.3
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64--netbsd".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
   .

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/src/sys/arch/amd64/compile/QUASAR/netbsd.gdb...
(gdb) target kvm netbsd.22.core
0x80224375 in cpu_reboot (howto=howto@entry=260,
   bootstr=bootstr@entry=0x0) at ../../../../arch/amd64/amd64/machdep.c:718
warning: Source file is more recent than executable.
718 if (s != IPL_NONE)

 Ok we got a stacktrace here, but we don't have a current thread...
 So we set it...

(gdb) info thread
 Id   Target Id Frame
* 2.1   0x80224375 in cpu_reboot (
   howto=howto@entry=260, bootstr=bootstr@entry=0x0)
   at ../../../../arch/amd64/amd64/machdep.c:718

No selected thread.  See `help thread'.
(gdb) thread 2.1

[Switching to thread 2.1 ()]
#0  0x80224375 in ?? ()

 Note that here we lost all symbol table access when we switched threads
 let's load it again..

(gdb) add-symbol-file /usr/src/sys/arch/amd64/compile/QUASAR/netbsd.gdb
add symbol table from file "/usr/src/sys/arch/amd64/compile/QUASAR/netbsd.gdb"
(y or n) y
Reading symbols from /usr/src/sys/arch/amd64/compile/QUASAR/netbsd.gdb...

 OK, lets load our modules

(gdb) source /usr/src/sys/gdbscripts/modload
(gdb) modload
add symbol table from file "/stand/amd64/9.99.59/modules/ping/ping.kmod" at
   .text_addr = 0x8266e000
   .data_addr = 0x8266b000
   .rodata_addr = 0x8266c000
add symbol table from file 
"/stand/amd64/9.99.59/modules/nfsserver/nfsserver.kmod" at
   .text_addr = 0x82a64000
   .data_addr = 0x82669000
   .rodata_addr = 0x8298e000
add symbol table from file 
"/stand/amd64/9.99.59/modules/npf_ext_log/npf_ext_log.kmod" at
   .text_addr = 0x82668000
   .data_addr = 0x82667000
   .rodata_addr = 0x82969000
add symbol table from file 
"/stand/amd64/9.99.59/modules/npf_alg_icmp/npf_alg_icmp.kmod" at
   .text_addr = 0x82666000
   .data_addr = 0x82665000
   .rodata_addr = 0x82952000
add symbol table from file "/stand/amd64/9.99.59/modules/bpfjit/bpfjit.kmod" at
   .text_addr = 0x82661000
   .data_addr = 0x0
   .rodata_addr = 0x828dd000
add symbol table from file "/stand/amd64/9.99.59/modules/sljit/sljit.kmod" at
   .text_addr = 0x82945000
   .data_addr = 0x82664000
   .rodata_addr = 0x828f9000
add symbol table from file 
"/stand/amd64/9.99.59/modules/if_npflog/if_npflog.kmod" at
   .text_addr = 0x8266
   .data_addr = 0x8265f000
   .rodata_addr = 0x828ca000
add symbol table from file "/stand/amd64/9.99.59/modules/npf/npf.kmod" at
   .text_addr = 0x82648000
   .data_addr = 0x82647000
   .rodata_addr = 0x826d6000
add symbol table from file "/stand/amd64/9.99.59/modules/bpf/bpf.kmod" at
   .text_addr = 0x82622000
   .data_addr = 0x82621000
   .rodata_addr = 0x826a3000
add symbol table from file 
"/stand/amd64/9.99.59/modules/bpf_filter/bpf_filter.kmod" at
   .text_addr = 0x8263c000
   .data_addr = 0x0
   .rodata_addr = 0x82627000
add symbol table from file 
"/stand/amd64/9.99.59/modules/scsiverbose/scsiverbose.kmod" at
   .text_addr = 0x826a2000
   .data_addr = 0x82686000
   .rodata_addr = 0x82687000
add symbol table from file 
"/stand/amd64/9.99.59/modules/usbverbose/usbverbose.kmod" at
   .text_addr = 0x82685000
   .data_addr = 0x8267
   .rodata_addr = 0x82671000
add symbol table from file 
"/stand/amd64/9.99.59/modules/miiverbose/miiverbose.kmod" at
   .text_addr =

Re: Symbol debugging support for kernel modules in crash dumps

2020-05-01 Thread David Young 
Fantastic! Thanks.

Dave

Spilling kerrectud by iPhone

> On May 1, 2020, at 6:34 PM, Christos Zoulas  wrote:
> 
> 
> Hi,
> 
> I just added symbol debugging support for modules in kernel dumps.
> Things are not perfect because of what I call "current thread
> confusion" in the kvm target, but as you see in the following
> session it works just fine if you follow the right steps. First of
> all you need a build from HEAD that has the capability to build
> .debug files for kernel modules.  Once that's done, you are all
> set; see how it works (comments prefixed by )
> 
> Enjoy,
> 
> christos
> 
> $ gdb /usr/src/sys/arch/amd64/compile/QUASAR/netbsd.gdb
> GNU gdb (GDB) 8.3
> Copyright (C) 2019 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Type "show copying" and "show warranty" for details.
> This GDB was configured as "x86_64--netbsd".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> .
> Find the GDB manual and other documentation resources online at:
>.
> 
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from /usr/src/sys/arch/amd64/compile/QUASAR/netbsd.gdb...
> (gdb) target kvm netbsd.22.core
> 0x80224375 in cpu_reboot (howto=howto@entry=260, 
>bootstr=bootstr@entry=0x0) at ../../../../arch/amd64/amd64/machdep.c:718
> warning: Source file is more recent than executable.
> 718 if (s != IPL_NONE)
> 
>  Ok we got a stacktrace here, but we don't have a current thread...
>  So we set it...
> 
> (gdb) info thread
>  Id   Target Id Frame 
> * 2.1   0x80224375 in cpu_reboot (
>howto=howto@entry=260, bootstr=bootstr@entry=0x0)
>at ../../../../arch/amd64/amd64/machdep.c:718
> 
> No selected thread.  See `help thread'.
> (gdb) thread 2.1
> 
> [Switching to thread 2.1 ()]
> #0  0x80224375 in ?? ()
> 
>  Note that here we lost all symbol table access when we switched threads
>  let's load it again..
> 
> (gdb) add-symbol-file /usr/src/sys/arch/amd64/compile/QUASAR/netbsd.gdb
> add symbol table from file "/usr/src/sys/arch/amd64/compile/QUASAR/netbsd.gdb"
> (y or n) y
> Reading symbols from /usr/src/sys/arch/amd64/compile/QUASAR/netbsd.gdb...
> 
>  OK, lets load our modules
> 
> (gdb) source /usr/src/sys/gdbscripts/modload 
> (gdb) modload
> add symbol table from file "/stand/amd64/9.99.59/modules/ping/ping.kmod" at
>.text_addr = 0x8266e000
>.data_addr = 0x8266b000
>.rodata_addr = 0x8266c000
> add symbol table from file 
> "/stand/amd64/9.99.59/modules/nfsserver/nfsserver.kmod" at
>.text_addr = 0x82a64000
>.data_addr = 0x82669000
>.rodata_addr = 0x8298e000
> add symbol table from file 
> "/stand/amd64/9.99.59/modules/npf_ext_log/npf_ext_log.kmod" at
>.text_addr = 0x82668000
>.data_addr = 0x82667000
>.rodata_addr = 0x82969000
> add symbol table from file 
> "/stand/amd64/9.99.59/modules/npf_alg_icmp/npf_alg_icmp.kmod" at
>.text_addr = 0x82666000
>.data_addr = 0x82665000
>.rodata_addr = 0x82952000
> add symbol table from file "/stand/amd64/9.99.59/modules/bpfjit/bpfjit.kmod" 
> at
>.text_addr = 0x82661000
>.data_addr = 0x0
>.rodata_addr = 0x828dd000
> add symbol table from file "/stand/amd64/9.99.59/modules/sljit/sljit.kmod" at
>.text_addr = 0x82945000
>.data_addr = 0x82664000
>.rodata_addr = 0x828f9000
> add symbol table from file 
> "/stand/amd64/9.99.59/modules/if_npflog/if_npflog.kmod" at
>.text_addr = 0x8266
>.data_addr = 0x8265f000
>.rodata_addr = 0x828ca000
> add symbol table from file "/stand/amd64/9.99.59/modules/npf/npf.kmod" at
>.text_addr = 0x82648000
>.data_addr = 0x82647000
>.rodata_addr = 0x826d6000
> add symbol table from file "/stand/amd64/9.99.59/modules/bpf/bpf.kmod" at
>.text_addr = 0x82622000
>.data_addr = 0x82621000
>.rodata_addr = 0x826a3000
> add symbol table from file 
> "/stand/amd64/9.99.59/modules/bpf_filter/bpf_filter.kmod" at
>.text_addr = 0x8263c000
>.data_addr = 0x0
>.rodata_addr = 0x82627000
> add symbol table from file 
> "/stand/amd64/9.99.59/modules/scsiverbose/scsiverbose.kmod" at
>.text_addr = 0x826a2000
>.data_addr = 0x82686000
>.rodata_addr = 0x82687000
> add symbol table from file 
>