Re: GPF on boot with devmatch

2021-01-13 Thread Rebecca Cran 
On 10/12/20 12:13 PM, Warner Losh wrote:
> Xin Li's traceback lead to code I just rewrote in current, while this code
> leads to code that's been there for a long time and hasn't been MFC'd. This
> suggests that Xin Li's backtrace isn't to be trusted, or there's two issues
> at play. Both are plausible. I've fixed a minor signedness bug and a
> possible one byte overflow that might have happened in the code I just
> rewrote. But I suspect this is due to something else related to how
> children are handled after we've raced. Maybe there's something special
> about how USB does things, because other buses will create the child early
> and the child list is stable. If USB's discovery code is adding something
> and is racing with devd's walking of the tree, that might explain it...  It
> would be nice if there were some way to provoke the race on a system I
> could get a core from for deeper analysis

I'm seeing this crash on 13-CURRENT main-c255937-g818390ce0ca-dirty when
running GENERIC (but not on GENERIC-NODEBUG).

I've uploaded the core dump etc. to
https://bsdio.com/freebsd/crashes/2021-01-13-devmatch/ .


-- 

Rebecca Cran


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: GPF on boot with devmatch

2020-10-12 Thread Xin Li 
On 10/12/20 11:13, Warner Losh wrote:
> 
> 
> On Mon, Oct 5, 2020 at 3:39 PM Alexander Motin  > wrote:
> 
> On 05.10.2020 17:20, Warner Losh wrote:
> > On Mon, Oct 5, 2020 at 12:36 PM Alexander Motin  
> > >> wrote:
> >
> >     I can add that we've received report about identical panic on
> FreeBSD
> >     releng/12.2 of r365436, AKA TrueNAS 12.0-RC1:
> >     https://jira.ixsystems.com/browse/NAS-107578
>  .  So it looks a) pretty
> >     rate (one report from thousands of early adopters and none in
> our lab),
> >     and b) it is in stable/12 too, not only head.
> >
> > Thanks! I'll see if I can recreate here  But we're accessing the
> > sysctl tree from devmatch to get some information, which should always
> > be OK (the fact that it isn't suggests either a bug in some driver
> > leaving bad pointers, or some race or both)...  It would be nice
> to know
> > which nodes they were, or to have a kernel panic I can look at...
> 
> All we have now in this case is a screenshot you may see in the ticket.
>  Also previously the same user on some earlier version of stable/12
> reported other very weird panics on process lock being dropped where it
> can't be in some other sysctls inside kern.proc, so if we guess those
> are related, I suspect there may be some kind of memory corruption
> happening, but have no clue where.  Unfortunately we have only textdumps
> for those.  So if Xin is able to reproduce it locally, it may be our
> best chance to debug it, at least this specific issue.
> 
> 
> That's totally weird. 
> 
> Xin Li's traceback lead to code I just rewrote in current, while this
> code leads to code that's been there for a long time and hasn't been
> MFC'd. This suggests that Xin Li's backtrace isn't to be trusted, or
> there's two issues at play. Both are plausible. I've fixed a minor
> signedness bug and a possible one byte overflow that might have happened
> in the code I just rewrote. But I suspect this is due to something else
> related to how children are handled after we've raced. Maybe there's
> something special about how USB does things, because other buses will
> create the child early and the child list is stable. If USB's discovery
> code is adding something and is racing with devd's walking of the tree,
> that might explain it...  It would be nice if there were some way to
> provoke the race on a system I could get a core from for deeper analysis

There might be some other players; I just don't have a lot of time
recently to shoot it down; the system is somewhat critical for my
internal network so I can't afford to have long downtimes (it's fine if
I have controllable downtimes, e.g. if you want me to deliberately panic
the system and get some debugging data, please feel free to ask as long
as I can continue to boot at the end of experiment :))

From what I was observing, it seems to be some kind of race condition
between the USB stack and sysctl tree; however, the race might be
delicate as I never successfully provoked the panic on my laptop, which
also runs -CURRENT).  If you want to add some instruments to the code,
please let me know and I'll get the tree patched to try to catch it.

Cheers,

> Warner
>  
> 
> -- 
> Alexander Motin
> 



OpenPGP_signature
Description: OpenPGP digital signature

Re: GPF on boot with devmatch

2020-10-12 Thread Warner Losh 
On Mon, Oct 5, 2020 at 3:39 PM Alexander Motin  wrote:

> On 05.10.2020 17:20, Warner Losh wrote:
> > On Mon, Oct 5, 2020 at 12:36 PM Alexander Motin  > > wrote:
> >
> > I can add that we've received report about identical panic on FreeBSD
> > releng/12.2 of r365436, AKA TrueNAS 12.0-RC1:
> > https://jira.ixsystems.com/browse/NAS-107578 .  So it looks a)
> pretty
> > rate (one report from thousands of early adopters and none in our
> lab),
> > and b) it is in stable/12 too, not only head.
> >
> > Thanks! I'll see if I can recreate here  But we're accessing the
> > sysctl tree from devmatch to get some information, which should always
> > be OK (the fact that it isn't suggests either a bug in some driver
> > leaving bad pointers, or some race or both)...  It would be nice to know
> > which nodes they were, or to have a kernel panic I can look at...
>
> All we have now in this case is a screenshot you may see in the ticket.
>  Also previously the same user on some earlier version of stable/12
> reported other very weird panics on process lock being dropped where it
> can't be in some other sysctls inside kern.proc, so if we guess those
> are related, I suspect there may be some kind of memory corruption
> happening, but have no clue where.  Unfortunately we have only textdumps
> for those.  So if Xin is able to reproduce it locally, it may be our
> best chance to debug it, at least this specific issue.
>

That's totally weird.

Xin Li's traceback lead to code I just rewrote in current, while this code
leads to code that's been there for a long time and hasn't been MFC'd. This
suggests that Xin Li's backtrace isn't to be trusted, or there's two issues
at play. Both are plausible. I've fixed a minor signedness bug and a
possible one byte overflow that might have happened in the code I just
rewrote. But I suspect this is due to something else related to how
children are handled after we've raced. Maybe there's something special
about how USB does things, because other buses will create the child early
and the child list is stable. If USB's discovery code is adding something
and is racing with devd's walking of the tree, that might explain it...  It
would be nice if there were some way to provoke the race on a system I
could get a core from for deeper analysis

Warner


> --
> Alexander Motin
>
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: GPF on boot with devmatch

2020-10-12 Thread Alexander Motin 
On 05.10.2020 17:39, Alexander Motin wrote:
> On 05.10.2020 17:20, Warner Losh wrote:
>> On Mon, Oct 5, 2020 at 12:36 PM Alexander Motin > > wrote:
>>
>> I can add that we've received report about identical panic on FreeBSD
>> releng/12.2 of r365436, AKA TrueNAS 12.0-RC1:
>> https://jira.ixsystems.com/browse/NAS-107578 .  So it looks a) pretty
>> rate (one report from thousands of early adopters and none in our lab),
>> and b) it is in stable/12 too, not only head.
>>
>> Thanks! I'll see if I can recreate here  But we're accessing the
>> sysctl tree from devmatch to get some information, which should always
>> be OK (the fact that it isn't suggests either a bug in some driver
>> leaving bad pointers, or some race or both)...  It would be nice to know
>> which nodes they were, or to have a kernel panic I can look at...
> 
> All we have now in this case is a screenshot you may see in the ticket.
>  Also previously the same user on some earlier version of stable/12
> reported other very weird panics on process lock being dropped where it
> can't be in some other sysctls inside kern.proc, so if we guess those
> are related, I suspect there may be some kind of memory corruption
> happening, but have no clue where.  Unfortunately we have only textdumps
> for those.  So if Xin is able to reproduce it locally, it may be our
> best chance to debug it, at least this specific issue.

The TrueNAS user has confirmed that preloading the modules as Xin did
workarounds the problem for him also.

-- 
Alexander Motin
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: GPF on boot with devmatch

2020-10-05 Thread Alexander Motin 
On 05.10.2020 17:20, Warner Losh wrote:
> On Mon, Oct 5, 2020 at 12:36 PM Alexander Motin  > wrote:
> 
> I can add that we've received report about identical panic on FreeBSD
> releng/12.2 of r365436, AKA TrueNAS 12.0-RC1:
> https://jira.ixsystems.com/browse/NAS-107578 .  So it looks a) pretty
> rate (one report from thousands of early adopters and none in our lab),
> and b) it is in stable/12 too, not only head.
> 
> Thanks! I'll see if I can recreate here  But we're accessing the
> sysctl tree from devmatch to get some information, which should always
> be OK (the fact that it isn't suggests either a bug in some driver
> leaving bad pointers, or some race or both)...  It would be nice to know
> which nodes they were, or to have a kernel panic I can look at...

All we have now in this case is a screenshot you may see in the ticket.
 Also previously the same user on some earlier version of stable/12
reported other very weird panics on process lock being dropped where it
can't be in some other sysctls inside kern.proc, so if we guess those
are related, I suspect there may be some kind of memory corruption
happening, but have no clue where.  Unfortunately we have only textdumps
for those.  So if Xin is able to reproduce it locally, it may be our
best chance to debug it, at least this specific issue.

-- 
Alexander Motin
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: GPF on boot with devmatch

2020-10-05 Thread Warner Losh 
On Mon, Oct 5, 2020 at 12:36 PM Alexander Motin  wrote:

> I can add that we've received report about identical panic on FreeBSD
> releng/12.2 of r365436, AKA TrueNAS 12.0-RC1:
> https://jira.ixsystems.com/browse/NAS-107578 .  So it looks a) pretty
> rate (one report from thousands of early adopters and none in our lab),
> and b) it is in stable/12 too, not only head.
>

Thanks! I'll see if I can recreate here  But we're accessing the sysctl
tree from devmatch to get some information, which should always be OK (the
fact that it isn't suggests either a bug in some driver leaving bad
pointers, or some race or both)...  It would be nice to know which nodes
they were, or to have a kernel panic I can look at...

Warner
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: GPF on boot with devmatch

2020-10-05 Thread Alexander Motin 
I can add that we've received report about identical panic on FreeBSD
releng/12.2 of r365436, AKA TrueNAS 12.0-RC1:
https://jira.ixsystems.com/browse/NAS-107578 .  So it looks a) pretty
rate (one report from thousands of early adopters and none in our lab),
and b) it is in stable/12 too, not only head.

-- 
Alexander Motin
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: GPF on boot with devmatch

2020-10-05 Thread Warner Losh 
On Sun, Oct 4, 2020 at 11:30 PM Xin Li  wrote:

>
>
> On 10/4/20 10:13 PM, Warner Losh wrote:
> >
> >
> > On Sun, Oct 4, 2020, 11:07 PM Xin Li  > > wrote:
> >
> > Hi,
> >
> > I'm seeing this panic at boot after upgrading from r366217 to
> r366364,
> > and continues to exist for r366421 (but I haven't find out the exact
> > change that caused it).  Preloading the relevant kernel modules
> > (uhid.ko, ums.ko and wmt.ko) seems to make the kernel boot correctly.
> >
> >
> > What happens if you disable devmatch and load these modules by hand?
>
> Loading these modules from loader or kld_list will prevent this panic
> regardless if devmatch is enabled.
>
> > What happens if you load them from rc.d scripts with devmatch disabled?
>
> It seems that the devmatch was started by devd and not rc.  Disabling
> devmatch (setting devmatch_enable="NO" without loading any of these klds
> would not provoke the panic).
>

That sounds like a bug to me... I'll have to look into it. I'm mostly
asking 'what happens if you load them after boot' to see if this is a
'devmatch doing weird things' issue or a 'these drivers don't like to be
loaded so late and we need to track down why' issue.

Warner


> > Warner
> >
> > This is not reproducible on my laptop, which will load many more
> kernel
> > modules.
> >
> > ===
> > Autoloading module: uhid.ko
> > Autoloading module: wmt.ko
> >
> >
> > Fatal trap 9: general protection fault while in kernel mode
> > cpuid = 2; apic id = 04
> > instruction pointer = 0x20:0x806ad6eb
> > stack pointer   = 0x28:0xfe01850cd960
> > frame pointer   = 0x28:0xfe01850cd9e0
> > code segment= base 0x0, limit 0xf, type 0x1b
> > = DPL 0, pres 1, long 1, def32 0, gran 1
> > processor eflags= interrupt enabled, resume, IOPL = 0
> > current process = 740 (devmatch)
> > trap number = 9
> > panic: general protection fault
> > cpuid = 3
> > time = 1601866799
> > KDB: stack backtrace:
> > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> > 0xfe01850cd670
> > vpanic() at vpanic+0x182/frame 0xfe01850cd6c0
> > panic() at panic+0x43/frame 0xfe01850cd720
> > trap_fatal() at trap_fatal+0x387/frame 0xfe01850cd780
> > trap() at trap+0xa4/frame 0xfe01850cd890
> > calltrap() at calltrap+0x8/frame 0xfe01850cd890
> > --- trap 0x9, rip = 0x806ad6eb, rsp = 0xfe01850cd960,
> rbp =
> > 0xfe01850cd9e0 ---
> > sysctl_devices() at sysctl_devices+0x24b/frame 0xfe01850cd9e0
> > sysctl_root_handler_locked() at sysctl_root_handler_locked+0x9c/frame
> > 0xfe01850cda30
> > sysctl_root() at sysctl_root+0x20a/frame 0xfe01850cdab0
> > userland_sysctl() at userland_sysctl+0x17d/frame 0xfe01850cdb60
> > sys___sysctl() at sys___sysctl+0x5f/frame 0xfe01850cdc10
> > amd64_syscall() at amd64_syscall+0x135/frame 0xfe01850cdd30
> > fast_syscall_common() at fast_syscall_common+0xf8/frame
> > 0xfe01850cdd30
> > --- syscall (202, FreeBSD ELF64, sys___sysctl), rip = 0x80038968a,
> rsp =
> > 0x7fffd988, rbp = 0x7fffd9c0 ---
> > ===
> >
> > sysctl_devices+0x24b (0x6dab) was:
> >
> > sb->s_len += strlen(p);
> > 6d50:   4c 89 e7mov%r12,%rdi
> > 6d53:   e8 00 00 00 00  callq  6d58
> > 
> > 6d58:   48 01 45 b0 add%rax,-0x50(%rbp)
> > 6d5c:   48 8d 7d 88 lea-0x78(%rbp),%rdi
> > sbuf_putc(&sb, '\0');
> > 6d60:   31 f6   xor%esi,%esi
> > 6d62:   e8 00 00 00 00  callq  6d67
> > 
> > MPASS((sb->s_flags & SBUF_INCLUDENUL) == 0);
> > 6d67:   f6 45 b8 02 testb  $0x2,-0x48(%rbp)
> > 6d6b:   0f 85 10 01 00 00   jne6e81
> > 
> > if (sb->s_error != 0)
> > 6d71:   83 7d a0 00 cmpl   $0x0,-0x60(%rbp)
> > 6d75:   0f 85 8c 00 00 00   jne6e07
> > 
> > p = EOB(sb);
> > 6d7b:   4c 8b 65 88 mov-0x78(%rbp),%r12
> > 6d7f:   48 8b 45 b0 mov-0x50(%rbp),%rax
> > *p = '\0';  /* sbuf buffer isn't NUL terminated until
> > sbuf_finish() */
> > 6d83:   41 c6 04 04 00  movb   $0x0,(%r12,%rax,1)
> > space = SPACE(sb);
> > 6d88:   4c 8b 6d a8 mov-0x58(%rbp),%r13
> > 6d8c:   4c 2b 6d b0 sub-0x50(%rbp),%r13
> > if (space <= 1) {
> > 6d90:   49 83 fd 01 cmp$0x1,%r13
> > 6d94:   77 09   ja 6d9f
> > 
> > sb->s_error = ENOMEM;
> >

Re: GPF on boot with devmatch

2020-10-05 Thread Xin Li 


On 10/4/20 10:13 PM, Warner Losh wrote:
> 
> 
> On Sun, Oct 4, 2020, 11:07 PM Xin Li  > wrote:
> 
> Hi,
> 
> I'm seeing this panic at boot after upgrading from r366217 to r366364,
> and continues to exist for r366421 (but I haven't find out the exact
> change that caused it).  Preloading the relevant kernel modules
> (uhid.ko, ums.ko and wmt.ko) seems to make the kernel boot correctly.
> 
> 
> What happens if you disable devmatch and load these modules by hand?

Loading these modules from loader or kld_list will prevent this panic
regardless if devmatch is enabled.

> What happens if you load them from rc.d scripts with devmatch disabled?

It seems that the devmatch was started by devd and not rc.  Disabling
devmatch (setting devmatch_enable="NO" without loading any of these klds
would not provoke the panic).

> Warner
> 
> This is not reproducible on my laptop, which will load many more kernel
> modules.
> 
> ===
> Autoloading module: uhid.ko
> Autoloading module: wmt.ko
> 
> 
> Fatal trap 9: general protection fault while in kernel mode
> cpuid = 2; apic id = 04
> instruction pointer     = 0x20:0x806ad6eb
> stack pointer           = 0x28:0xfe01850cd960
> frame pointer           = 0x28:0xfe01850cd9e0
> code segment            = base 0x0, limit 0xf, type 0x1b
>                         = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags        = interrupt enabled, resume, IOPL = 0
> current process         = 740 (devmatch)
> trap number             = 9
> panic: general protection fault
> cpuid = 3
> time = 1601866799
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> 0xfe01850cd670
> vpanic() at vpanic+0x182/frame 0xfe01850cd6c0
> panic() at panic+0x43/frame 0xfe01850cd720
> trap_fatal() at trap_fatal+0x387/frame 0xfe01850cd780
> trap() at trap+0xa4/frame 0xfe01850cd890
> calltrap() at calltrap+0x8/frame 0xfe01850cd890
> --- trap 0x9, rip = 0x806ad6eb, rsp = 0xfe01850cd960, rbp =
> 0xfe01850cd9e0 ---
> sysctl_devices() at sysctl_devices+0x24b/frame 0xfe01850cd9e0
> sysctl_root_handler_locked() at sysctl_root_handler_locked+0x9c/frame
> 0xfe01850cda30
> sysctl_root() at sysctl_root+0x20a/frame 0xfe01850cdab0
> userland_sysctl() at userland_sysctl+0x17d/frame 0xfe01850cdb60
> sys___sysctl() at sys___sysctl+0x5f/frame 0xfe01850cdc10
> amd64_syscall() at amd64_syscall+0x135/frame 0xfe01850cdd30
> fast_syscall_common() at fast_syscall_common+0xf8/frame
> 0xfe01850cdd30
> --- syscall (202, FreeBSD ELF64, sys___sysctl), rip = 0x80038968a, rsp =
> 0x7fffd988, rbp = 0x7fffd9c0 ---
> ===
> 
> sysctl_devices+0x24b (0x6dab) was:
> 
>         sb->s_len += strlen(p);
>     6d50:       4c 89 e7                mov    %r12,%rdi
>     6d53:       e8 00 00 00 00          callq  6d58
> 
>     6d58:       48 01 45 b0             add    %rax,-0x50(%rbp)
>     6d5c:       48 8d 7d 88             lea    -0x78(%rbp),%rdi
>         sbuf_putc(&sb, '\0');
>     6d60:       31 f6                   xor    %esi,%esi
>     6d62:       e8 00 00 00 00          callq  6d67
> 
>         MPASS((sb->s_flags & SBUF_INCLUDENUL) == 0);
>     6d67:       f6 45 b8 02             testb  $0x2,-0x48(%rbp)
>     6d6b:       0f 85 10 01 00 00       jne    6e81
> 
>         if (sb->s_error != 0)
>     6d71:       83 7d a0 00             cmpl   $0x0,-0x60(%rbp)
>     6d75:       0f 85 8c 00 00 00       jne    6e07
> 
>         p = EOB(sb);
>     6d7b:       4c 8b 65 88             mov    -0x78(%rbp),%r12
>     6d7f:       48 8b 45 b0             mov    -0x50(%rbp),%rax
>         *p = '\0';      /* sbuf buffer isn't NUL terminated until
> sbuf_finish() */
>     6d83:       41 c6 04 04 00          movb   $0x0,(%r12,%rax,1)
>         space = SPACE(sb);
>     6d88:       4c 8b 6d a8             mov    -0x58(%rbp),%r13
>     6d8c:       4c 2b 6d b0             sub    -0x50(%rbp),%r13
>         if (space <= 1) {
>     6d90:       49 83 fd 01             cmp    $0x1,%r13
>     6d94:       77 09                   ja     6d9f
> 
>                 sb->s_error = ENOMEM;
>     6d96:       c7 45 a0 0c 00 00 00    movl   $0xc,-0x60(%rbp)
>     6d9d:       eb 68                   jmp    6e07
> 
>     6d9f:       49 01 c4                add    %rax,%r12
>         return (dev->parent);
>     6da2:       48 8b 7b 28             mov    0x28(%rbx),%rdi
>         if (parent == NULL) {
>     6da6:       48 85 ff                test   %rdi,%rdi
>     6da9:       74 4b                   je     6df6
> 
>         KOBJOPLOOKUP(((kobj_t)_dev)->ops,bus_child_location_

Re: GPF on boot with devmatch

2020-10-04 Thread Warner Losh 
On Sun, Oct 4, 2020, 11:07 PM Xin Li  wrote:

> Hi,
>
> I'm seeing this panic at boot after upgrading from r366217 to r366364,
> and continues to exist for r366421 (but I haven't find out the exact
> change that caused it).  Preloading the relevant kernel modules
> (uhid.ko, ums.ko and wmt.ko) seems to make the kernel boot correctly.
>

What happens if you disable devmatch and load these modules by hand? What
happens if you load them from rc.d scripts with devmatch disabled?

Warner

This is not reproducible on my laptop, which will load many more kernel
> modules.
>
> ===
> Autoloading module: uhid.ko
> Autoloading module: wmt.ko
>
>
> Fatal trap 9: general protection fault while in kernel mode
> cpuid = 2; apic id = 04
> instruction pointer = 0x20:0x806ad6eb
> stack pointer   = 0x28:0xfe01850cd960
> frame pointer   = 0x28:0xfe01850cd9e0
> code segment= base 0x0, limit 0xf, type 0x1b
> = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags= interrupt enabled, resume, IOPL = 0
> current process = 740 (devmatch)
> trap number = 9
> panic: general protection fault
> cpuid = 3
> time = 1601866799
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> 0xfe01850cd670
> vpanic() at vpanic+0x182/frame 0xfe01850cd6c0
> panic() at panic+0x43/frame 0xfe01850cd720
> trap_fatal() at trap_fatal+0x387/frame 0xfe01850cd780
> trap() at trap+0xa4/frame 0xfe01850cd890
> calltrap() at calltrap+0x8/frame 0xfe01850cd890
> --- trap 0x9, rip = 0x806ad6eb, rsp = 0xfe01850cd960, rbp =
> 0xfe01850cd9e0 ---
> sysctl_devices() at sysctl_devices+0x24b/frame 0xfe01850cd9e0
> sysctl_root_handler_locked() at sysctl_root_handler_locked+0x9c/frame
> 0xfe01850cda30
> sysctl_root() at sysctl_root+0x20a/frame 0xfe01850cdab0
> userland_sysctl() at userland_sysctl+0x17d/frame 0xfe01850cdb60
> sys___sysctl() at sys___sysctl+0x5f/frame 0xfe01850cdc10
> amd64_syscall() at amd64_syscall+0x135/frame 0xfe01850cdd30
> fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfe01850cdd30
> --- syscall (202, FreeBSD ELF64, sys___sysctl), rip = 0x80038968a, rsp =
> 0x7fffd988, rbp = 0x7fffd9c0 ---
> ===
>
> sysctl_devices+0x24b (0x6dab) was:
>
> sb->s_len += strlen(p);
> 6d50:   4c 89 e7mov%r12,%rdi
> 6d53:   e8 00 00 00 00  callq  6d58 
> 6d58:   48 01 45 b0 add%rax,-0x50(%rbp)
> 6d5c:   48 8d 7d 88 lea-0x78(%rbp),%rdi
> sbuf_putc(&sb, '\0');
> 6d60:   31 f6   xor%esi,%esi
> 6d62:   e8 00 00 00 00  callq  6d67 
> MPASS((sb->s_flags & SBUF_INCLUDENUL) == 0);
> 6d67:   f6 45 b8 02 testb  $0x2,-0x48(%rbp)
> 6d6b:   0f 85 10 01 00 00   jne6e81 
> if (sb->s_error != 0)
> 6d71:   83 7d a0 00 cmpl   $0x0,-0x60(%rbp)
> 6d75:   0f 85 8c 00 00 00   jne6e07 
> p = EOB(sb);
> 6d7b:   4c 8b 65 88 mov-0x78(%rbp),%r12
> 6d7f:   48 8b 45 b0 mov-0x50(%rbp),%rax
> *p = '\0';  /* sbuf buffer isn't NUL terminated until
> sbuf_finish() */
> 6d83:   41 c6 04 04 00  movb   $0x0,(%r12,%rax,1)
> space = SPACE(sb);
> 6d88:   4c 8b 6d a8 mov-0x58(%rbp),%r13
> 6d8c:   4c 2b 6d b0 sub-0x50(%rbp),%r13
> if (space <= 1) {
> 6d90:   49 83 fd 01 cmp$0x1,%r13
> 6d94:   77 09   ja 6d9f 
> sb->s_error = ENOMEM;
> 6d96:   c7 45 a0 0c 00 00 00movl   $0xc,-0x60(%rbp)
> 6d9d:   eb 68   jmp6e07 
> 6d9f:   49 01 c4add%rax,%r12
> return (dev->parent);
> 6da2:   48 8b 7b 28 mov0x28(%rbx),%rdi
> if (parent == NULL) {
> 6da6:   48 85 fftest   %rdi,%rdi
> 6da9:   74 4b   je 6df6 
> KOBJOPLOOKUP(((kobj_t)_dev)->ops,bus_child_location_str);
> 6dab:   48 8b 07mov(%rdi),%rax
> 6dae:   48 c7 c2 00 00 00 00mov$0x0,%rdx
> 6db5:   0f b6 0d 00 00 00 00movzbl 0x0(%rip),%ecx#
> 6dbc 
> 6dbc:   4c 8b 04 c8 mov(%rax,%rcx,8),%r8
> 6dc0:   49 39 10cmp%rdx,(%r8)
> 6dc3:   74 22   je 6de7 
> 6dc5:   48 8d 34 c8 lea(%rax,%rcx,8),%rsi
> 6dc9:   48 89 7d d0 mov%rdi,-0x30(%rbp)
> 6dcd:   48 8b b8 00 08 00 00mov0x800(%rax),%rdi
> 6dd4:   48 c7 c2 00 00 00 00mov$0x0,%rdx
> 6ddb:   e8 00 00 00 00  callq  6de0 
> 6de0:   48 8b 7d d0 mov-0