Re: ia64 r225789 panic during "make installworld": Bad buffer logic, remain = 0

[Kostik Belousov]

On Wed, Sep 28, 2011 at 04:27:39PM +0300, Jaakko Heinonen wrote:
> On 2011-09-28, Anton Shterenlikht wrote:
> > KDB: stack backtrace:
> > getenv with the following non-sleepable locks held:
> > exclusive sleep mutex vnode interlock (vnode interlock) r = 0 
> > (0xe00011950488) locked @ /usr/src/sys/fs/devfs/devfs_vnops.c:406
> > 
> > etc. until a hang, requiring cold reset via MP.
> 
> Someone is calling getenv with a vnode interlock held. You need to
> figure out the caller. Unfortunately the backtrace is missing above.
> 
> As a temporary workaround you could comment the WITNESS_WARN() line in
> getenv() (sys/kern/kern_environment.c) but it is not a real fix.

I do not think that this is the real cause of the panic. Line 406
in devfs_vnops.c belongs to devfs_allocv(), and vnode interlock taken
there must be consumed by LK_INTERLOCK call to vget(). The getenv()
cannot be called from the vget() or two unlock calls between lines
406 and 409.

It seems there is something broken elsewere.


pgpP9hKdJwqiy.pgp
Description: PGP signature

Re: ia64 r225789 panic during "make installworld": Bad buffer logic, remain = 0

[Jaakko Heinonen]

On 2011-09-28, Anton Shterenlikht wrote:
> KDB: stack backtrace:
> getenv with the following non-sleepable locks held:
> exclusive sleep mutex vnode interlock (vnode interlock) r = 0 
> (0xe00011950488) locked @ /usr/src/sys/fs/devfs/devfs_vnops.c:406
> 
> etc. until a hang, requiring cold reset via MP.

Someone is calling getenv with a vnode interlock held. You need to
figure out the caller. Unfortunately the backtrace is missing above.

As a temporary workaround you could comment the WITNESS_WARN() line in
getenv() (sys/kern/kern_environment.c) but it is not a real fix.

-- 
Jaakko
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

ia64 r225789 panic during "make installworld": Bad buffer logic, remain = 0

[Anton Shterenlikht]

This is in single user mode.

===> usr.sbin/amd (install)
===> usr.sbin/amd/include (install)
===> usr.sbin/amd/libamu (install)
===> usr.sbin/amd/amd (install)
install -s -o root -g wheel -m 555   amd /usr/sbin
lock order reversal:
 1st 0xe000141cdcb8 ufs (ufs) @ /usr/src/sys/kern/vfs_vnops.c:618
 2nd 0xa0005ed59c38 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:1911
 3rd 0xe000141ce098 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2134
KDB: stack backtrace:
getenv with the following non-sleepable locks held:
exclusive sleep mutex vnode interlock (vnode interlock) r = 0 
(0xe000141ce0c8) locked @ /usr/src/sys/kern/vfs_subr.c:2132
KDB: stack backtrace:
getenv with the following non-sleepable locks held:
exclusive sleep mutex vnode interlock (vnode interlock) r = 0 
(0xe000141ce0c8) locked @ /usr/src/sys/kern/vfs_subr.c:2132
KDB: stack backtrace:
getenv with the following non-sleepable locks held:
exclusive sleep mutex vnode interlock (vnode interlock) r = 0 
(0xe000141ce0c8) locked @ /usr/src/sys/kern/vfs_subr.c:2132
KDB: stack backtrace:
getenv with the following non-sleepable locks held:
exclusive sleep mutex vnode interlock (vnode interlock) r = 0 
(0xe000141ce0c8) locked @ /usr/src/sys/kern/vfs_subr.c:2132
KDB: stack backtrace:
getenv with the following non-sleepable locks held:
panic: Bad buffer logic, remain = 0
cpuid = 1
KDB: enter: panic
[ thread pid 20507 tid 100097 ]
Stopped at  kdb_enter+0x92: [I2]addl r14=0xffe211d8,gp ;;
db> 

db> show proc
Process 20507 (install) at 0xe00012dca000:
 state: NORMAL
 uid: 0  gids: 0
 parent: pid 20506 at 0xe00012cd2d50
 ABI: FreeBSD ELF64
 arguments: install
 threads: 1
100097   Run CPU 1   install
db>

db> show thread
Thread 100097 at 0xe00012dce450:
 proc (pid 20507): 0xe00012dca000
 name: install
 stack: 0xa00096cf6000-0xa00096cfdfff
 flags: 0x4  pflags: 0x40
 state: RUNNING (CPU 1)
 priority: 152
 container lock: sched lock 1 (0x9ffc00a73b00)
db>

db> thread 100097
[ thread pid 20507 tid 100097 ]
kdb_enter+0x92: [I2]addl r14=0xffe211d8,gp ;;
db>

db> bt
Tracing pid 20507 tid 100097 td 0xe00012dce450
kdb_enter(0x9ffc0095f418, 0x9ffc0095f418, 0x9ffc003756d0, 0x793) at 
kdb_enter+0x92
panic(0x9ffc009649c8, 0x0, 0x1, 0x9ffc003ecec0, 0x38c) at panic+0x2e0
putbuf(0xa, 0xa00096cf7c70) at putbuf+0x4d0
putchar(0xa, 0xa00096cf7c70, 0x5, 0x9ffc003eaa30, 0xda0) at 
putchar+0x110
kvprintf(0x9ffc0095f1d1, 0x9ffc009a98c0, 0xa00096cf7c70, 0xa, 
0xa00096cf7d38) at kvprintf+0xf0
vprintf(0x9ffc0095f1d0, 0xa00096cf7d38) at vprintf+0xa0
printf(0x9ffc0095f1d0, 0x0, 0xa0008de2ede8, 0xa0008de2edf0) at 
printf+0x80
printtrap(0x18, 0xa00096cf8800, 0x1, 0x0) at printtrap+0x20
trap_panic(0x9ffc00c276b0, 0x0, 0x9ffc0081fdb0, 0x716, 
0x9ffc00a27500) at trap_panic+0x50
trap(0x18, 0xa00096cf8800) at trap+0xb60
ivt_General_Exception() at ivt_General_Exception+0x30
--- trapframe at 0xa00096cf8800
spinlock_exit() at spinlock_exit+0x71
uart_quicc_class(0xa00096cf813e, 0x0, 0x1, 0x9ffc003ecec0, 0x38c, 
0x16a5a41, 0xe0011fb8, 0x) at 0x18
db> 

I rebooted, run "make installworld" again,
no panic this time, but on reboot:

KDB: stack backtrace:
getenv with the following non-sleepable locks held:
exclusive sleep mutex vnode interlock (vnode interlock) r = 0 
(0xe00011950488) locked @ /usr/src/sys/fs/devfs/devfs_vnops.c:406
KDB: stack backtrace:
getenv with the following non-sleepable locks held:
exclusive sleep mutex vnode interlock (vnode interlock) r = 0 
(0xe00011950488) locked @ /usr/src/sys/fs/devfs/devfs_vnops.c:406
KDB: stack backtrace:
getenv with the following non-sleepable locks held:
exclusive sleep mutex vnode interlock (vnode interlock) r = 0 
(0xe00011950488) locked @ /usr/src/sys/fs/devfs/devfs_vnops.c:406
KDB: stack backtrace:
getenv with the following non-sleepable locks held:
exclusive sleep mutex vnode interlock (vnode interlock) r = 0 
(0xe00011950488) locked @ /usr/src/sys/fs/devfs/devfs_vnops.c:406
KDB: stack backtrace:
getenv with the following non-sleepable locks held:
exclusive sleep mutex vnode interlock (vnode interlock) r = 0 
(0xe00011950488) locked @ /usr/src/sys/fs/devfs/devfs_vnops.c:406
KDB: stack backtrace:
getenv with the following non-sleepable locks held:
exclusive sleep mutex vnode interlock (vnode interlock) r = 0 
(0xe00011950488) locked @ /usr/src/sys/fs/devfs/devfs_vnops.c:406
KDB: stack backtrace:
getenv with the following non-sleepable locks held:
exclusive sleep mutex vnode interlock (vnode interlock) r = 0 
(0xe00011950488) locked @ /usr/src/sys/fs/devfs/devfs_vnops.c:406
KDB: stack backtrace:
getenv with the following non-sleepable locks held:
exclusive sleep mutex vnode interlock (vnode interlock) r = 0 
(0xe00011950488) locked @ /usr/src/sys/fs/devfs/devfs_vnops.c:406

etc. until a h