Re: tmpfs deadlock on stable/9
On Thu, 8 Dec 2011, Dmitry Morozovsky wrote: > > > > It is available at http://bsd.woozle.net/tmpfs-lock-20111207.txt (~260k) > > > > > > > > BTW, at least some of the debugger commands referenced (show locks, > > > > show > > > > alllocks) are no longer exist > > > This means that you do not have witness in your kernel. > > > Look at the reference I pointed you once more. > > > > Ah I see, my bad. I'll test this with WITNESS-enabled kernel tomorrow and > > return with updated results. > > Hmm, I'd rebuilt kernel with WITNESS/WITNESS_SKIPSPIN and can't reproduce > this > deadlock. Will try further. On the other hand, after boot I see several LORs: Dec 8 02:34:18 beaver kernel: lock order reversal: Dec 8 02:34:18 beaver kernel: 1st 0xfe005163b818 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:829 Dec 8 02:34:18 beaver kernel: 2nd 0xfe005163cbd8 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2166 Dec 8 02:34:18 beaver kernel: KDB: stack backtrace: Dec 8 02:34:18 beaver kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Dec 8 02:34:18 beaver kernel: kdb_backtrace() at kdb_backtrace+0x37 Dec 8 02:34:18 beaver kernel: _witness_debugger() at _witness_debugger+0x2c Dec 8 02:34:18 beaver kernel: witness_checkorder() at witness_checkorder+0x651 Dec 8 02:34:18 beaver kernel: __lockmgr_args() at __lockmgr_args+0xb98 Dec 8 02:34:18 beaver kernel: vop_stdlock() at vop_stdlock+0x39 Dec 8 02:34:18 beaver kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 Dec 8 02:34:18 beaver kernel: _vn_lock() at _vn_lock+0x47 Dec 8 02:34:18 beaver kernel: vget() at vget+0x56 Dec 8 02:34:18 beaver kernel: devfs_allocv() at devfs_allocv+0x13f Dec 8 02:34:18 beaver kernel: devfs_root() at devfs_root+0x4d Dec 8 02:34:18 beaver kernel: vfs_donmount() at vfs_donmount+0xdcb Dec 8 02:34:18 beaver kernel: sys_nmount() at sys_nmount+0x63 Dec 8 02:34:18 beaver kernel: amd64_syscall() at amd64_syscall+0x25a Dec 8 02:34:18 beaver kernel: Xfast_syscall() at Xfast_syscall+0xf7 Dec 8 02:34:18 beaver kernel: --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x800ab5ecc, rsp = 0x7fffccc8, rbp = 0x801009048 --- Dec 8 02:34:29 beaver kernel: 4-i386.lab.rinet.ru Dec 8 02:34:32 beaver kernel: 6-i386.lab.rinet.ru. Dec 8 02:41:15 beaver kernel: lock order reversal: Dec 8 02:41:15 beaver kernel: 1st 0xfe004ea069f8 tmpfs (tmpfs) @ /usr/src/sys/fs/nullfs/null_vnops.c:597 Dec 8 02:41:15 beaver kernel: 2nd 0xfe019db46bd8 zfs (zfs) @ /usr/src/sys/fs/nullfs/null_vnops.c:597 Dec 8 02:41:15 beaver kernel: KDB: stack backtrace: Dec 8 02:41:15 beaver kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Dec 8 02:41:15 beaver kernel: kdb_backtrace() at kdb_backtrace+0x37 Dec 8 02:41:15 beaver kernel: _witness_debugger() at _witness_debugger+0x2c Dec 8 02:41:15 beaver kernel: witness_checkorder() at witness_checkorder+0x651 Dec 8 02:41:15 beaver kernel: __lockmgr_args() at __lockmgr_args+0xb98 Dec 8 02:41:15 beaver kernel: vop_stdlock() at vop_stdlock+0x39 Dec 8 02:41:15 beaver kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 Dec 8 02:41:15 beaver kernel: null_lock() at null_lock+0xbb Dec 8 02:41:15 beaver kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 Dec 8 02:41:15 beaver kernel: _vn_lock() at _vn_lock+0x47 Dec 8 02:41:15 beaver kernel: nullfs_root() at nullfs_root+0x45 Dec 8 02:41:15 beaver kernel: vfs_donmount() at vfs_donmount+0xdcb Dec 8 02:41:15 beaver kernel: sys_nmount() at sys_nmount+0x63 Dec 8 02:41:15 beaver kernel: amd64_syscall() at amd64_syscall+0x25a Dec 8 02:41:15 beaver kernel: Xfast_syscall() at Xfast_syscall+0xf7 Dec 8 02:41:15 beaver kernel: --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x8008a1ecc, rsp = 0x7fffceb8, rbp = 0x7fffcf30 --- Dec 8 02:41:16 beaver kernel: lock order reversal: Dec 8 02:41:16 beaver kernel: 1st 0xfe0171868638 tmpfs (tmpfs) @ /usr/src/sys/fs/nullfs/null_vnops.c:597 Dec 8 02:41:16 beaver kernel: 2nd 0xfe004ea0ebd8 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2166 Dec 8 02:41:16 beaver kernel: KDB: stack backtrace: Dec 8 02:41:16 beaver kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Dec 8 02:41:16 beaver kernel: kdb_backtrace() at kdb_backtrace+0x37 Dec 8 02:41:16 beaver kernel: _witness_debugger() at _witness_debugger+0x2c Dec 8 02:41:16 beaver kernel: witness_checkorder() at witness_checkorder+0x651 Dec 8 02:41:16 beaver kernel: __lockmgr_args() at __lockmgr_args+0xb98 Dec 8 02:41:16 beaver kernel: vop_stdlock() at vop_stdlock+0x39 Dec 8 02:41:16 beaver kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 Dec 8 02:41:16 beaver kernel: _vn_lock() at _vn_lock+0x47 Dec 8 02:41:16 beaver kernel: vget() at vget+0x56 Dec 8 02:41:16 beaver kernel: devfs_allocv() at devfs_allocv+0x13f Dec 8 02:41:16 beaver kernel: devfs_root() at devfs_root+0x4d Dec 8 02:41:16 beaver kernel: vfs_donmount() at vfs_donmount+0xdcb Dec 8 02:41:16 bea
Re: tmpfs deadlock on stable/9
On Thu, 8 Dec 2011, Dmitry Morozovsky wrote: > > > It is available at http://bsd.woozle.net/tmpfs-lock-20111207.txt (~260k) > > > > > > BTW, at least some of the debugger commands referenced (show locks, show > > > alllocks) are no longer exist > > This means that you do not have witness in your kernel. > > Look at the reference I pointed you once more. > > Ah I see, my bad. I'll test this with WITNESS-enabled kernel tomorrow and > return with updated results. Hmm, I'd rebuilt kernel with WITNESS/WITNESS_SKIPSPIN and can't reproduce this deadlock. Will try further. -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: ma...@freebsd.org ] *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- ma...@rinet.ru *** ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: tmpfs deadlock on stable/9
On Wed, 7 Dec 2011, Kostik Belousov wrote: > > It is available at http://bsd.woozle.net/tmpfs-lock-20111207.txt (~260k) > > > > BTW, at least some of the debugger commands referenced (show locks, show > > alllocks) are no longer exist > This means that you do not have witness in your kernel. > Look at the reference I pointed you once more. Ah I see, my bad. I'll test this with WITNESS-enabled kernel tomorrow and return with updated results. -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: ma...@freebsd.org ] *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- ma...@rinet.ru *** ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: tmpfs deadlock on stable/9
On Wed, Dec 07, 2011 at 09:59:32PM +0400, Dmitry Morozovsky wrote: > On Wed, 7 Dec 2011, Kostik Belousov wrote: > > > > I have ports tinderbox runnign on stable/9-amd64, with working > > > directories on > > > tmpfs. I have two consecutive tmpfs deadlocks like > > > > > > root@beaver:/usr/local/tb/scripts# ps t2 > > > PID TT STATTIME COMMAND > > > 2337 2 Is 0:00.04 /bin/tcsh > > > 3079 2 I0:00.01 sudo -sE > > > 3260 2 I0:00.02 /bin/tcsh > > > 20309 2 I+ 0:00.06 /bin/sh ./tc tinderbuild -nullfs -norebuild -b > > > 9-i386-RiNet > > > 27035 2 S+ 0:00.13 make PACKAGES=/usr/local/tb/packages/9-i386-RiNet > > > -k > > > -j1 all > > > 46470 2 I+ 0:00.00 sh -ev > > > 46471 2 I+ 0:00.01 /bin/sh /usr/local/tb/scripts/lib/portbuild > > > 9-i386-RiNet 9-i386 RiNet -nullfs gsm-1.0.13.tbz /usr/ports/audio/gsm > > > 46677 2 I+ 0:00.00 /bin/sh /buildscript /usr/ports/audio/gsm 2 > > > 46766 2 I+ 0:00.00 /pnohang 7200 /tmp/make.log4 gsm-1.0.13 make build > > > 46767 2 I+ 0:00.02 make build > > > 46768 2 I+ 0:00.00 /pnohang 7200 /tmp/make.log4 gsm-1.0.13 make build > > > 46789 2 I+ 0:00.00 [sh] > > > 46790 2 D+ 0:00.01 make -f Makefile -j4 all > > > 46918 2 I+ 0:00.00 sh -ev > > > 46926 2 I+ 0:00.00 sh -ev > > > 46928 2 Z+ 0:00.09 > > > 46938 2 D+ 0:00.00 mv gsm_create.o ./src/gsm_create.o > > > 46940 2 D+ 0:00.00 mv gsm_print.o ./src/gsm_print.o > > > > > > (this is parallel build, last 2 rm's are deadlocked on tmpfs) > > > > > > what kind of additional info should I send? I have debugging turned on in > > > kernel, if it's needed. > > http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html > > > > No, I do not promise to look into it. > > It is available at http://bsd.woozle.net/tmpfs-lock-20111207.txt (~260k) > > BTW, at least some of the debugger commands referenced (show locks, show > alllocks) are no longer exist This means that you do not have witness in your kernel. Look at the reference I pointed you once more. pgpcWVtYWuC4L.pgp Description: PGP signature
Re: tmpfs deadlock on stable/9
On Wed, 7 Dec 2011, Kostik Belousov wrote: > > I have ports tinderbox runnign on stable/9-amd64, with working directories > > on > > tmpfs. I have two consecutive tmpfs deadlocks like > > > > root@beaver:/usr/local/tb/scripts# ps t2 > > PID TT STATTIME COMMAND > > 2337 2 Is 0:00.04 /bin/tcsh > > 3079 2 I0:00.01 sudo -sE > > 3260 2 I0:00.02 /bin/tcsh > > 20309 2 I+ 0:00.06 /bin/sh ./tc tinderbuild -nullfs -norebuild -b > > 9-i386-RiNet > > 27035 2 S+ 0:00.13 make PACKAGES=/usr/local/tb/packages/9-i386-RiNet > > -k > > -j1 all > > 46470 2 I+ 0:00.00 sh -ev > > 46471 2 I+ 0:00.01 /bin/sh /usr/local/tb/scripts/lib/portbuild > > 9-i386-RiNet 9-i386 RiNet -nullfs gsm-1.0.13.tbz /usr/ports/audio/gsm > > 46677 2 I+ 0:00.00 /bin/sh /buildscript /usr/ports/audio/gsm 2 > > 46766 2 I+ 0:00.00 /pnohang 7200 /tmp/make.log4 gsm-1.0.13 make build > > 46767 2 I+ 0:00.02 make build > > 46768 2 I+ 0:00.00 /pnohang 7200 /tmp/make.log4 gsm-1.0.13 make build > > 46789 2 I+ 0:00.00 [sh] > > 46790 2 D+ 0:00.01 make -f Makefile -j4 all > > 46918 2 I+ 0:00.00 sh -ev > > 46926 2 I+ 0:00.00 sh -ev > > 46928 2 Z+ 0:00.09 > > 46938 2 D+ 0:00.00 mv gsm_create.o ./src/gsm_create.o > > 46940 2 D+ 0:00.00 mv gsm_print.o ./src/gsm_print.o > > > > (this is parallel build, last 2 rm's are deadlocked on tmpfs) > > > > what kind of additional info should I send? I have debugging turned on in > > kernel, if it's needed. > http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html > > No, I do not promise to look into it. It is available at http://bsd.woozle.net/tmpfs-lock-20111207.txt (~260k) BTW, at least some of the debugger commands referenced (show locks, show alllocks) are no longer exist Thank you! -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: ma...@freebsd.org ] *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- ma...@rinet.ru *** ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: tmpfs deadlock on stable/9
On Wed, Dec 07, 2011 at 01:57:08PM +0400, Dmitry Morozovsky wrote: > Dear colleagues, > > I have ports tinderbox runnign on stable/9-amd64, with working directories on > tmpfs. I have two consecutive tmpfs deadlocks like > > root@beaver:/usr/local/tb/scripts# ps t2 > PID TT STATTIME COMMAND > 2337 2 Is 0:00.04 /bin/tcsh > 3079 2 I0:00.01 sudo -sE > 3260 2 I0:00.02 /bin/tcsh > 20309 2 I+ 0:00.06 /bin/sh ./tc tinderbuild -nullfs -norebuild -b > 9-i386-RiNet > 27035 2 S+ 0:00.13 make PACKAGES=/usr/local/tb/packages/9-i386-RiNet -k > -j1 all > 46470 2 I+ 0:00.00 sh -ev > 46471 2 I+ 0:00.01 /bin/sh /usr/local/tb/scripts/lib/portbuild > 9-i386-RiNet 9-i386 RiNet -nullfs gsm-1.0.13.tbz /usr/ports/audio/gsm > 46677 2 I+ 0:00.00 /bin/sh /buildscript /usr/ports/audio/gsm 2 > 46766 2 I+ 0:00.00 /pnohang 7200 /tmp/make.log4 gsm-1.0.13 make build > 46767 2 I+ 0:00.02 make build > 46768 2 I+ 0:00.00 /pnohang 7200 /tmp/make.log4 gsm-1.0.13 make build > 46789 2 I+ 0:00.00 [sh] > 46790 2 D+ 0:00.01 make -f Makefile -j4 all > 46918 2 I+ 0:00.00 sh -ev > 46926 2 I+ 0:00.00 sh -ev > 46928 2 Z+ 0:00.09 > 46938 2 D+ 0:00.00 mv gsm_create.o ./src/gsm_create.o > 46940 2 D+ 0:00.00 mv gsm_print.o ./src/gsm_print.o > > (this is parallel build, last 2 rm's are deadlocked on tmpfs) > > what kind of additional info should I send? I have debugging turned on in > kernel, if it's needed. http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html No, I do not promise to look into it. pgpQrBHe06ciC.pgp Description: PGP signature
tmpfs deadlock on stable/9
Dear colleagues, I have ports tinderbox runnign on stable/9-amd64, with working directories on tmpfs. I have two consecutive tmpfs deadlocks like root@beaver:/usr/local/tb/scripts# ps t2 PID TT STATTIME COMMAND 2337 2 Is 0:00.04 /bin/tcsh 3079 2 I0:00.01 sudo -sE 3260 2 I0:00.02 /bin/tcsh 20309 2 I+ 0:00.06 /bin/sh ./tc tinderbuild -nullfs -norebuild -b 9-i386-RiNet 27035 2 S+ 0:00.13 make PACKAGES=/usr/local/tb/packages/9-i386-RiNet -k -j1 all 46470 2 I+ 0:00.00 sh -ev 46471 2 I+ 0:00.01 /bin/sh /usr/local/tb/scripts/lib/portbuild 9-i386-RiNet 9-i386 RiNet -nullfs gsm-1.0.13.tbz /usr/ports/audio/gsm 46677 2 I+ 0:00.00 /bin/sh /buildscript /usr/ports/audio/gsm 2 46766 2 I+ 0:00.00 /pnohang 7200 /tmp/make.log4 gsm-1.0.13 make build 46767 2 I+ 0:00.02 make build 46768 2 I+ 0:00.00 /pnohang 7200 /tmp/make.log4 gsm-1.0.13 make build 46789 2 I+ 0:00.00 [sh] 46790 2 D+ 0:00.01 make -f Makefile -j4 all 46918 2 I+ 0:00.00 sh -ev 46926 2 I+ 0:00.00 sh -ev 46928 2 Z+ 0:00.09 46938 2 D+ 0:00.00 mv gsm_create.o ./src/gsm_create.o 46940 2 D+ 0:00.00 mv gsm_print.o ./src/gsm_print.o (this is parallel build, last 2 rm's are deadlocked on tmpfs) what kind of additional info should I send? I have debugging turned on in kernel, if it's needed. Thanks! -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: ma...@freebsd.org ] *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- ma...@rinet.ru *** ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
