New Lock Order Reversal in 12.0?
Hi, I'm running 12.0-current and noticed a LOR message from WITNESS which I couldn't find a report about. I looked at http://sources.zabbadoz.net/freebsd/lor.html, among other places. system details: root@triskelion:~ # uname -a FreeBSD triskelion 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r311461: Thu Jan 5 22:46:38 UTC 2017 r...@releng3.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 root@triskelion:~ # freebsd-version 12.0-CURRENT WITNESS report: lock order reversal: 1st 0xf8002e8049a0 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2598 2nd 0xfe01e7ce9b40 bufwait (bufwait) @ /usr/src/sys/ufs/ffs/ffs_vnops.c:277 3rd 0xf8002ec7b9a0 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2598 stack backtrace: #0 0x80aa6fd0 at witness_debugger+0x70 #1 0x80aa6ed3 at witness_checkorder+0xde3 #2 0x80a20c15 at __lockmgr_args+0x725 #3 0x80d06fc5 at ffs_lock+0xa5 #4 0x8101c0c0 at VOP_LOCK1_APV+0xe0 #5 0x80b1a6aa at _vn_lock+0x9a #6 0x80b0ac94 at vget+0x64 #7 0x80afd19c at vfs_hash_get+0xcc #8 0x80d02e5e at ffs_vgetf+0x3e #9 0x80cf9787 at softdep_sync_buf+0xc37 #10 0x80d07c51 at ffs_syncvnode+0x2a1 #11 0x80d06e60 at ffs_fsync+0x20 #12 0x8101b110 at VOP_FSYNC_APV+0xe0 #13 0x80d0f2f0 at ufs_direnter+0x870 #14 0x80d18050 at ufs_makeinode+0x5c0 #15 0x80d13d7a at ufs_create+0x3a #16 0x810199ca at VOP_CREATE_APV+0xda #17 0x80b19f77 at vn_open_cred+0x2c7 This is based on the FreeBSD-12.0-CURRENT-amd64-20170105-r311461-memstick.img installer. Known issue? It happened during a portsnap fetch. the filesystem is UFS with default mount options. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: New Lock Order Reversal in 12.0?
On Mon, Jan 09, 2017 at 12:32:28AM +, Anindya Mukherjee wrote: > Hi, I'm running 12.0-current and noticed a LOR message from WITNESS which I > couldn't find a report about. I looked at > http://sources.zabbadoz.net/freebsd/lor.html, among other places. > > system details: > root@triskelion:~ # uname -a > FreeBSD triskelion 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r311461: Thu Jan 5 > 22:46:38 UTC 2017 > r...@releng3.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 > root@triskelion:~ # freebsd-version > 12.0-CURRENT > > > WITNESS report: > lock order reversal: > 1st 0xf8002e8049a0 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2598 > 2nd 0xfe01e7ce9b40 bufwait (bufwait) @ > /usr/src/sys/ufs/ffs/ffs_vnops.c:277 > 3rd 0xf8002ec7b9a0 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2598 > stack backtrace: > #0 0x80aa6fd0 at witness_debugger+0x70 > #1 0x80aa6ed3 at witness_checkorder+0xde3 > #2 0x80a20c15 at __lockmgr_args+0x725 > #3 0x80d06fc5 at ffs_lock+0xa5 > #4 0x8101c0c0 at VOP_LOCK1_APV+0xe0 > #5 0x80b1a6aa at _vn_lock+0x9a > #6 0x80b0ac94 at vget+0x64 > #7 0x80afd19c at vfs_hash_get+0xcc > #8 0x80d02e5e at ffs_vgetf+0x3e > #9 0x80cf9787 at softdep_sync_buf+0xc37 > #10 0x80d07c51 at ffs_syncvnode+0x2a1 > #11 0x80d06e60 at ffs_fsync+0x20 > #12 0x8101b110 at VOP_FSYNC_APV+0xe0 > #13 0x80d0f2f0 at ufs_direnter+0x870 > #14 0x80d18050 at ufs_makeinode+0x5c0 > #15 0x80d13d7a at ufs_create+0x3a > #16 0x810199ca at VOP_CREATE_APV+0xda > #17 0x80b19f77 at vn_open_cred+0x2c7 > > This is based on the FreeBSD-12.0-CURRENT-amd64-20170105-r311461-memstick.img > installer. Known issue? You do not think it looks like http://sources.zabbadoz.net/freebsd/lor/238.html ? -Ben ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
RE: New Lock Order Reversal in 12.0?
Hi Ben, Thanks for your reply, and yes, I did notice #238. I should say at this point that I'm a newbie when it comes to kernel code and am trying to learn about it (hence current). The entry you refer to does look a bit like the one I've got, but I'm not totally sure if the same code path is being followed to arrive at this LOR. An inode is being created in my case, vs a directory creation (entry + inode probably) in #238, and then a sync is being attempted, which causes locks to activate in the softdep code. I've read a bit about this from McCusick's book but the details are still fuzzy. Perhaps you are trying to tell me that it's benign? I know that WITNESS has false positives, an example being #236 where due to shared vs exclusive vnode locks required on the two ways to lock bufwait and dirhash a deadlock never happens. Best, Anindya From: Benjamin Kaduk [ka...@mit.edu] Sent: January 8, 2017 4:47 PM To: Anindya Mukherjee Cc: freebsd-current@freebsd.org Subject: Re: New Lock Order Reversal in 12.0? On Mon, Jan 09, 2017 at 12:32:28AM +, Anindya Mukherjee wrote: > Hi, I'm running 12.0-current and noticed a LOR message from WITNESS which I > couldn't find a report about. I looked at > http://sources.zabbadoz.net/freebsd/lor.html, among other places. > > system details: > root@triskelion:~ # uname -a > FreeBSD triskelion 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r311461: Thu Jan 5 > 22:46:38 UTC 2017 > r...@releng3.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 > root@triskelion:~ # freebsd-version > 12.0-CURRENT > > > WITNESS report: > lock order reversal: > 1st 0xf8002e8049a0 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2598 > 2nd 0xfe01e7ce9b40 bufwait (bufwait) @ > /usr/src/sys/ufs/ffs/ffs_vnops.c:277 > 3rd 0xf8002ec7b9a0 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2598 > stack backtrace: > #0 0x80aa6fd0 at witness_debugger+0x70 > #1 0x80aa6ed3 at witness_checkorder+0xde3 > #2 0x80a20c15 at __lockmgr_args+0x725 > #3 0x80d06fc5 at ffs_lock+0xa5 > #4 0x8101c0c0 at VOP_LOCK1_APV+0xe0 > #5 0x80b1a6aa at _vn_lock+0x9a > #6 0x80b0ac94 at vget+0x64 > #7 0x80afd19c at vfs_hash_get+0xcc > #8 0x80d02e5e at ffs_vgetf+0x3e > #9 0x80cf9787 at softdep_sync_buf+0xc37 > #10 0x80d07c51 at ffs_syncvnode+0x2a1 > #11 0x80d06e60 at ffs_fsync+0x20 > #12 0x8101b110 at VOP_FSYNC_APV+0xe0 > #13 0x80d0f2f0 at ufs_direnter+0x870 > #14 0x80d18050 at ufs_makeinode+0x5c0 > #15 0x80d13d7a at ufs_create+0x3a > #16 0x810199ca at VOP_CREATE_APV+0xda > #17 0x80b19f77 at vn_open_cred+0x2c7 > > This is based on the FreeBSD-12.0-CURRENT-amd64-20170105-r311461-memstick.img > installer. Known issue? You do not think it looks like http://sources.zabbadoz.net/freebsd/lor/238.html ? -Ben ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: New Lock Order Reversal in 12.0?
On Mon, Jan 09, 2017 at 02:31:39AM +, Anindya Mukherjee wrote: > Hi Ben, > > Thanks for your reply, and yes, I did notice #238. I should say at this point > that I'm a newbie when it comes to kernel code and am trying to learn about > it (hence current). Understood. It's good that you are ready to try! > The entry you refer to does look a bit like the one I've got, but I'm not > totally sure if the same code path is being followed to arrive at this LOR. > An inode is being created in my case, vs a directory creation (entry + inode > probably) in #238, and then a sync is being attempted, which causes locks to > activate in the softdep code. I've read a bit about this from McCusick's book > but the details are still fuzzy. > > Perhaps you are trying to tell me that it's benign? I know that WITNESS has > false positives, an example being #236 where due to shared vs exclusive vnode > locks required on the two ways to lock bufwait and dirhash a deadlock never > happens. Well, it's hard to say for certain, but there are a few ufs/bufwait/ufs LORs that are very commonly reported as WITNESS output but have never (IIRC) been identified as causing a deadlock. So, it seems pretty likely that this one is similarly benign -- I, for one, do not plan to put in time trying to analyze it until there is a hang or deadlock associated with it. -Ben ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
RE: New Lock Order Reversal in 12.0?
Good, that makes me feel better :) The system is running fine, so I think you're right and it's nothing to worry about. Thanks again for your responses. Best, Anindya From: Benjamin Kaduk [ka...@mit.edu] Sent: January 9, 2017 1:53 PM To: Anindya Mukherjee Cc: freebsd-current@freebsd.org Subject: Re: New Lock Order Reversal in 12.0? On Mon, Jan 09, 2017 at 02:31:39AM +, Anindya Mukherjee wrote: > Hi Ben, > > Thanks for your reply, and yes, I did notice #238. I should say at this point > that I'm a newbie when it comes to kernel code and am trying to learn about > it (hence current). Understood. It's good that you are ready to try! > The entry you refer to does look a bit like the one I've got, but I'm not > totally sure if the same code path is being followed to arrive at this LOR. > An inode is being created in my case, vs a directory creation (entry + inode > probably) in #238, and then a sync is being attempted, which causes locks to > activate in the softdep code. I've read a bit about this from McCusick's book > but the details are still fuzzy. > > Perhaps you are trying to tell me that it's benign? I know that WITNESS has > false positives, an example being #236 where due to shared vs exclusive vnode > locks required on the two ways to lock bufwait and dirhash a deadlock never > happens. Well, it's hard to say for certain, but there are a few ufs/bufwait/ufs LORs that are very commonly reported as WITNESS output but have never (IIRC) been identified as causing a deadlock. So, it seems pretty likely that this one is similarly benign -- I, for one, do not plan to put in time trying to analyze it until there is a hang or deadlock associated with it. -Ben ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"