On Mon, Jul 25, 2011 at 12:21:07PM +0200, Herve Boulouis wrote: > Hi list, > > We have 2 freebsd 8.2-STABLE (cvsuped june 22) that keeps crashing in a bad > way : > > The are doing heavy apache / php4 web serving from a nfs mount and panic at > least once a day > with the following message (no crash dump produced, hand copied from the > console) : > > Sleeping on "vmopar" with the following non-sleepable locks held: > exclusive sleep mutex NFSnode lock (NFSnode lock) r = 0 (0xffffff0201798000) > locked @ nfsclient/nfs_subs.c:538 > lock order reversal: > 1st 0xffffffff018ff6da80 turnstile lock (turnstile lock) @ > kern/subr_turnstile.c:190 > 2nd 0xffffffffff80b52b10 scrlock (scrlock) @ dev/syscons.c:2570 > lock order reversal: > 1st 0xffffffff018ff6da80 turnstile lock (turnstile lock) @ > kern/subr_turnstile.c:190 > 2nd 0xffffffffff80b78ef8 sleepq chain (sleepq chain) @ > kern/subr_turnstile.c:203 > lock order reversal: > 1st 0xffffffffff80b78ef8 sleepq chain (sleepq chain) @ > kern/subr_turnstile.c:203 > 2nd 0xffffffffff80b52b10 scrlock (scrlock) @ dev/syscons.c:2570 > Sleeping thread (tid 100998, pid 20700) owns a non-sleepable lock > panic: sleeping thread > cpuid = 1 > panic: bufwrite: buffer is not busy??? > cpuid = 1 > > The 2 servers share the same load and panic consistently. I enabled WITNESS > on the 2 in the hope > it would allow the boxes to auto reboot after panic and get extra debug info. > I got debug info > but the servers still hangs after the double panic :(
Try this. Calling vnode_pager_setsize() while holding a mutex is prohibited. On the other hand, I remember that my attempt to add a strict assert that a vnode is exclusively locked in vnode_pager_setsize() had to be reversed because nfs_loadattrcache() sometimes called without vnode lock held. commit 2aa7d15c38b0c01e3f724f04d7ed02ce11c82cc0 Author: Konstantin Belousov <kostik...@gmail.com> Date: Mon Jul 25 11:56:04 2011 +0300 Postpone the vnode_pager_setsize() call until the nfs node mutex is dropped. diff --git a/sys/nfsclient/nfs_subs.c b/sys/nfsclient/nfs_subs.c index 19fde06..351885a 100644 --- a/sys/nfsclient/nfs_subs.c +++ b/sys/nfsclient/nfs_subs.c @@ -478,7 +478,9 @@ nfs_loadattrcache(struct vnode **vpp, struct mbuf **mdp, caddr_t *dposp, struct timespec mtime, mtime_save; int v3 = NFS_ISV3(vp); int error = 0; + int do_setsize; + do_setsize = 0; md = *mdp; t1 = (mtod(md, caddr_t) + md->m_len) - *dposp; cp2 = nfsm_disct(mdp, dposp, NFSX_FATTR(v3), t1, M_WAIT); @@ -606,7 +608,7 @@ nfs_loadattrcache(struct vnode **vpp, struct mbuf **mdp, caddr_t *dposp, np->n_size = vap->va_size; np->n_flag |= NSIZECHANGED; } - vnode_pager_setsize(vp, np->n_size); + do_setsize = 1; } else { np->n_size = vap->va_size; } @@ -643,6 +645,8 @@ nfs_loadattrcache(struct vnode **vpp, struct mbuf **mdp, caddr_t *dposp, KDTRACE_NFS_ATTRCACHE_LOAD_DONE(vp, &np->n_vattr, 0); #endif mtx_unlock(&np->n_mtx); + if (do_setsize) + vnode_pager_setsize(vp, np->n_size); out: #ifdef KDTRACE_HOOKS if (error)
pgpIRWwyIFV50.pgp
Description: PGP signature