In article <20180120144510.ga61...@atom.tmux.org>, Tobias Ulmer <tobi...@tmux.org> wrote: >TLDR: nfs_reqq crrptin, see diff below. > >I've been trying to track down the source of NFS client hangs and >crashes over the last weeks, in order to get back to the things I really >wanted to do... > >No special setup, the NFS Server is an OpenBSD machine serving files >(pkgsrc, src, xsrc) from SSD (= fast replies). gbit net, udp intr >mounts. > >There are any number of bug reports in gnats about this. >1) Can't interrupt hanging process on 'intr' mount. > >2) Client goes dead silent and hangs despite tcpdump showing correct > reply from server. > >3) Client crashes in all places dealing with nfsreq/nfsmount pointers. > Mostly nfs_timer. > >The first was easy to fix. The main loop of nfs_receive() doesn't check >for signals. > >The second issue is looping in nfs_receive, waiting for any reply to >arrive. This is supposed to be fulfilled by nfs_timer re-sending the >request, but investigation shows nfs_timer isn't scheduled because >nfs_reqq is empty(!) > >I'm not 100% certain multiple lwps are racing to add and remove their >requests to the global list. I could never spot contention between them >(using mutex_trylock), but I assume they're doing so. Once insert and >nfsreq removal is locked, crashes were confined to nfs_timer. > >Before one could observe funny situations where nfs_receive() would >return the desired reply but nfs_reply() could not find the outstanding >request in the nfs_reqq, then discarding the valid reply as unexptected >data. > >Finally, the crashes in nfs_timer. Allocating zero'ed nfsreq and >overwriting with junk before freeing will quickly crash the machine. > >Sprinkling a few printfs around, it's clear that nfs_timer (callout, >kernel_lock'ed, blocked from starting by splsoftnet) is giving up its >lock at some point (presumably when calling into pr_send). Lwps >will then modify and free their nfsreq in between nfs_timer >operating on the nfs_reqq. > >PR 40491 (http://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=40491) >basically spells it out in one sentence, it just doesn't go far >enough. Both nfs_reqq and nfsreq can evaporate from under you. > >Below is my attempt at patching this up. It seems a bit heavy handed but >so far it's been 100% reliable (even serving nfs mounts) and perfomance is >unchanged. Contention is low because nfs_timer is mostly blocked from >running by splsoftnet, and it's a recovery thing anyway. >
That is fine as it is! Thanks for working on it, committed. christos