On Tuesday, May 14, 2013 1:15:47 pm Roger Pau Monné wrote: > On 14/05/13 18:31, Konstantin Belousov wrote: > > On Tue, May 14, 2013 at 06:08:45PM +0200, Roger Pau Monn? wrote: > >> On 13/05/13 17:00, Konstantin Belousov wrote: > >>> On Mon, May 13, 2013 at 04:33:04PM +0200, Roger Pau Monn? wrote: > >>>> On 13/05/13 13:18, Roger Pau Monn? wrote: > >> > >> Thanks for taking a look, > >> > >>>> I would like to explain this a little bit more, the syncer process > >>>> doesn't get blocked on the _mtx_trylock_flags_ call, it just continues > >>>> looping forever in what seems to be an endless loop around > >>>> mnt_vnode_next_active/ffs_sync. Also while in this state there is no > >>>> noticeable disk activity, so I'm unsure of what is happening. > >>> How many CPUs does your VM have ? > >> > >> 7 vCPUs, but I've also seen this issue with 4 and 16 vCPUs. > >> > >>> > >>> The loop you describing means that other thread owns the vnode > >>> interlock. Can you track what this thread does ? E.g. look at the > >>> vp->v_interlock.mtx_lock, which is basically a pointer to the struct > >>> thread owning the mutex, clear low bits as needed. Then you can > >>> inspect the thread and get a backtrace. > >> > >> There are no other threads running, only syncer is running on CPU 1 (see > >> ps in previous email). All other CPUs are idle, and as seen from the ps > >> quite a lot of threads are blocked in vnode related operations, either > >> "*Name Cac", "*vnode_fr" or "*vnode in". I've also attached the output > >> of alllocks in the previous email. > > This is not useful. You need to look at the mutex which fails the > > trylock operation in the mnt_vnode_next_active(), see who owns it, > > and then 'unwind' the locking dependencies from there. > > Sorry, now I get it, let's see if I can find the locked vnodes and the > thread that owns them...
You can use 'show lock <address of vp->v_interlock>' to find an owning thread and then use 'show sleepchain <thread>'. If you are using kgdb on the live system (probably easier) then you can grab my scripts at www.freebsd.org/~jhb/gdb/ (do 'cd /path/to/scripts; source gdb6'). You can then find the offending thread and do 'mtx_owner &vp->v_interlock' and then 'sleepchain <tid>' -- John Baldwin _______________________________________________ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
