On Mon, Dec 1, 2014 at 3:08 PM, Chris Mason <c...@fb.com> wrote: > I'm not sure if this is related, but running trinity here, I noticed it > was stuck at 100% system time on every CPU. perf report tells me we are > spending all of our time in spin_lock under the sync system call. > > I think it's coming from contention in the bdi_queue_work() call from > inside sync_inodes_sb, which is spin_lock_bh().
Please do a perf run with -g to get the call chain to make sure.. > I wonder if we're just spinning so hard on this one bh lock that we're > starving the watchdog? If it was that simple, we should see it in the actual soft-lockup stack trace. That said, looking at the bdi_queue_work() function, I don't think you should see any real contention there, although: - spin-lock debugging can make any bad situation about 10x worse by making the spinlocks just that much more horrible from a performance standpoint - the whole "complete(work->done)" thing seems to be pointlessly done inside the spinlock, and that just seems horrible. Do you have a ton of BDI's that might fail that BDI_registered thing? - even the "mod_delayed_work()" is dubious wrt the wb_lock. From what I can tell, the spinlock is supposed to just protect the list. So I think that bdi_queue_work() quite possibly is horribly broken crap and *if* it really is contention on wb_lock, we could rewrite it to not be so bad locking-wise. That said, contention that happens with spinlock debugging enabled really tends to fall under the heading of "that's your own fault". Linus -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/