Public bug reported: I have an environment with Dell R630 servers with RAID controllers with two virtual disks and 22 passthru devices. 2 SAS SSDs and 20 HDDs are setup in 2 bcache cachesets with a resulting 20 mounted xfs filesystems running bcache backending an 11 node swift cluster (one zone has 1 fewer nodes). Two of the zones have these nodes as described above and they appear to be exibiting soft lockups in the bcache thread of the kernel causing other kernel threads to go into i/o blocking state an keeping processes on any bcache from being successful. disk access to the virtual disks mounted with out bcache is still possible when this lockup occurs.
https://pastebin.ubuntu.com/p/mtn47QqBJ3/ There are several softlockup messages found in the dmesg and many of the dumpstack are locked inside the bch_writeback_thread(); static int bch_writeback_thread(void *arg) { [...] while (!kthread_should_stop()) { down_write(&dc->writeback_lock); [...] } One coredump is found when the kswapd is doing the reclaim about the xfs inode cache. __xfs_iflock( struct xfs_inode *ip) { do { prepare_to_wait_exclusive(wq, &wait.wait, TASK_UNINTERRUPTIBLE); if (xfs_isiflocked(ip)) io_schedule(); } while (!xfs_iflock_nowait(ip)); - Possible fix commits: 1). 9baf30972b55 bcache: fix for gc and write-back race https://www.spinics.net/lists/linux-bcache/msg04713.html - Related discussions: 1). Re: [PATCH] md/bcache: Fix a deadlock while calculating writeback rate https://www.spinics.net/lists/linux-bcache/msg04617.html 2). Re: hang during suspend to RAM when bcache cache device is attached https://www.spinics.net/lists/linux-bcache/msg04636.html We are running trusty/mitaka swift storage on these nodes with 4.4.0-111 kernel (linux-image-generic-lts-xenial). ** Affects: linux (Ubuntu) Importance: Undecided Status: Incomplete ** Tags: canonical-bootstack trusty -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1757277 Title: soft lockup from bcache leading to high load and lockup on trusty To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1757277/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
