Hi Benjamin, I think Dan is onto something.
The following commit was merged in upstream 5.5-rc1, but was backported to 4.15.0-87-generic through upstream stable: commit 9fcc34b1a6dd4b8e5337e2b6ef45e428897eca6b Author: Coly Li <col...@suse.de> Date: Wed Nov 13 16:03:24 2019 +0800 Subject: bcache: at least try to shrink 1 node in bch_mca_scan() Link: https://github.com/torvalds/linux/commit/9fcc34b1a6dd4b8e5337e2b6ef45e428897eca6b It mentions in the description that: > If sc->nr_to_scan is smaller than c->btree_pages, after the above > calculation, variable 'nr' will be 0 and nothing will be shrunk. It is > frequeently observed that only 1 or 2 is set to sc->nr_to_scan and make > nr to be zero. Then bch_mca_scan() will do nothing more then acquiring > and releasing mutex c->bucket_lock. Spending a lot of idle time in bch_mca_scan() waiting on locks seems to be a key symptom from the logs you have gathered. I went and had a look through the commit history, and I think these three patches might solve the problem: commit 125d98edd11464c8e0ec9eaaba7d682d0f832686 Author: Coly Li <col...@suse.de> Date: Fri Jan 24 01:01:40 2020 +0800 Subject: bcache: remove member accessed from struct btree Link: https://github.com/torvalds/linux/commit/125d98edd11464c8e0ec9eaaba7d682d0f832686 commit d5c9c470b01177e4d90cdbf178b8c7f37f5b8795 Author: Coly Li <col...@suse.de> Date: Fri Jan 24 01:01:41 2020 +0800 Subject: bcache: reap c->btree_cache_freeable from the tail in bch_mca_scan() Link: https://github.com/torvalds/linux/commit/d5c9c470b01177e4d90cdbf178b8c7f37f5b8795 commit e3de04469a49ee09c89e80bf821508df458ccee6 Author: Coly Li <col...@suse.de> Date: Fri Jan 24 01:01:42 2020 +0800 Subject: bcache: reap from tail of c->btree_cache in bch_mca_scan() Link: Link: https://github.com/torvalds/linux/commit/e3de04469a49ee09c89e80bf821508df458ccee6 The first commit is a dependency of the other two. The first commit removes a "recently accessed" marker, used to indicate if a particular cache has been used recently, and if it has been, not consider it for cache eviction. The commit mentions that under heavy IO, all caches will end up being recently accessed, and nothing will ever be shrunk. The second commit changes a previous design decision of skipping the first 3 caches to shrink, since it is a common case to call bch_mca_scan() with nr being 1, or 2, just like 0 was common in the very first commit I mentioned. This time, if we land on 1 or 2, the loop exits and nothing happens, and we waste time waiting on locks, just like the very first commit I mentioned. The fix is to try shrink caches from the tail of the list, and not the beginning. The third commit fixes a minor issue where we don't want to re-arrange the linked list, which is what the second commit ended up doing, and instead, just shrink the cache at the end of the linked list, and not change the order. All three commits are clean cherry picks to 4.15.0-118-generic, and I have got a test kernel building for you in: https://launchpad.net/~mruffell/+archive/ubuntu/sf294907-test It should be ready in a few hours time. Make sure the build has finished before trying to install. Please note this package is NOT SUPPORTED by Canonical, and is for TESTING PURPOSES ONLY. ONLY Install in a dedicated test environment. Instructions to install (on a Bionic system): 1) sudo add-apt-repository ppa:mruffell/sf294907-test 2) sudo apt update 3) sudo apt install linux-image-unsigned-4.15.0-118-generic linux-modules-4.15.0-118-generic \ linux-modules-extra-4.15.0-118-generic linux-headers-4.15.0-118 linux-headers-4.15.0-118-generic 4) sudo reboot 5) uname -rv The uname string should be along the lines of 4.15.0-118.119+TEST294907v20201009b1. You may need to change your grub settings if you don't boot into the correct kernel. If you can, schedule some downtime and install the test kernel. Enable bcache shrinking and try reproduce the problem. Hopefully the test kernel is an improvement and fixes your issues. Let me know how the test kernel goes. Thanks, Matthew -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1898786 Title: Issue with bcache bch_mca_scan causing huge IO wait To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1898786/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
