@jason-hobbs, I don't see any kernel error in the log attached to comment #53, so it seems to be a different issue, maybe a performance regression that is causing curtin to timeout. I would keep this bug closed / fixed and open a new one.
If you can easily reproduce it, could you open a new bug and collect some info on the box after the problem happens? (in particular uname -a, dmesg, ps axuw). Thanks! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: bcache: bch_allocator_thread(): hung task timeout Status in linux package in Ubuntu: Fix Released Status in linux source package in Xenial: Fix Committed Status in linux source package in Bionic: Fix Committed Status in linux source package in Disco: Fix Committed Status in linux source package in Eoan: Fix Released Bug description: [Impact] bcache_allocator() can call the following: bch_allocator_thread() -> bch_prio_write() -> bch_bucket_alloc() -> wait on &ca->set->bucket_wait But the wake up event on bucket_wait is supposed to come from bch_allocator_thread() itself causing a deadlock. [Test Case] This is a simple script that can easily trigger the deadlock condition: https://launchpadlibrarian.net/381282009/bcache-basic-repro.sh A better test case has been also provided in bug 1796292 (duplicate of this bug): https://bugs.launchpad.net/curtin/+bug/1796292/+attachment/5280353/+files/curtin-nvme.sh [Fix] Fix by making the call to bch_prio_write() non-blocking, so that bch_allocator_thread() never waits on itself. Moreover, make sure to wake up the garbage collector thread when bch_prio_write() is failing to allocate buckets to increase the chance of freeing up more buckets. In addition to that it would be safe to also import other upstream bcache fixes (all clean cherry picks): 7e865eba00a3df2dc8c4746173a8ca1c1c7f042e bcache: fix potential deadlock in cached_def_free() 80265d8dfd77792e133793cef44a21323aac2908 bcache: acquire bch_register_lock later in cached_dev_free() ce4c3e19e5201424357a0c82176633b32a98d2ec bcache: Replace bch_read_string_list() by __sysfs_match_string() ecb37ce9baac653cc09e2b631393dde3df82979f bcache: Move couple of functions to sysfs.c 04cbc21137bfa4d7b8771a5b14f3d6c9b2aee671 bcache: Move couple of string arrays to sysfs.c 5f2b18ec8e1643410a2369f06888951cdedea0bf bcache: Fix a compiler warning in bcache_device_init() 20d3a518713e394efa5a899c84574b4b79ec5098 bcache: Reduce the number of sparse complaints about lock imbalances 42361469ae84c851e40cb1f94c8c9a14cdd94039 bcache: Suppress more warnings about set-but-not-used variables f0d3814090ac77de94c42b7124c37ece23629197 bcache: Remove an unused variable 47344e330eabc1515cbe6061eb337100a3ab6d37 bcache: Fix kernel-doc warnings 9dfbdec7b7fea1ff1b7b5d5d12980dbc7dca46c7 bcache: Annotate switch fall-through 4a4e443835a43a79113cc237c472c0d268eb1e1c bcache: Add __printf annotation to __bch_check_keys() fd01991d5c20098c5c1ffc4dca6c821cc60a2f74 bcache: Fix indentation ca71df31661a0518ed58a1a59cf1993962153ebb bcache: fix using of loop variable in memory shrink f3641c3abd1da978ee969b0203b71b86ec1bfa93 bcache: fix error return value in memory shrink 688892b3bc05e25da94866e32210e5f503f16f69 bcache: fix incorrect sysfs output value of strip size 09a44ca2114737e0932257619c16a2b50c7807f1 bcache: use pr_info() to inform duplicated CACHE_SET_IO_DISABLE set c4dc2497d50d9c6fb16aa0d07b6a14f3b2adb1e0 bcache: fix high CPU occupancy during journal a728eacbbdd229d1d903e46261c57d5206f87a4a bcache: add journal statistic 616486ab52ab7f9739b066d958bdd20e65aefd74 bcache: fix writeback target calc on large devices 1f0ffa67349c56ea54c03ccfd1e073c990e7411e bcache: only set BCACHE_DEV_WB_RUNNING when cached device attached eb8cbb6df38f6e5124a3d5f1f8a3dbf519537c60 bcache: improve bcache_reboot() 9951379b0ca88c95876ad9778b9099e19a95d566 bcache: never writeback a discard operation [Regression Potential] The upstream fixes are all clean cherry picks from stable (most of them are small cleanups), so regression potential is minimal. The only special patch is "UBUNTU: SAUCE: bcache: fix deadlock in bcache_allocator()" that is addressing the main deadlock bug (that seems to be a mainline bug - not fixed yet). We should spend more time trying to reproduce this deadlock with a mainline kernel and post the patch to the LKML for review / feedback. However, considering that this patch seems to fix/prevent the specific deadlock problem reported in this bug (tested on the affected platform) it can be considered safe to apply it. [Original Bug Report] $ cat /proc/version_signature Ubuntu 4.15.0-29.31-generic 4.15.18 $ lsb_release -rd Description: Ubuntu Cosmic Cuttlefish (development branch) Release: 18.10 $ apt-cache policy linux-image-`uname -r` linux-image-4.15.0-29-generic: Installed: 4.15.0-29.31 Candidate: 4.15.0-29.31 Version table: *** 4.15.0-29.31 500 500 http://archive.ubuntu.com/ubuntu cosmic/main amd64 Packages 100 /var/lib/dpkg/status 3) mkfs.ext4 /dev/bcache0 returns successful creating an ext4 filesystem on top of a bcache device 4) mkfs.ext4 doesn't return and kernel prints hung process info [ 58.018099] cloud-init[920]: Running command ['mkfs.ext4', '-F', '-L', 'root-fs', '-U', 'f01aec97-9457-11e8-b8d6-525400123401', '/dev/bcache0'] with allowed return codes [0] (capture=True) [ 242.652018] INFO: task kworker/u4:0:5 blocked for more than 120 seconds. [ 242.653767] Tainted: P O 4.15.0-29-generic #31-Ubuntu [ 242.655391] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 242.657397] INFO: task kworker/0:2:410 blocked for more than 120 seconds. [ 242.659126] Tainted: P O 4.15.0-29-generic #31-Ubuntu [ 242.660980] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 242.663000] INFO: task bcache_allocato:2326 blocked for more than 120 seconds. [ 242.664807] Tainted: P O 4.15.0-29-generic #31-Ubuntu [ 242.666516] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 242.668503] INFO: task bcache_writebac:2345 blocked for more than 120 seconds. [ 242.670301] Tainted: P O 4.15.0-29-generic #31-Ubuntu [ 242.671936] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 242.673909] INFO: task mkfs.ext4:2803 blocked for more than 120 seconds. [ 242.675414] Tainted: P O 4.15.0-29-generic #31-Ubuntu [ 242.677038] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 363.483998] INFO: task kworker/u4:0:5 blocked for more than 120 seconds. [ 363.488441] Tainted: P O 4.15.0-29-generic #31-Ubuntu [ 363.489598] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 363.491043] INFO: task kworker/0:2:410 blocked for more than 120 seconds. [ 363.492252] Tainted: P O 4.15.0-29-generic #31-Ubuntu [ 363.494085] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 363.495659] INFO: task bcache_allocato:2326 blocked for more than 120 seconds. [ 363.496957] Tainted: P O 4.15.0-29-generic #31-Ubuntu [ 363.498454] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 363.499866] INFO: task bcache_writebac:2345 blocked for more than 120 seconds. [ 363.501156] Tainted: P O 4.15.0-29-generic #31-Ubuntu [ 363.502597] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 363.504048] INFO: task mkfs.ext4:2803 blocked for more than 120 seconds. [ 363.505505] Tainted: P O 4.15.0-29-generic #31-Ubuntu [ 363.506677] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. System has two virtio block devices. bcache was created like so: make-bcache -C /dev/vdb make-bcache -B /dev/vda2 resulting in /dev/bcache0 ProblemType: Bug DistroRelease: Ubuntu 18.10 Package: linux-image-4.15.0-29-generic 4.15.0-29.31 ProcVersionSignature: User Name 4.15.0-29.31-generic 4.15.18 Uname: Linux 4.15.0-29-generic x86_64 AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Jul 31 15:52 seq crw-rw---- 1 root audio 116, 33 Jul 31 15:52 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay' ApportVersion: 2.20.10-0ubuntu7 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A Date: Tue Jul 31 15:53:56 2018 IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig' Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub MachineType: QEMU Standard PC (i440FX + PIIX, 1996) PciMultimedia: ProcEnviron: TERM=xterm PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=C.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-29-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0 RelatedPackageVersions: linux-restricted-modules-4.15.0-29-generic N/A linux-backports-modules-4.15.0-29-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 04/01/2014 dmi.bios.vendor: SeaBIOS dmi.bios.version: 1.11.1-1 dmi.chassis.type: 1 dmi.chassis.vendor: QEMU dmi.chassis.version: pc-i440fx-bionic dmi.modalias: dmi:bvnSeaBIOS:bvr1.11.1-1:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-bionic:cvnQEMU:ct1:cvrpc-i440fx-bionic: dmi.product.name: Standard PC (i440FX + PIIX, 1996) dmi.product.version: pc-i440fx-bionic dmi.sys.vendor: QEMU To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
