[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
** Description changed: - $ cat /proc/version_signature + [Impact] + + bcache_allocator() can call the following: + + bch_allocator_thread() + -> bch_prio_write() + -> bch_bucket_alloc() + -> wait on >set->bucket_wait + + But the wake up event on bucket_wait is supposed to come from bch_allocator_thread() itself causing a deadlock. + + [Test Case] + + This is a simple script that can easily trigger the deadlock condition: + https://launchpadlibrarian.net/381282009/bcache-basic-repro.sh + + A better test case has been also provided in bug 1796292 (duplicate of this bug): + https://bugs.launchpad.net/curtin/+bug/1796292/+attachment/5280353/+files/curtin-nvme.sh + + [Fix] + + Fix by making the call to bch_prio_write() non-blocking, so that + bch_allocator_thread() never waits on itself. Moreover, make sure to + wake up the garbage collector thread when bch_prio_write() is failing to + allocate buckets to increase the chance of freeing up more buckets. + + In addition to that it would be safe to also import other upstream + bcache fixes (all clean cherry picks): + + 7e865eba00a3df2dc8c4746173a8ca1c1c7f042e bcache: fix potential deadlock in cached_def_free() + 80265d8dfd77792e133793cef44a21323aac2908 bcache: acquire bch_register_lock later in cached_dev_free() + ce4c3e19e5201424357a0c82176633b32a98d2ec bcache: Replace bch_read_string_list() by __sysfs_match_string() + ecb37ce9baac653cc09e2b631393dde3df82979f bcache: Move couple of functions to sysfs.c + 04cbc21137bfa4d7b8771a5b14f3d6c9b2aee671 bcache: Move couple of string arrays to sysfs.c + 5f2b18ec8e1643410a2369f06888951cdedea0bf bcache: Fix a compiler warning in bcache_device_init() + 20d3a518713e394efa5a899c84574b4b79ec5098 bcache: Reduce the number of sparse complaints about lock imbalances + 42361469ae84c851e40cb1f94c8c9a14cdd94039 bcache: Suppress more warnings about set-but-not-used variables + f0d3814090ac77de94c42b7124c37ece23629197 bcache: Remove an unused variable + 47344e330eabc1515cbe6061eb337100a3ab6d37 bcache: Fix kernel-doc warnings + 9dfbdec7b7fea1ff1b7b5d5d12980dbc7dca46c7 bcache: Annotate switch fall-through + 4a4e443835a43a79113cc237c472c0d268eb1e1c bcache: Add __printf annotation to __bch_check_keys() + fd01991d5c20098c5c1ffc4dca6c821cc60a2f74 bcache: Fix indentation + ca71df31661a0518ed58a1a59cf1993962153ebb bcache: fix using of loop variable in memory shrink + f3641c3abd1da978ee969b0203b71b86ec1bfa93 bcache: fix error return value in memory shrink + 688892b3bc05e25da94866e32210e5f503f16f69 bcache: fix incorrect sysfs output value of strip size + 09a44ca2114737e0932257619c16a2b50c7807f1 bcache: use pr_info() to inform duplicated CACHE_SET_IO_DISABLE set + c4dc2497d50d9c6fb16aa0d07b6a14f3b2adb1e0 bcache: fix high CPU occupancy during journal + a728eacbbdd229d1d903e46261c57d5206f87a4a bcache: add journal statistic + 616486ab52ab7f9739b066d958bdd20e65aefd74 bcache: fix writeback target calc on large devices + 1f0ffa67349c56ea54c03ccfd1e073c990e7411e bcache: only set BCACHE_DEV_WB_RUNNING when cached device attached + eb8cbb6df38f6e5124a3d5f1f8a3dbf519537c60 bcache: improve bcache_reboot() + 9951379b0ca88c95876ad9778b9099e19a95d566 bcache: never writeback a discard operation + + [Regression Potential] + + The upstream fixes are all clean cherry picks from stable (most of them + are small cleanups), so regression potential is minimal. + + The only special patch is "UBUNTU: SAUCE: bcache: fix deadlock in + bcache_allocator()" that is addressing the main deadlock bug (that seems + to be a mainline bug - not fixed yet). We should spend more time trying + to reproduce this deadlock with a mainline kernel and post the patch to + the LKML for review / feedback. + + However, considering that this patch seems to fix/prevent the specific + deadlock problem reported in this bug (tested on the affected platform) + it can be considered safe to apply it. + + [Original Bug Report] + + $ cat /proc/version_signature Ubuntu 4.15.0-29.31-generic 4.15.18 $ lsb_release -rd Description: Ubuntu Cosmic Cuttlefish (development branch) Release: 18.10 $ apt-cache policy linux-image-`uname -r` linux-image-4.15.0-29-generic: - Installed: 4.15.0-29.31 - Candidate: 4.15.0-29.31 - Version table: - *** 4.15.0-29.31 500 - 500 http://archive.ubuntu.com/ubuntu cosmic/main amd64 Packages - 100 /var/lib/dpkg/status + Installed: 4.15.0-29.31 + Candidate: 4.15.0-29.31 + Version table: + *** 4.15.0-29.31 500 + 500 http://archive.ubuntu.com/ubuntu cosmic/main amd64 Packages + 100 /var/lib/dpkg/status - - 3) mkfs.ext4 /dev/bcache0 returns successful creating an ext4 filesystem on top of a bcache device + 3) mkfs.ext4 /dev/bcache0 returns successful creating an ext4 filesystem + on top of a bcache device 4) mkfs.ext4 doesn't return and kernel prints hung process info [ 58.018099] cloud-init[920]: Running command ['mkfs.ext4', '-F', '-L',
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
** Tags added: cscc -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
** This bug is no longer a duplicate of bug 1796292 Tight timeout for bcache removal causes spurious failures ** Changed in: linux (Ubuntu Bionic) Assignee: (unassigned) => Andrea Righi (arighi) ** Changed in: linux (Ubuntu Cosmic) Assignee: (unassigned) => Andrea Righi (arighi) ** Changed in: linux (Ubuntu) Assignee: (unassigned) => Andrea Righi (arighi) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
*** This bug is a duplicate of bug 1796292 *** https://bugs.launchpad.net/bugs/1796292 ** This bug has been marked a duplicate of bug 1796292 Tight timeout for bcache removal causes spurious failures -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
Possibly a duplicate of bug 1796292, since we've got the same hung task timeout trace in bch_bucket_alloc() with the latest test kernel (4.15.0-55-generic #60+lp796292). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
** Changed in: linux (Ubuntu) Status: In Progress => Confirmed ** Changed in: linux (Ubuntu Bionic) Status: In Progress => Confirmed ** Changed in: linux (Ubuntu Cosmic) Status: In Progress => Confirmed ** Changed in: linux (Ubuntu Cosmic) Assignee: Joseph Salisbury (jsalisbury) => (unassigned) ** Changed in: linux (Ubuntu Bionic) Assignee: Joseph Salisbury (jsalisbury) => (unassigned) ** Changed in: linux (Ubuntu) Assignee: Joseph Salisbury (jsalisbury) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
v4.13 shows the same search_free stack as 4.14, but doesn't appear to be fatal (it doesn't result in a hung process nor a block device that cannot be wiped. I suspect we've got a new upstream issue since 4.20 didn't fix things. Re-running 4.20 to see if we had a stack trace, we don't NULL pointer oops, but there is a stuck worker queue: [ 242.887966] kworker/0:47D0 2165 2 0x8000 [ 242.888014] Workqueue: events cache_set_flush [bcache] [ 242.888027] Call Trace: [ 242.888084] __schedule+0x2a2/0x880 [ 242.888097] ? __switch_to+0xfc/0x4f0 [ 242.888105] schedule+0x2c/0x80 [ 242.888113] schedule_timeout+0x246/0x350 [ 242.888122] wait_for_completion+0xb7/0x140 [ 242.888141] ? wake_up_q+0x80/0x80 [ 242.888151] kthread_stop+0x49/0x110 [ 242.888176] cache_set_flush+0xdb/0x190 [bcache] [ 242.888186] process_one_work+0x20f/0x410 [ 242.888194] worker_thread+0x34/0x400 [ 242.888202] kthread+0x120/0x140 [ 242.888209] ? process_one_work+0x410/0x410 [ 242.888216] ? kthread_park+0x90/0x90 [ 242.888236] ret_from_fork+0x35/0x40 [ 242.888435] INFO: task bcache_allocato:9015 blocked for more than 120 seconds. [ 242.893786] Not tainted 4.20.0-042000rc3-generic #201811182231 [ 242.896669] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 242.900428] bcache_allocato D0 9015 2 0x8000 [ 242.900434] Call Trace: [ 242.900448] __schedule+0x2a2/0x880 [ 242.900455] ? __schedule+0x2aa/0x880 [ 242.900462] schedule+0x2c/0x80 [ 242.900480] bch_bucket_alloc+0x19d/0x380 [bcache] [ 242.900503] ? wait_woken+0x80/0x80 [ 242.900519] bch_prio_write+0x190/0x340 [bcache] [ 242.900530] bch_allocator_thread+0x482/0xd10 [bcache] [ 242.900535] kthread+0x120/0x140 [ 242.900546] ? bch_invalidate_one_bucket+0x80/0x80 [bcache] [ 242.900549] ? kthread_park+0x90/0x90 [ 242.900554] ret_from_fork+0x35/0x40 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
I also tested v4.15-rc5, which failed with the same stack trace. Then I returned to 4.14 mainline, and got this one: [ 953.659489] BUG: unable to handle kernel NULL pointer dereference at 0d40 [ 953.668707] IP: search_free+0x31/0x40 [bcache] [ 953.674872] PGD 0 P4D 0 [ 953.677680] Oops: [#34] SMP [ 953.678796] Modules linked in: nls_utf8 isofs nls_iso8859_1 ppdev kvm_intel kvm irqbypass parport_pc joydev input_leds parport serio_raw mac_hid sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi virtio_rng ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear bcache psmouse virtio_scsi virtio_blk virtio_net i2c_piix4 floppy pata_acpi [ 953.700549] CPU: 1 PID: 3132 Comm: kworker/1:229 Tainted: G D 4.14.0-041400-generic #201711122031 [ 953.704522] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014 [ 953.707128] Workqueue: bcache bch_data_insert_keys [bcache] [ 953.708986] task: 985fe605 task.stack: c01402c4c000 [ 953.711147] RIP: 0010:search_free+0x31/0x40 [bcache] [ 953.712997] RSP: 0018:c01402c4fdf8 EFLAGS: 00010282 [ 953.714287] RAX: RBX: 985ffb385e78 RCX: e83580d58e00 [ 953.716883] RDX: 0007 RSI: 0001f540 RDI: 985ffb385e78 [ 953.718731] RBP: c01402c4fe00 R08: 985ff8121800 R09: 0001801f [ 953.720269] R10: c01402c4fd70 R11: 0001 R12: 985fe3670010 [ 953.721583] R13: 985ffb386000 R14: 985fffc9bb40 R15: 985ff83eab40 [ 953.723315] FS: () GS:985fffc8() knlGS: [ 953.725738] CS: 0010 DS: ES: CR0: 80050033 [ 953.727401] CR2: 0d40 CR3: 28c09000 CR4: 06e0 [ 953.729737] Call Trace: [ 953.730687] cached_dev_cache_miss_done+0x38/0x80 [bcache] [ 953.732939] closure_put+0x8b/0xc0 [bcache] [ 953.734647] closure_sub+0x6f/0xc0 [bcache] [ 953.736104] bch_data_insert_keys+0xd8/0x130 [bcache] [ 953.737311] process_one_work+0x1e7/0x410 [ 953.738233] worker_thread+0x32/0x410 [ 953.739246] kthread+0x125/0x140 [ 953.740231] ? process_one_work+0x410/0x410 [ 953.741407] ? kthread_create_on_node+0x70/0x70 [ 953.742631] ret_from_fork+0x25/0x30 [ 953.743431] Code: 48 89 e5 53 48 89 fb e8 0e ff ff ff 48 8b bb 68 01 00 00 48 85 ff 74 05 e8 cd f7 2d e7 48 8b 83 e8 00 00 00 48 89 df 48 8b 40 70 <48> 8b b0 40 0d 00 00 e8 33 51 07 e7 5b 5d c3 0f 1f 44 00 00 55 [ 953.747654] RIP: search_free+0x31/0x40 [bcache] RSP: c01402c4fdf8 [ 953.749962] CR2: 0d40 [ 953.751475] ---[ end trace c40c3beab1ff3af1 ]--- Retesting 4.13 mainline as well. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
It took a while, but v4.15-rc6 fails; so we need to move back a bit. [ 2541.139904] bcache: bcache_device_free() bcache0 stopped [ 2541.141848] BUG: unable to handle kernel NULL pointer dereference at 0d38 [ 2541.143813] IP: search_free+0ex01-b6435-e1/5f0x40 [bcache] [ 2541.145676] PGD 0 P4D 0 [ 2541.146493] Oops: [#58] SMP [ 2541.147509] Modules linked in: bcache nls_utf8 isofs nls_iso8859_1 ppdev kvm_intel kvm irqbypass joydev input_leds parport_pc parport mac_hid serio_raw sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi virtio_rng ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear psmouse virtio_net virtio_scsi virtio_blk i2c_piix4 pata_acpi floppy [ 2541.154248] CPU: 3 PID: 13332 Comm: kworker/3:200 Tainted: G D 4.15.0-041500rc6-generic #201712312330 [ 2541.156921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014 [ 2541.159381] Workqueue: bcache bch_data_insert_keys [bcache] [ 2541.160566] RIP: 0010:search_free+0x31/0x40 [bcache] [ 2541.161300] RSP: 0018:b04e45b6bdf8 EFLAGS: 00010282 [ 2541.162098] RAX: RBX: 942c22b3b390 RCX: 00018016 [ 2541.163791] RDX: 00018017 RSI: ded4c086a7c0 RDI: 942c22b3b390 [ 2541.165486] RBP: b04e45b6be00 R08: 942c21a9fc00 R09: 00018016 [ 2541.167191] R10: b04e45b6bd70 R11: 942c3ffb4000 R12: 942c39610010 [ 2541.168897] R13: 942c22b3b518 R14: 942c3fda3080 R15: 942c26b526c0 [ 2541.170637] FS: () GS:942c3fd8() knlGS: [ 2541.172574] CS: 0010 DS: ES: CR0: 80050033 [ 2541.174043] CR2: 0d38 CR3: 11409000 CR4: 06e0 [ 2541.175847] Call Trace: [ 2541.176349] cached_dev_cache_miss_done+0x38/0x80 [bcache] [ 2541.177131] closure_put+0x8b/0xc0 [bcache] [ 2541.177791] closure_sub+0x6f/0xc0 [bcache] [ 2541.178924] bch_data_insert_keys+0xd8/0x130 [bcache] [ 2541.180107] process_one_work+0x1ea/0x410 [ 2541.181176] worker_thread+0x32/0x410 [ 2541.182157] kthread+0x11e/0x140 [ 2541.183043] ? process_one_work+0x410/0x410 [ 2541.184154] ? kthread_create_worker_on_cpu+0x70/0x70 [ 2541.185460] ret_from_fork+0x1f/0x30 [ 2541.186397] Code: 48 89 e5 53 48 89 fb e8 0e ff ff ff 48 8b bb 68 01 00 00 48 85 ff 74 05 e8 cd 0b d0 f5 48 8b 83 e8 00 00 00 48 89 df 48 8b 40 70 <48> 8b b0 38 0d 00 00 e8 c3 18 a9 f5 5b 5d c3 0f 1f 44 00 00 55 [ 2541.189920] RIP: search_free+0x31/0x40 [bcache] RSP: b04e45b6bdf8 [ 2541.191636] CR2: 0d38 [ 2541.192503] ---[ end trace e2b7818e7472ac25 ]--- On Mon, Nov 26, 2018 at 8:26 PM Joseph Salisbury wrote: > > Can you confirm that v4.15-rc6 does not exhibit the bug? That will > ensure I'm reviewing the correct commits: > > http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15-rc6/ > > -- > You received this bug notification because you are subscribed to the bug > report. > https://bugs.launchpad.net/bugs/1784665 > > Title: > mkfs.ext4 over /dev/bcache0 hangs > > To manage notifications about this bug go to: > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
Can you confirm that v4.15-rc6 does not exhibit the bug? That will ensure I'm reviewing the correct commits: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15-rc6/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
The 4.15.0-39 kernel FAILs: + make-bcache -C /dev/sdb Can't open dev /dev/sdb: Device or resource busy + exit 1 Successful runs: 24 root@ubuntu:~# [ 242.655013] INFO: task kworker/3:103:2610 blocked for more than 120 seconds. [ 242.665486] Not tainted 4.15.0-39-generic #43~lp1784665 [ 242.672393] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 242.682480] INFO: task bcache_allocato:5864 blocked for more than 120 seconds. [ 242.698149] Not tainted 4.15.0-39-generic #43~lp1784665 [ 242.705883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
I built a new Bionic test kernel. The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1784665 Can you test this kernel and see if it resolves this bug? Note about installing test kernels: • If the test kernel is prior to 4.15(Bionic) you need to install the linux-image and linux-image-extra .deb packages. • If the test kernel is 4.15(Bionic) or newer, you need to install the linux-modules, linux-modules-extra and linux-image-unsigned .deb packages. Thanks in advance! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
4.20-rc3 also fails. Successful runs: 95 root@ubuntu:~# [ 484.565298] INFO: task kworker/3:206:2091 blocked for more than 120 seconds. [ 484.566976] Not tainted 4.20.0-042000rc3-generic #201811182231 [ 484.568069] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 484.569579] INFO: task bcache_allocato:16294 blocked for more than 120 seconds. [ 484.571275] Not tainted 4.20.0-042000rc3-generic #201811182231 [ 484.572300] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
The 4.20 kernel is now available. Can you give v4.20-rc3 a test: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.20-rc3 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
** Tags removed: kernel-key ** Tags added: kernel-da-key -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
Thanks for testing. I was hoping a fix would be in 4.19, but I'll dig some more. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
On Wed, Aug 29, 2018 at 12:01 PM Joseph Salisbury wrote: > > Thanks for digging up this additional information. I'll investigate > further. While I do that, the v4.19-rc1 kernel is now out. It might be > worthwhile to give that one a go, and see if the fix was already > committed to mainline: > > http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19-rc1/ v4.19 also hangs like 4.16+ does, but does not oops like 4.15. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
Thanks for digging up this additional information. I'll investigate further. While I do that, the v4.19-rc1 kernel is now out. It might be worthwhile to give that one a go, and see if the fix was already committed to mainline: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19-rc1/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
https://www.spinics.net/lists/linux-bcache/msg04869.html
https://www.spinics.net/lists/linux-bcache/msg05774.html
On Tue, Aug 28, 2018 at 2:30 PM Ryan Harper wrote:
>
> It's always been a locking issue, so either misuse (or missing) locks
> in the bcache attach/detach path, or generic locking changes to block
> layer path I'd think.
>
> I'll see if I can find any hits on those oops tracebacks too
>
> On Tue, Aug 28, 2018 at 2:29 PM Ryan Harper wrote:
> >
> > https://patchwork.kernel.org/patch/10094201/
> >
> > diff --git a/arch/arm64/boot/dts/rockchip/rk3328-rock64.dts
> > b/arch/arm64/boot/dts/rockchip/rk3328-rock64.dts
> > index d4f80786e7c2..3890468678ce 100644
> > --- a/arch/arm64/boot/dts/rockchip/rk3328-rock64.dts
> > +++ b/arch/arm64/boot/dts/rockchip/rk3328-rock64.dts
> > @@ -132,6 +132,8 @@
> > assigned-clocks = < SCLK_MAC2IO>, < SCLK_MAC2IO_EXT>;
> > assigned-clock-parents = <_clkin>, <_clkin>;
> > clock_in_out = "input";
> > + /* shows instability at 1GBit right now */
> > + max-speed = <100>;
> > phy-supply = <_io>;
> > phy-mode = "rgmii";
> > pinctrl-names = "default";
> >
> >
> > That can't be it, which means maybe some of the PASSED didn't pass; I
> > just didn't wait long enough?
> > On Tue, Aug 28, 2018 at 2:11 PM Joseph Salisbury
> > wrote:
> > >
> > > The bisect reported the following commit as the first bad commit:
> > >
> > > bc631943faba ("arm64: dts: rockchip: limit rk3328-rock64 gmac speed to
> > > 100MBit for now")
> > >
> > > It's strange that this is an arm64 commit. I'll investigate a bit and
> > > see why the bisect came back with this result.
> > >
> > > --
> > > You received this bug notification because you are subscribed to the bug
> > > report.
> > > https://bugs.launchpad.net/bugs/1784665
> > >
> > > Title:
> > > mkfs.ext4 over /dev/bcache0 hangs
> > >
> > > To manage notifications about this bug go to:
> > > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1784665
Title:
mkfs.ext4 over /dev/bcache0 hangs
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
It's always been a locking issue, so either misuse (or missing) locks
in the bcache attach/detach path, or generic locking changes to block
layer path I'd think.
I'll see if I can find any hits on those oops tracebacks too
On Tue, Aug 28, 2018 at 2:29 PM Ryan Harper wrote:
>
> https://patchwork.kernel.org/patch/10094201/
>
> diff --git a/arch/arm64/boot/dts/rockchip/rk3328-rock64.dts
> b/arch/arm64/boot/dts/rockchip/rk3328-rock64.dts
> index d4f80786e7c2..3890468678ce 100644
> --- a/arch/arm64/boot/dts/rockchip/rk3328-rock64.dts
> +++ b/arch/arm64/boot/dts/rockchip/rk3328-rock64.dts
> @@ -132,6 +132,8 @@
> assigned-clocks = < SCLK_MAC2IO>, < SCLK_MAC2IO_EXT>;
> assigned-clock-parents = <_clkin>, <_clkin>;
> clock_in_out = "input";
> + /* shows instability at 1GBit right now */
> + max-speed = <100>;
> phy-supply = <_io>;
> phy-mode = "rgmii";
> pinctrl-names = "default";
>
>
> That can't be it, which means maybe some of the PASSED didn't pass; I
> just didn't wait long enough?
> On Tue, Aug 28, 2018 at 2:11 PM Joseph Salisbury
> wrote:
> >
> > The bisect reported the following commit as the first bad commit:
> >
> > bc631943faba ("arm64: dts: rockchip: limit rk3328-rock64 gmac speed to
> > 100MBit for now")
> >
> > It's strange that this is an arm64 commit. I'll investigate a bit and
> > see why the bisect came back with this result.
> >
> > --
> > You received this bug notification because you are subscribed to the bug
> > report.
> > https://bugs.launchpad.net/bugs/1784665
> >
> > Title:
> > mkfs.ext4 over /dev/bcache0 hangs
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1784665
Title:
mkfs.ext4 over /dev/bcache0 hangs
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
https://patchwork.kernel.org/patch/10094201/
diff --git a/arch/arm64/boot/dts/rockchip/rk3328-rock64.dts
b/arch/arm64/boot/dts/rockchip/rk3328-rock64.dts
index d4f80786e7c2..3890468678ce 100644
--- a/arch/arm64/boot/dts/rockchip/rk3328-rock64.dts
+++ b/arch/arm64/boot/dts/rockchip/rk3328-rock64.dts
@@ -132,6 +132,8 @@
assigned-clocks = < SCLK_MAC2IO>, < SCLK_MAC2IO_EXT>;
assigned-clock-parents = <_clkin>, <_clkin>;
clock_in_out = "input";
+ /* shows instability at 1GBit right now */
+ max-speed = <100>;
phy-supply = <_io>;
phy-mode = "rgmii";
pinctrl-names = "default";
That can't be it, which means maybe some of the PASSED didn't pass; I
just didn't wait long enough?
On Tue, Aug 28, 2018 at 2:11 PM Joseph Salisbury
wrote:
>
> The bisect reported the following commit as the first bad commit:
>
> bc631943faba ("arm64: dts: rockchip: limit rk3328-rock64 gmac speed to
> 100MBit for now")
>
> It's strange that this is an arm64 commit. I'll investigate a bit and
> see why the bisect came back with this result.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1784665
>
> Title:
> mkfs.ext4 over /dev/bcache0 hangs
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1784665
Title:
mkfs.ext4 over /dev/bcache0 hangs
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
The bisect reported the following commit as the first bad commit:
bc631943faba ("arm64: dts: rockchip: limit rk3328-rock64 gmac speed to
100MBit for now")
It's strange that this is an arm64 commit. I'll investigate a bit and
see why the bisect came back with this result.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1784665
Title:
mkfs.ext4 over /dev/bcache0 hangs
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
FAILED: Kernel 4.15-rc1 up to bc631943faba6fc3f755748091ada31798fb7d50 Attached is the oops of the failure. ** Attachment added: "rc1-bisect7-oops.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+attachment/5181823/+files/rc1-bisect7-oops.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
I built the next test kernel, up to the following commit: bc631943faba6fc3f755748091ada31798fb7d50 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1784665 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results. Thanks in advance -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
PASS: Kernel 4.15-rc1 (bisect6) up to 87eba0716011e528f7841026f2cc65683219d0ad -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
I built the next test kernel, up to the following commit: 87eba0716011e528f7841026f2cc65683219d0ad The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1784665 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results. Thanks in advance -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
PASS: Kernel 4.15-rc1 (bisect5) up to 6b457409169b7686d293b408da0b6446ccb57a76 I've 2600 seconds of uptime with over 800 loops in the test-case. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
I built the next test kernel, up to the following commit: 6b457409169b7686d293b408da0b6446ccb57a76 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1784665 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results. Thanks in advance -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
FAILED: Kernel 4.15-rc1 up to f48f66a962a54c3d26d688c3df5441c9d1ba8730 Attached is the oops of the failure. ** Attachment added: "rc1-bisect4-oops.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+attachment/5181417/+files/rc1-bisect4-oops.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
I built the next test kernel, up to the following commit: f48f66a962a54c3d26d688c3df5441c9d1ba8730 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1784665 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results. Thanks in advance -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
FAILED: Kernel 4.15-rc1 up to f17b9e764dfcd838dab51572d620a371c05a8e60 Attached is the oops of the failure. ** Attachment added: "rc1-bisect3-ops.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+attachment/5178807/+files/rc1-bisect3-ops.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
I built the next test kernel, up to the following commit: f17b9e764dfcd838dab51572d620a371c05a8e60 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1784665 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results. Thanks in advance -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
test kernel 4.15-rc1 up to abb62c46d4949d44979fa647740feff3f7538799 FAILED ** Attachment added: "kernel oops for 4.15-rc1 up to abb62c46d4949d44979fa647740feff3f7538799" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+attachment/5175947/+files/rc1-bisect2-oops.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
I built the next test kernel, up to the following commit: abb62c46d4949d44979fa647740feff3f7538799 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1784665 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results. Thanks in advance -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
test kernel 4.15-rc6 up to d8887f1c7289848e74c92bd4322789a9cd7de699 FAILED ** Attachment added: "kernel oops for 4.15-rc6-up-to-d8887f1c7289848e74c92bd4322789a9cd7de699" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+attachment/5175379/+files/rc6-bisect1-oops.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
test kernel 4.15-rc6 up to d8887f1c7289848e74c92bd4322789a9cd7de699 FAILED Attaching oops. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
I started a kernel bisect between v4.15-rc6 and v4.15-rc7. The kernel bisect will require testing of about 7-10 test kernels. I built the first test kernel, up to the following commit: d8887f1c7289848e74c92bd4322789a9cd7de699 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1784665 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results. Thanks in advance -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
** Also affects: linux (Ubuntu Bionic) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Bionic) Status: New => In Progress ** Changed in: linux (Ubuntu Cosmic) Status: Confirmed => In Progress ** Changed in: linux (Ubuntu Bionic) Importance: Undecided => High ** Changed in: linux (Ubuntu Bionic) Assignee: (unassigned) => Joseph Salisbury (jsalisbury) ** Changed in: linux (Ubuntu Cosmic) Assignee: (unassigned) => Joseph Salisbury (jsalisbury) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
4.14 Final: OK 4.15-rc1:OK I started a further bisect: 4.15-rc9:FAIL 4.15-rc5:OK 4.15-rc7:FAIL 4.15-rc6:OK So, it appears something between rc6 (last known good) and rc7 (FAIL) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
To narrow it down further, can you test the following two kernels: 4.14 Final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.14/ 4.15-rc1: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15-rc1/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
4.10: Status: OK Comment: Survived 30 minute loop over recreating with no issues. 4.13: Status: OK Comment: Survived 30 minute loop over recreating with no issues. 4.15: Status: FAIL Comment: triggered within less than a minute of the loop. Kernel Trace: [ 122.500943] Oops: [#24] SMP PTI [ 122.501612] Modules linked in: bcache nls_utf8 isofs nls_iso8859_1 ppdev kvm_intel kvm irqbypass parport_pc input_leds parport joydev mac_hid serio_raw sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi virtio_rng ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear psmouse virtio_net virtio_blk virtio_scsi i2c_piix4 pata_acpi floppy [ 122.508306] CPU: 3 PID: 2212 Comm: kworker/3:249 Tainted: G D 4.15.0-041500-generic #201802011154 [ 122.510459] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014 [ 122.512541] Workqueue: bcache bch_data_insert_keys [bcache] [ 122.513779] RIP: 0010:__bch_bset_search+0x80/0x470 [bcache] [ 122.515015] RSP: 0018:b49141a83c10 EFLAGS: 00010286 [ 122.516127] RAX: fff6cacc RBX: fff6cacc RCX: [ 122.516947] RDX: b49141a83e18 RSI: 0069 RDI: 8f6a7ad64cc8 [ 122.517768] RBP: b49141a83c70 R08: 0001 R09: 8f6a7ad64ce0 [ 122.518586] R10: 8f6a6735ab80 R11: 00027ec0 R12: b49141a83e18 [ 122.519858] R13: 000400656369 R14: 66643165 R15: 8f6a7ad64ce0 [ 122.521302] FS: () GS:8f6a7fd8() knlGS: [ 122.522272] CS: 0010 DS: ES: CR0: 80050033 [ 122.522969] CR2: 000400656389 CR3: 38fc2000 CR4: 06e0 [ 122.524521] Call Trace: [ 122.525153] ? ttwu_do_activate+0x7a/0x90 [ 122.526147] __bch_btree_iter_init+0x5d/0xc0 [bcache] [ 122.527411] bch_btree_iter_init+0x12/0x20 [bcache] [ 122.528578] bch_btree_map_nodes_recurse+0x93/0x190 [bcache] [ 122.529904] ? __slab_free+0x14d/0x2c0 [ 122.530814] ? bch_btree_insert_check_key+0x1f0/0x1f0 [bcache] [ 122.532331] __bch_btree_map_nodes+0x142/0x1a0 [bcache] [ 122.533685] ? bch_btree_insert_check_key+0x1f0/0x1f0 [bcache] [ 122.534994] bch_btree_insert+0xf9/0x170 [bcache] [ 122.536059] ? wait_woken+0x80/0x80 [ 122.536876] bch_data_insert_keys+0x36/0x130 [bcache] [ 122.538004] process_one_work+0x1ef/0x410 [ 122.538928] worker_thread+0x32/0x410 [ 122.539779] kthread+0x121/0x140 [ 122.540538] ? process_one_work+0x410/0x410 [ 122.541487] ? kthread_create_worker_on_cpu+0x70/0x70 [ 122.542619] ret_from_fork+0x35/0x40 [ 122.543547] Code: 41 08 81 e3 ff ff 0f 00 25 ff ff 0f 00 48 39 d8 48 89 5d d0 0f 85 b6 03 00 00 48 8b 42 08 49 2b 41 10 48 85 c0 0f 89 7c 03 00 00 <49> 8b 45 20 25 ff ff 0f 00 48 39 45 d0 0f 85 c0 03 00 00 48 8b [ 122.546019] RIP: __bch_bset_search+0x80/0x470 [bcache] RSP: b49141a83c10 [ 122.546836] CR2: 000400656389 [ 122.547567] ---[ end trace c539406c76c0edd2 ]--- -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
Thanks for testing mainline. We should be able to bisect this down if we confirm it's a regression. Can you test the following kernels and report back? We are looking for the first kernel version that exhibits this bug: 4.10 Final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10/ 4.13 Final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.13/ 4.15 Final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15/ You don't have to test every kernel, just up until the kernel that first has this bug. If 4.10 final exhibits the bug, we'll have to test even earlier versions. Thanks in advance! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
I can recreate on upstream: # uname -r; dmesg | tail -n 16 4.18.0-041800rc7-generic [ 967.657492] INFO: task bcache_allocato:14240 blocked for more than 120 seconds. [ 967.664138] Not tainted 4.18.0-041800rc7-generic #201807292230 [ 967.669641] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 967.673503] bcache_allocato D0 14240 2 0x8000 [ 967.673512] Call Trace: [ 967.673529] __schedule+0x29e/0x840 [ 967.673535] ? reschedule_interrupt+0xa/0x20 [ 967.673539] schedule+0x2c/0x80 [ 967.673555] bch_bucket_alloc+0x19d/0x380 [bcache] [ 967.673561] ? wait_woken+0x80/0x80 [ 967.673574] bch_prio_write+0x186/0x330 [bcache] [ 967.673583] bch_allocator_thread+0x482/0xd10 [bcache] [ 967.673589] kthread+0x120/0x140 [ 967.673598] ? bch_invalidate_one_bucket+0x80/0x80 [bcache] [ 967.673601] ? kthread_bind+0x40/0x40 [ 967.673605] ret_from_fork+0x35/0x40 ** Tags added: kernel-bug-exists-upstream ** Changed in: linux (Ubuntu Cosmic) Status: Triaged => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
I'll see if I can reproduce on the mainline kernel. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
I think I've got one. It doesn't always hang the mkfs, but it does wedge bcache in a few ways. [ 484.322546] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 484.326230] INFO: task bcache_allocato:32503 blocked for more than 120 seconds. 1) when it occurs, a backing device never get's released from the kernel, such that a blockdev --rereadpt on the device says the device is busy, but examining /sys/class/block//holders shows nothing 2) attempts to open the device exclusively fail (so things like mkfs or make-bcache -B). The reproducer has two virtio disks, vda, vdb; size doesn't really matter, but they are 10G and 2G respectively. /dev/vdb is a cache set (make-bcache -C /dev/vdb), /dev/vda2 is the backing device (make-bcache -B /dev/vda2) and then they are joined (echo $CSET_UUID > /sys/class/block/vda/vda2/bcache/attach), which results in a /dev/bcache0. Then we mkfs.ext4 it. Then we tear it down, wipe the partitions, the disks and repeat. I'll attach the script. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
** Attachment added: "bcache-basic-repro.sh" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+attachment/5170528/+files/bcache-basic-repro.sh -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
It would be great if we can identify a reproducer. It would make it much easier to track down. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
We've just had a second occurrence of this, same kernel, but with Xenial. (Xenial HWE Edge kernel 4.15.0-29) with the same scenario, mkfs.ext4 a /dev/bcache0. I'll see if I can recreate by stressing things. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
> Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem? We've had plenty of successful runs on this kernel. I suspect io load or something else may trigger this. We've only seen it once out of tens of not hundreds of runs on Cosmic 4.15.0-29-generic. We don't currently have an easy way to test upstream kernels in the automation runs. If we see this again, I'll poke a bit more on running mkfs.ext4 over bcache devices on Cosmic to see if we can recreate outside of the automation. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs
Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem? Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.18 kernel[0]. If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'. If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'. Once testing of the upstream kernel is complete, please mark this bug as "Confirmed". Thanks in advance. [0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.18-rc7 ** Changed in: linux (Ubuntu) Importance: Undecided => High ** Also affects: linux (Ubuntu Cosmic) Importance: High Status: Confirmed ** Tags added: kernel-key ** Changed in: linux (Ubuntu Cosmic) Status: Confirmed => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
