[Kernel-packages] [Bug 1980925] Re: [SRU] bcache deadlock during read IO in writeback mode
This bug is awaiting verification that the linux-mtk/5.15.0-1030.34 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-mtk' to 'verification-done-jammy- linux-mtk'. If the problem still exists, change the tag 'verification- needed-jammy-linux-mtk' to 'verification-failed-jammy-linux-mtk'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: kernel-spammed-jammy-linux-mtk-v2 verification-needed-jammy-linux-mtk -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1980925 Title: [SRU] bcache deadlock during read IO in writeback mode Status in linux package in Ubuntu: Invalid Status in linux source package in Focal: Invalid Status in linux source package in Jammy: Fix Released Bug description: SRU Justification: [Impact] When Random Read I/O is started with a test like - fio --name=read_iops --directory=/home/ubuntu/bcache_mount/ --size=16G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --randrepeat=0 or random read-writes with a test like, fio --filename=/home/ubuntu/bcache_mount/cachedfile --size=15GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=128 --name=iops-test-job --randrepeat=0 traces are seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] The bug exists till kernel 5.15.50-051550-generic The reproducer is pasted below: # uname -a Linux bronzor 5.15.50-051550-generic #202206251445 SMP Sat Jun 25 14:51:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sdd 8:48 0 279.4G 0 disk └─sdd18:49 060G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount nvme0n1 259:00 372.6G 0 disk └─nvme0n1p1 259:2015G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount fio --name=read_iops --directory=/home/ubuntu/bcache_mount --size=12G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --group_reporting=1 read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.28 Starting 1 process read_iops: Laying out IO file (1 file / 12288MiB) The test does not progress beyond a few minutes, and this trace is then seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] [Fix] These 3 fixes are needed for the SRU. dea3560e5f31965165bcf34ecf0b47af28bfd155, 6445ec3df23f24677064a327dce437ef3e02dc6, dc60301fb408e06e0b718c0980cdd31d2b238bee I have built these fixes into kernel 5.15.0-39-generic (jammy) and tested to verify the problem is fixed. [Regression Potential] Regression potential should be minimal. I have not seen any potential drawbacks or harmful effects of this fix in my testing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1980925/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages
[Kernel-packages] [Bug 1980925] Re: [SRU] bcache deadlock during read IO in writeback mode
I have verified that this issue is fixed in the jammy kernel 5.15.0-48-generic. ** Tags added: verification-done-jammy -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1980925 Title: [SRU] bcache deadlock during read IO in writeback mode Status in linux package in Ubuntu: Invalid Status in linux source package in Focal: Invalid Status in linux source package in Jammy: Fix Released Bug description: SRU Justification: [Impact] When Random Read I/O is started with a test like - fio --name=read_iops --directory=/home/ubuntu/bcache_mount/ --size=16G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --randrepeat=0 or random read-writes with a test like, fio --filename=/home/ubuntu/bcache_mount/cachedfile --size=15GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=128 --name=iops-test-job --randrepeat=0 traces are seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] The bug exists till kernel 5.15.50-051550-generic The reproducer is pasted below: # uname -a Linux bronzor 5.15.50-051550-generic #202206251445 SMP Sat Jun 25 14:51:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sdd 8:48 0 279.4G 0 disk └─sdd18:49 060G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount nvme0n1 259:00 372.6G 0 disk └─nvme0n1p1 259:2015G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount fio --name=read_iops --directory=/home/ubuntu/bcache_mount --size=12G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --group_reporting=1 read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.28 Starting 1 process read_iops: Laying out IO file (1 file / 12288MiB) The test does not progress beyond a few minutes, and this trace is then seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] [Fix] These 3 fixes are needed for the SRU. dea3560e5f31965165bcf34ecf0b47af28bfd155, 6445ec3df23f24677064a327dce437ef3e02dc6, dc60301fb408e06e0b718c0980cdd31d2b238bee I have built these fixes into kernel 5.15.0-39-generic (jammy) and tested to verify the problem is fixed. [Regression Potential] Regression potential should be minimal. I have not seen any potential drawbacks or harmful effects of this fix in my testing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1980925/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1980925] Re: [SRU] bcache deadlock during read IO in writeback mode
This bug is awaiting verification that the linux- gkeop-5.15/5.15.0-1003.5~20.04.2 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1980925 Title: [SRU] bcache deadlock during read IO in writeback mode Status in linux package in Ubuntu: Invalid Status in linux source package in Focal: Invalid Status in linux source package in Jammy: Fix Released Bug description: SRU Justification: [Impact] When Random Read I/O is started with a test like - fio --name=read_iops --directory=/home/ubuntu/bcache_mount/ --size=16G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --randrepeat=0 or random read-writes with a test like, fio --filename=/home/ubuntu/bcache_mount/cachedfile --size=15GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=128 --name=iops-test-job --randrepeat=0 traces are seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] The bug exists till kernel 5.15.50-051550-generic The reproducer is pasted below: # uname -a Linux bronzor 5.15.50-051550-generic #202206251445 SMP Sat Jun 25 14:51:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sdd 8:48 0 279.4G 0 disk └─sdd18:49 060G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount nvme0n1 259:00 372.6G 0 disk └─nvme0n1p1 259:2015G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount fio --name=read_iops --directory=/home/ubuntu/bcache_mount --size=12G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --group_reporting=1 read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.28 Starting 1 process read_iops: Laying out IO file (1 file / 12288MiB) The test does not progress beyond a few minutes, and this trace is then seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] [Fix] These 3 fixes are needed for the SRU. dea3560e5f31965165bcf34ecf0b47af28bfd155, 6445ec3df23f24677064a327dce437ef3e02dc6, dc60301fb408e06e0b718c0980cdd31d2b238bee I have built these fixes into kernel 5.15.0-39-generic (jammy) and tested to verify the problem is fixed. [Regression Potential] Regression potential should be minimal. I have not seen any potential drawbacks or harmful effects of this fix in my testing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1980925/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help :
[Kernel-packages] [Bug 1980925] Re: [SRU] bcache deadlock during read IO in writeback mode
This bug is awaiting verification that the linux-nvidia/5.15.0-1007.7 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1980925 Title: [SRU] bcache deadlock during read IO in writeback mode Status in linux package in Ubuntu: Invalid Status in linux source package in Focal: Invalid Status in linux source package in Jammy: Fix Released Bug description: SRU Justification: [Impact] When Random Read I/O is started with a test like - fio --name=read_iops --directory=/home/ubuntu/bcache_mount/ --size=16G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --randrepeat=0 or random read-writes with a test like, fio --filename=/home/ubuntu/bcache_mount/cachedfile --size=15GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=128 --name=iops-test-job --randrepeat=0 traces are seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] The bug exists till kernel 5.15.50-051550-generic The reproducer is pasted below: # uname -a Linux bronzor 5.15.50-051550-generic #202206251445 SMP Sat Jun 25 14:51:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sdd 8:48 0 279.4G 0 disk └─sdd18:49 060G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount nvme0n1 259:00 372.6G 0 disk └─nvme0n1p1 259:2015G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount fio --name=read_iops --directory=/home/ubuntu/bcache_mount --size=12G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --group_reporting=1 read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.28 Starting 1 process read_iops: Laying out IO file (1 file / 12288MiB) The test does not progress beyond a few minutes, and this trace is then seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] [Fix] These 3 fixes are needed for the SRU. dea3560e5f31965165bcf34ecf0b47af28bfd155, 6445ec3df23f24677064a327dce437ef3e02dc6, dc60301fb408e06e0b718c0980cdd31d2b238bee I have built these fixes into kernel 5.15.0-39-generic (jammy) and tested to verify the problem is fixed. [Regression Potential] Regression potential should be minimal. I have not seen any potential drawbacks or harmful effects of this fix in my testing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1980925/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help :
[Kernel-packages] [Bug 1980925] Re: [SRU] bcache deadlock during read IO in writeback mode
This bug is awaiting verification that the linux-nvidia/5.15.0-1006.6 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1980925 Title: [SRU] bcache deadlock during read IO in writeback mode Status in linux package in Ubuntu: Invalid Status in linux source package in Focal: Invalid Status in linux source package in Jammy: Fix Released Bug description: SRU Justification: [Impact] When Random Read I/O is started with a test like - fio --name=read_iops --directory=/home/ubuntu/bcache_mount/ --size=16G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --randrepeat=0 or random read-writes with a test like, fio --filename=/home/ubuntu/bcache_mount/cachedfile --size=15GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=128 --name=iops-test-job --randrepeat=0 traces are seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] The bug exists till kernel 5.15.50-051550-generic The reproducer is pasted below: # uname -a Linux bronzor 5.15.50-051550-generic #202206251445 SMP Sat Jun 25 14:51:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sdd 8:48 0 279.4G 0 disk └─sdd18:49 060G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount nvme0n1 259:00 372.6G 0 disk └─nvme0n1p1 259:2015G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount fio --name=read_iops --directory=/home/ubuntu/bcache_mount --size=12G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --group_reporting=1 read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.28 Starting 1 process read_iops: Laying out IO file (1 file / 12288MiB) The test does not progress beyond a few minutes, and this trace is then seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] [Fix] These 3 fixes are needed for the SRU. dea3560e5f31965165bcf34ecf0b47af28bfd155, 6445ec3df23f24677064a327dce437ef3e02dc6, dc60301fb408e06e0b718c0980cdd31d2b238bee I have built these fixes into kernel 5.15.0-39-generic (jammy) and tested to verify the problem is fixed. [Regression Potential] Regression potential should be minimal. I have not seen any potential drawbacks or harmful effects of this fix in my testing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1980925/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help :
[Kernel-packages] [Bug 1980925] Re: [SRU] bcache deadlock during read IO in writeback mode
This bug is awaiting verification that the linux- hwe-5.15/5.15.0-48.54~20.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1980925 Title: [SRU] bcache deadlock during read IO in writeback mode Status in linux package in Ubuntu: Invalid Status in linux source package in Focal: Invalid Status in linux source package in Jammy: Fix Released Bug description: SRU Justification: [Impact] When Random Read I/O is started with a test like - fio --name=read_iops --directory=/home/ubuntu/bcache_mount/ --size=16G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --randrepeat=0 or random read-writes with a test like, fio --filename=/home/ubuntu/bcache_mount/cachedfile --size=15GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=128 --name=iops-test-job --randrepeat=0 traces are seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] The bug exists till kernel 5.15.50-051550-generic The reproducer is pasted below: # uname -a Linux bronzor 5.15.50-051550-generic #202206251445 SMP Sat Jun 25 14:51:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sdd 8:48 0 279.4G 0 disk └─sdd18:49 060G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount nvme0n1 259:00 372.6G 0 disk └─nvme0n1p1 259:2015G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount fio --name=read_iops --directory=/home/ubuntu/bcache_mount --size=12G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --group_reporting=1 read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.28 Starting 1 process read_iops: Laying out IO file (1 file / 12288MiB) The test does not progress beyond a few minutes, and this trace is then seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] [Fix] These 3 fixes are needed for the SRU. dea3560e5f31965165bcf34ecf0b47af28bfd155, 6445ec3df23f24677064a327dce437ef3e02dc6, dc60301fb408e06e0b718c0980cdd31d2b238bee I have built these fixes into kernel 5.15.0-39-generic (jammy) and tested to verify the problem is fixed. [Regression Potential] Regression potential should be minimal. I have not seen any potential drawbacks or harmful effects of this fix in my testing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1980925/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help :
[Kernel-packages] [Bug 1980925] Re: [SRU] bcache deadlock during read IO in writeback mode
This bug is awaiting verification that the linux-lowlatency- hwe-5.15/5.15.0-48.54~20.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1980925 Title: [SRU] bcache deadlock during read IO in writeback mode Status in linux package in Ubuntu: Invalid Status in linux source package in Focal: Invalid Status in linux source package in Jammy: Fix Released Bug description: SRU Justification: [Impact] When Random Read I/O is started with a test like - fio --name=read_iops --directory=/home/ubuntu/bcache_mount/ --size=16G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --randrepeat=0 or random read-writes with a test like, fio --filename=/home/ubuntu/bcache_mount/cachedfile --size=15GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=128 --name=iops-test-job --randrepeat=0 traces are seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] The bug exists till kernel 5.15.50-051550-generic The reproducer is pasted below: # uname -a Linux bronzor 5.15.50-051550-generic #202206251445 SMP Sat Jun 25 14:51:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sdd 8:48 0 279.4G 0 disk └─sdd18:49 060G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount nvme0n1 259:00 372.6G 0 disk └─nvme0n1p1 259:2015G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount fio --name=read_iops --directory=/home/ubuntu/bcache_mount --size=12G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --group_reporting=1 read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.28 Starting 1 process read_iops: Laying out IO file (1 file / 12288MiB) The test does not progress beyond a few minutes, and this trace is then seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] [Fix] These 3 fixes are needed for the SRU. dea3560e5f31965165bcf34ecf0b47af28bfd155, 6445ec3df23f24677064a327dce437ef3e02dc6, dc60301fb408e06e0b718c0980cdd31d2b238bee I have built these fixes into kernel 5.15.0-39-generic (jammy) and tested to verify the problem is fixed. [Regression Potential] Regression potential should be minimal. I have not seen any potential drawbacks or harmful effects of this fix in my testing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1980925/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More
[Kernel-packages] [Bug 1980925] Re: [SRU] bcache deadlock during read IO in writeback mode
This bug was fixed in the package linux - 5.15.0-47.51 --- linux (5.15.0-47.51) jammy; urgency=medium * jammy/linux: 5.15.0-47.51 -proposed tracker (LP: #1983903) * Jammy update: v5.15.46 upstream stable release (LP: #1981864) - UBUNTU: [Packaging] Move python3-dev to build-depends * touchpad and touchscreen doesn't work at all on ACER Spin 5 (SP513-54N) (LP: #1884232) - x86/PCI: Eliminate remove_e820_regions() common subexpressions - x86: Log resource clipping for E820 regions - x86/PCI: Clip only host bridge windows for E820 regions - x86/PCI: Add kernel cmdline options to use/ignore E820 reserved regions - x86/PCI: Disable E820 reserved region clipping via quirks - x86/PCI: Revert "x86/PCI: Clip only host bridge windows for E820 regions" * [SRU][H/OEM-5.13/OEM-5.14/U][J/OEM-5.17/U] Fix invalid MAC address after hotplug tbt dock (LP: #1942999) - SAUCE: igc: wait for the MAC copy when enabled MAC passthrough * Mass Storage Gadget driver truncates device >2TB (LP: #1981390) - usb: gadget: storage: add support for media larger than 2T * AMD Rembrandt: DP tunneling fails with Thunderbolt monitors (LP: #1983143) - SAUCE: drm/amd: Fix DP Tunneling with Thunderbolt monitors - drm/amd/display: Fix for dmub outbox notification enable - Revert "drm/amd/display: Fix DPIA outbox timeout after S3/S4/reset" - drm/amd/display: Reset link encoder assignments for GPU reset - drm/amd/display: Fix DPIA outbox timeout after S3/S4/reset - drm/amd/display: Fix new dmub notification enabling in DM - SAUCE: thunderbolt: Add DP out resource when DP tunnel is discovered. * Fix sub-optimal I210 network speed (LP: #1976438) - igb: Make DMA faster when CPU is active on the PCIe link * e1000e report hardware hang (LP: #1973104) - e1000e: Enable GPT clock before sending message to CSME - Revert "e1000e: Fix possible HW unit hang after an s0ix exit" * ioam6.sh in net from ubuntu_kernel_selftests fails with 5.15 kernels in Focal (LP: #1982930) - selftests: net: fix IOAM test skip return code * Additional fix for TGL + AUO panel flickering (LP: #1983297) - Revert "UBUNTU: SAUCE: drm/i915/display/psr: Fix flicker on TGL + AUO panel" - drm/i915/display: Fix sel fetch plane offset calculation - drm/i915: Nuke ORIGIN_GTT - drm/i915/display: Drop PSR support from HSW and BDW - drm/i915/display/psr: Handle plane and pipe restrictions at every page flip - drm/i915/display/psr: Do full fetch when handling multi-planar formats - drm/i915/display: Drop unnecessary frontbuffer flushes - drm/i915/display: Handle frontbuffer rendering when PSR2 selective fetch is enabled - drm/i915/display: Fix glitches when moving cursor with PSR2 selective fetch enabled - SAUCE: drm/i915/display/psr: Reinstate fix for TGL + AUO panel flicker * AMD Yellow Carp DMCUB fw update for s0i3 B0 fixes (LP: #1957026) - drm/amd/display: Optimize bandwidth on following fast update - drm/amd/display: Fix surface optimization regression on Carrizo - drm/amd/display: Reset DMCUB before HW init * GPIO character device v1 API not enabled in kernel (LP: #1953613) - [Config] Enable CONFIG_GPIO_CDEV_V1 * intel_iommu: Fix enable intel_iommu, Ubuntu 22.04 installation crashes (LP: #1982104) - iommu/vt-d: Fix RID2PASID setup/teardown failure * Headset mic with Cirrus logic codec doesn't work (LP: #1972815) - ASoC: cs42l42: Move CS42L42 register descriptions to general include - ALSA: hda/cs8409: Use general cs42l42 include in cs8409 hda driver - ALSA: hda/cs8409: Support manual mode detection for CS42L42 * Failed to resume from S3 blocked by atlantic driver[1d6a:94c0] (LP: #1981950) - net: atlantic: remove deep parameter on suspend/resume functions - net: atlantic: remove aq_nic_deinit() when resume * Make cm32181 sensor work after system suspend (LP: #1981773) - iio: light: cm32181: Add PM support * Clear PCI errors left from BIOS (LP: #1981173) - PCI: Clear PCI_STATUS when setting up device * Fix AMDGPU blank screen when Type-C DP alt is in use (LP: #1980060) - drm/amd/display: Query DMCUB for dp alt status - drm/amd/display: Add version check before using DP alt query interface * Fix WD22TB4 suspend and resume, two external monitor can not output (LP: #1979267) - drm/dp/mst: Read the extended DPCD capabilities during system resume * [SRU] bcache deadlock during read IO in writeback mode (LP: #1980925) - bcache: memset on stack variables in bch_btree_check() and bch_sectors_dirty_init() * Audio mute key (f5) LED and Mic mute key (f8) LED are no function on HP 440/450/640/650 G9 (LP: #1982716) - ALSA: hda/realtek: fix mute/micmute LEDs for HP machines * Enable WiFi hotspot feature for MediaTek MT7921 (LP: #1979173) - mt76: mt7921: Add AP mode support - mt76: mt7921:
[Kernel-packages] [Bug 1980925] Re: [SRU] bcache deadlock during read IO in writeback mode
2/3 patches were already included in upstream v5.15.46. Updated the shared commits to refer to both reports and committed the 3rd patch for next cycle (the stable updates also are for next cycle). ** Changed in: linux (Ubuntu Jammy) Status: Confirmed => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1980925 Title: [SRU] bcache deadlock during read IO in writeback mode Status in linux package in Ubuntu: Invalid Status in linux source package in Focal: Invalid Status in linux source package in Jammy: Fix Committed Bug description: SRU Justification: [Impact] When Random Read I/O is started with a test like - fio --name=read_iops --directory=/home/ubuntu/bcache_mount/ --size=16G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --randrepeat=0 or random read-writes with a test like, fio --filename=/home/ubuntu/bcache_mount/cachedfile --size=15GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=128 --name=iops-test-job --randrepeat=0 traces are seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] The bug exists till kernel 5.15.50-051550-generic The reproducer is pasted below: # uname -a Linux bronzor 5.15.50-051550-generic #202206251445 SMP Sat Jun 25 14:51:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sdd 8:48 0 279.4G 0 disk └─sdd18:49 060G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount nvme0n1 259:00 372.6G 0 disk └─nvme0n1p1 259:2015G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount fio --name=read_iops --directory=/home/ubuntu/bcache_mount --size=12G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --group_reporting=1 read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.28 Starting 1 process read_iops: Laying out IO file (1 file / 12288MiB) The test does not progress beyond a few minutes, and this trace is then seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] [Fix] These 3 fixes are needed for the SRU. dea3560e5f31965165bcf34ecf0b47af28bfd155, 6445ec3df23f24677064a327dce437ef3e02dc6, dc60301fb408e06e0b718c0980cdd31d2b238bee I have built these fixes into kernel 5.15.0-39-generic (jammy) and tested to verify the problem is fixed. [Regression Potential] Regression potential should be minimal. I have not seen any potential drawbacks or harmful effects of this fix in my testing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1980925/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1980925] Re: [SRU] bcache deadlock during read IO in writeback mode
The patches which introduced the problem are both v5.7 and have not been backported into v5.4. ** Changed in: linux (Ubuntu) Status: Confirmed => Invalid ** Changed in: linux (Ubuntu Focal) Status: Confirmed => Invalid -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1980925 Title: [SRU] bcache deadlock during read IO in writeback mode Status in linux package in Ubuntu: Invalid Status in linux source package in Focal: Invalid Status in linux source package in Jammy: Confirmed Bug description: SRU Justification: [Impact] When Random Read I/O is started with a test like - fio --name=read_iops --directory=/home/ubuntu/bcache_mount/ --size=16G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --randrepeat=0 or random read-writes with a test like, fio --filename=/home/ubuntu/bcache_mount/cachedfile --size=15GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=128 --name=iops-test-job --randrepeat=0 traces are seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] The bug exists till kernel 5.15.50-051550-generic The reproducer is pasted below: # uname -a Linux bronzor 5.15.50-051550-generic #202206251445 SMP Sat Jun 25 14:51:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sdd 8:48 0 279.4G 0 disk └─sdd18:49 060G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount nvme0n1 259:00 372.6G 0 disk └─nvme0n1p1 259:2015G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount fio --name=read_iops --directory=/home/ubuntu/bcache_mount --size=12G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --group_reporting=1 read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.28 Starting 1 process read_iops: Laying out IO file (1 file / 12288MiB) The test does not progress beyond a few minutes, and this trace is then seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] [Fix] These 3 fixes are needed for the SRU. dea3560e5f31965165bcf34ecf0b47af28bfd155, 6445ec3df23f24677064a327dce437ef3e02dc6, dc60301fb408e06e0b718c0980cdd31d2b238bee I have built these fixes into kernel 5.15.0-39-generic (jammy) and tested to verify the problem is fixed. [Regression Potential] Regression potential should be minimal. I have not seen any potential drawbacks or harmful effects of this fix in my testing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1980925/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1980925] Re: [SRU] bcache deadlock during read IO in writeback mode
** Changed in: linux (Ubuntu Jammy) Importance: Undecided => Medium ** Changed in: linux (Ubuntu Focal) Importance: Undecided => Medium -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1980925 Title: [SRU] bcache deadlock during read IO in writeback mode Status in linux package in Ubuntu: Confirmed Status in linux source package in Focal: Confirmed Status in linux source package in Jammy: Confirmed Bug description: SRU Justification: [Impact] When Random Read I/O is started with a test like - fio --name=read_iops --directory=/home/ubuntu/bcache_mount/ --size=16G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --randrepeat=0 or random read-writes with a test like, fio --filename=/home/ubuntu/bcache_mount/cachedfile --size=15GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=128 --name=iops-test-job --randrepeat=0 traces are seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] The bug exists till kernel 5.15.50-051550-generic The reproducer is pasted below: # uname -a Linux bronzor 5.15.50-051550-generic #202206251445 SMP Sat Jun 25 14:51:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sdd 8:48 0 279.4G 0 disk └─sdd18:49 060G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount nvme0n1 259:00 372.6G 0 disk └─nvme0n1p1 259:2015G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount fio --name=read_iops --directory=/home/ubuntu/bcache_mount --size=12G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --group_reporting=1 read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.28 Starting 1 process read_iops: Laying out IO file (1 file / 12288MiB) The test does not progress beyond a few minutes, and this trace is then seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] [Fix] These 3 fixes are needed for the SRU. dea3560e5f31965165bcf34ecf0b47af28bfd155, 6445ec3df23f24677064a327dce437ef3e02dc6, dc60301fb408e06e0b718c0980cdd31d2b238bee I have built these fixes into kernel 5.15.0-39-generic (jammy) and tested to verify the problem is fixed. [Regression Potential] Regression potential should be minimal. I have not seen any potential drawbacks or harmful effects of this fix in my testing. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1980925/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1980925] Re: [SRU] bcache deadlock during read IO in writeback mode
** Description changed: SRU Justification: [Impact] This bug in bcache affects (at least) focal and jammy releases. When Random Read I/O is started with a test like - fio --name=read_iops --directory=/home/ubuntu/bcache_mount/ --size=16G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --randrepeat=0 or random read-writes with a test like, fio --filename=/home/ubuntu/bcache_mount/cachedfile --size=15GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=128 --name=iops-test-job --randrepeat=0 traces are seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] The bug exists till kernel 5.15.50-051550-generic The reproducer is pasted below: # uname -a Linux bronzor 5.15.50-051550-generic #202206251445 SMP Sat Jun 25 14:51:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sdd 8:48 0 279.4G 0 disk └─sdd18:49 060G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount nvme0n1 259:00 372.6G 0 disk └─nvme0n1p1 259:2015G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount fio --name=read_iops --directory=/home/ubuntu/bcache_mount --size=12G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --group_reporting=1 read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.28 Starting 1 process read_iops: Laying out IO file (1 file / 12288MiB) The test does not progress beyond a few minutes, and this trace is then seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] - [Fix] These 3 fixes are needed for the SRU. dea3560e5f31965165bcf34ecf0b47af28bfd155, 6445ec3df23f24677064a327dce437ef3e02dc6, dc60301fb408e06e0b718c0980cdd31d2b238bee I have built these fixes into kernel 5.15.0-39-generic (jammy) and tested to verify the problem is fixed. [Regression Potential] - I have not seen any potential drawbacks or harmful effects of this fix - in my testing. In fact it is required, without which the deadlock is - easily reproduced both on focal as well as jammy GA. + Regression potential should be minimal. I have not seen any potential + drawbacks or harmful effects of this fix in my testing. ** Description changed: SRU Justification: [Impact] - This bug in bcache affects (at least) focal and jammy releases. When Random Read I/O is started with a test like - fio --name=read_iops --directory=/home/ubuntu/bcache_mount/ --size=16G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --randrepeat=0 or random read-writes with a test like, fio --filename=/home/ubuntu/bcache_mount/cachedfile --size=15GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=128 --name=iops-test-job --randrepeat=0 traces are seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [
[Kernel-packages] [Bug 1980925] Re: [SRU] bcache deadlock during read IO in writeback mode
** Description changed: SRU Justification: [Impact] This bug in bcache affects (at least) focal and jammy releases. When Random Read I/O is started with a test like - fio --name=read_iops --directory=/home/ubuntu/bcache_mount/ --size=16G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --randrepeat=0 or random read-writes with a test like, fio --filename=/home/ubuntu/bcache_mount/cachedfile --size=15GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=128 --name=iops-test-job --randrepeat=0 traces are seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] - The bug exists till kernel 5.15.50-051550-generic, and is fixed through - upstream commit 7d6b902ea0e02b2a25c480edf471cbaa4ebe6b3c + The bug exists till kernel 5.15.50-051550-generic The reproducer is pasted below: # uname -a Linux bronzor 5.15.50-051550-generic #202206251445 SMP Sat Jun 25 14:51:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sdd 8:48 0 279.4G 0 disk └─sdd18:49 060G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount nvme0n1 259:00 372.6G 0 disk └─nvme0n1p1 259:2015G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount fio --name=read_iops --directory=/home/ubuntu/bcache_mount --size=12G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --group_reporting=1 read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.28 Starting 1 process read_iops: Laying out IO file (1 file / 12288MiB) The test does not progress beyond a few minutes, and this trace is then seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] - [ 4474.731735] INFO: task fio:16626 blocked for more than 121 seconds. - [ 4475.035858] Not tainted 5.15.50-051550-generic #202206251445 - [ 4475.335859] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. - [ 4475.716526] task:fio state:D stack:0 pid:16626 ppid: 1715 flags:0x4002 - [ 4475.716531] Call Trace: - [ 4475.716546] - [ 4475.716549] __schedule+0x23d/0x5a0 - [ 4475.716555] ? sysvec_apic_timer_interrupt+0x4e/0x90 - [ 4475.716560] schedule+0x4e/0xb0 - [ 4475.716563] rwsem_down_read_slowpath+0x32e/0x380 - [ 4475.716567] down_read+0x43/0x90 - [ 4475.716571] cached_dev_write+0x7e/0x480 [bcache] - [ 4475.716604] cached_dev_submit_bio+0x502/0x550 [bcache] - [ 4475.716618] __submit_bio+0x1a1/0x220 - [ 4475.716623] __submit_bio_noacct+0x85/0x1f0 - [ 4475.716626] submit_bio_noacct+0x4e/0x120 - [ 4475.716628] submit_bio+0x4a/0x130 - [ 4475.716631] iomap_submit_ioend+0x53/0x80 - [ 4475.716634] iomap_writepages+0x35/0x40 - [ 4475.716636] xfs_vm_writepages+0x84/0xb0 [xfs] - [ 4475.716730] do_writepages+0xda/0x200 - [ 4475.716736] filemap_fdatawrite_wbc+0x81/0xd0 - [ 4475.716739] file_write_and_wait_range+0xac/0xf0 - [ 4475.716742] xfs_file_fsync+0x5b/0x250 [xfs] - [ 4475.716821] vfs_fsync_range+0x49/0x80 - [ 4475.716826] ? __fget_light+0x32/0x80 - [ 4475.716829] __x64_sys_fsync+0x38/0x60 - [ 4475.716832] do_syscall_64+0x5c/0xc0 - [ 4475.716836] ? ksys_write+0xce/0xe0 - [ 4475.716838] ? exit_to_user_mode_prepare+0x37/0xb0 - [
[Kernel-packages] [Bug 1980925] Re: [SRU] bcache deadlock during read IO in writeback mode
** Description changed: SRU Justification: [Impact] - This bug in bcache affects (at least) focal and jammy releases. (not tested bionic as yet) + This bug in bcache affects (at least) focal and jammy releases. When Random Read I/O is started with a test like - fio --name=read_iops --directory=/home/ubuntu/bcache_mount/ --size=16G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --randrepeat=0 or random read-writes with a test like, fio --filename=/home/ubuntu/bcache_mount/cachedfile --size=15GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=128 --name=iops-test-job --randrepeat=0 traces are seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] The bug exists till kernel 5.15.50-051550-generic, and is fixed through upstream commit 7d6b902ea0e02b2a25c480edf471cbaa4ebe6b3c The reproducer is pasted below: # uname -a Linux bronzor 5.15.50-051550-generic #202206251445 SMP Sat Jun 25 14:51:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sdd 8:48 0 279.4G 0 disk └─sdd18:49 060G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount nvme0n1 259:00 372.6G 0 disk └─nvme0n1p1 259:2015G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount fio --name=read_iops --directory=/home/ubuntu/bcache_mount --size=12G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --group_reporting=1 read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.28 Starting 1 process read_iops: Laying out IO file (1 file / 12288MiB) The test does not progress beyond a few minutes, and this trace is then seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] [ 4474.731735] INFO: task fio:16626 blocked for more than 121 seconds. [ 4475.035858] Not tainted 5.15.50-051550-generic #202206251445 [ 4475.335859] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4475.716526] task:fio state:D stack:0 pid:16626 ppid: 1715 flags:0x4002 [ 4475.716531] Call Trace: [ 4475.716546] [ 4475.716549] __schedule+0x23d/0x5a0 [ 4475.716555] ? sysvec_apic_timer_interrupt+0x4e/0x90 [ 4475.716560] schedule+0x4e/0xb0 [ 4475.716563] rwsem_down_read_slowpath+0x32e/0x380 [ 4475.716567] down_read+0x43/0x90 [ 4475.716571] cached_dev_write+0x7e/0x480 [bcache] [ 4475.716604] cached_dev_submit_bio+0x502/0x550 [bcache] [ 4475.716618] __submit_bio+0x1a1/0x220 [ 4475.716623] __submit_bio_noacct+0x85/0x1f0 [ 4475.716626] submit_bio_noacct+0x4e/0x120 [ 4475.716628] submit_bio+0x4a/0x130 [ 4475.716631] iomap_submit_ioend+0x53/0x80 [ 4475.716634] iomap_writepages+0x35/0x40 [ 4475.716636] xfs_vm_writepages+0x84/0xb0 [xfs] [ 4475.716730] do_writepages+0xda/0x200 [ 4475.716736] filemap_fdatawrite_wbc+0x81/0xd0 [ 4475.716739] file_write_and_wait_range+0xac/0xf0 [ 4475.716742] xfs_file_fsync+0x5b/0x250 [xfs] [ 4475.716821] vfs_fsync_range+0x49/0x80 [ 4475.716826] ? __fget_light+0x32/0x80 [ 4475.716829] __x64_sys_fsync+0x38/0x60 [ 4475.716832] do_syscall_64+0x5c/0xc0 [ 4475.716836] ? ksys_write+0xce/0xe0 [ 4475.716838] ?
[Kernel-packages] [Bug 1980925] Re: [SRU] bcache deadlock during read IO in writeback mode
** Description changed: SRU Justification: [Impact] This bug in bcache affects (at least) focal and jammy releases. (not tested bionic as yet) When Random Read I/O is started with a test like - fio --name=read_iops --directory=/home/ubuntu/bcache_mount/ --size=16G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --randrepeat=0 or random read-writes with a test like, fio --filename=/home/ubuntu/bcache_mount/cachedfile --size=15GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=128 --name=iops-test-job --randrepeat=0 traces are seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] The bug exists till kernel 5.15.50-051550-generic, and is fixed through - upstream commit - - commit b57a534df246999bc09a510dc0e0125cacd1724c (HEAD) - Author: Coly Li - Date: Fri May 27 23:28:16 2022 +0800 - - bcache: memset on stack variables in bch_btree_check() and - bch_sectors_dirty_init() - - The local variables check_state (in bch_btree_check()) and state (in - bch_sectors_dirty_init()) should be fully filled by 0, because before - allocating them on stack, they were dynamically allocated by kzalloc(). - - Signed-off-by: Coly Li - Link: https://lore.kernel.org/r/20220527152818.27545-2-col...@suse.de - Signed-off-by: Jens Axboe + upstream commit 7d6b902ea0e02b2a25c480edf471cbaa4ebe6b3c The reproducer is pasted below: # uname -a Linux bronzor 5.15.50-051550-generic #202206251445 SMP Sat Jun 25 14:51:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sdd 8:48 0 279.4G 0 disk └─sdd18:49 060G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount nvme0n1 259:00 372.6G 0 disk └─nvme0n1p1 259:2015G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount fio --name=read_iops --directory=/home/ubuntu/bcache_mount --size=12G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --group_reporting=1 read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.28 Starting 1 process read_iops: Laying out IO file (1 file / 12288MiB) The test does not progress beyond a few minutes, and this trace is then seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] [ 4474.731735] INFO: task fio:16626 blocked for more than 121 seconds. [ 4475.035858] Not tainted 5.15.50-051550-generic #202206251445 [ 4475.335859] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4475.716526] task:fio state:D stack:0 pid:16626 ppid: 1715 flags:0x4002 [ 4475.716531] Call Trace: [ 4475.716546] [ 4475.716549] __schedule+0x23d/0x5a0 [ 4475.716555] ? sysvec_apic_timer_interrupt+0x4e/0x90 [ 4475.716560] schedule+0x4e/0xb0 [ 4475.716563] rwsem_down_read_slowpath+0x32e/0x380 [ 4475.716567] down_read+0x43/0x90 [ 4475.716571] cached_dev_write+0x7e/0x480 [bcache] [ 4475.716604] cached_dev_submit_bio+0x502/0x550 [bcache] [ 4475.716618] __submit_bio+0x1a1/0x220 [ 4475.716623] __submit_bio_noacct+0x85/0x1f0 [ 4475.716626] submit_bio_noacct+0x4e/0x120 [ 4475.716628] submit_bio+0x4a/0x130 [ 4475.716631]
[Kernel-packages] [Bug 1980925] Re: [SRU] bcache deadlock during read IO in writeback mode
This is the fix that will need SRU to jammy/focal, root@nikhil-Lenovo-Legion-Y540-15IRH-PG0:/home/nikhil/Downloads/upstream_kernel/linux/fs/fuse# git show 7d6b902ea0e02b2a25c480edf471cbaa4ebe6b3c commit 7d6b902ea0e02b2a25c480edf471cbaa4ebe6b3c Author: Coly Li Date: Fri May 27 23:28:16 2022 +0800 bcache: memset on stack variables in bch_btree_check() and bch_sectors_dirty_init() The local variables check_state (in bch_btree_check()) and state (in bch_sectors_dirty_init()) should be fully filled by 0, because before allocating them on stack, they were dynamically allocated by kzalloc(). Signed-off-by: Coly Li Link: https://lore.kernel.org/r/20220527152818.27545-2-col...@suse.de Signed-off-by: Jens Axboe diff --git a/drivers/md/bcache/btree.c b/drivers/md/bcache/btree.c index 2362bb8ef6d1..e136d6edc1ed 100644 --- a/drivers/md/bcache/btree.c +++ b/drivers/md/bcache/btree.c @@ -2017,6 +2017,7 @@ int bch_btree_check(struct cache_set *c) if (c->root->level == 0) return 0; + memset(_state, 0, sizeof(struct btree_check_state)); check_state.c = c; check_state.total_threads = bch_btree_chkthread_nr(); check_state.key_idx = 0; diff --git a/drivers/md/bcache/writeback.c b/drivers/md/bcache/writeback.c index 75b71199800d..d138a2d73240 100644 --- a/drivers/md/bcache/writeback.c +++ b/drivers/md/bcache/writeback.c @@ -950,6 +950,7 @@ void bch_sectors_dirty_init(struct bcache_device *d) return; } + memset(, 0, sizeof(struct bch_dirty_init_state)); state.c = c; state.d = d; state.total_threads = bch_btre_dirty_init_thread_nr(); ** Description changed: SRU Justification: [Impact] - This bug in bcache affects focal and jammy releases. + This bug in bcache affects (at least) focal and jammy releases. (not tested bionic as yet) When Random Read I/O is started with a test like - fio --name=read_iops --directory=/home/ubuntu/bcache_mount/ --size=16G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --randrepeat=0 or random read-writes with a test like, fio --filename=/home/ubuntu/bcache_mount/cachedfile --size=15GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=128 --name=iops-test-job --randrepeat=0 traces are seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] The bug exists till kernel 5.15.50-051550-generic, and is fixed through upstream commit commit b57a534df246999bc09a510dc0e0125cacd1724c (HEAD) Author: Coly Li Date: Fri May 27 23:28:16 2022 +0800 - bcache: memset on stack variables in bch_btree_check() and bch_sectors_dirty_init() - - The local variables check_state (in bch_btree_check()) and state (in - bch_sectors_dirty_init()) should be fully filled by 0, because before - allocating them on stack, they were dynamically allocated by kzalloc(). - - Signed-off-by: Coly Li - Link: https://lore.kernel.org/r/20220527152818.27545-2-col...@suse.de - Signed-off-by: Jens Axboe - + bcache: memset on stack variables in bch_btree_check() and + bch_sectors_dirty_init() + + The local variables check_state (in bch_btree_check()) and state (in + bch_sectors_dirty_init()) should be fully filled by 0, because before + allocating them on stack, they were dynamically allocated by kzalloc(). + + Signed-off-by: Coly Li + Link: https://lore.kernel.org/r/20220527152818.27545-2-col...@suse.de + Signed-off-by: Jens Axboe The reproducer is pasted below: # uname -a Linux bronzor 5.15.50-051550-generic #202206251445 SMP Sat Jun 25 14:51:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux - NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINTS - sdd 8:48 0 279.4G 0 disk - └─sdd18:49 060G 0 part - └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount - nvme0n1 259:00 372.6G 0 disk - └─nvme0n1p1 259:2015G 0 part - └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount - + sdd 8:48 0 279.4G 0 disk + └─sdd18:49 060G 0
[Kernel-packages] [Bug 1980925] Re: [SRU] bcache deadlock during read IO in writeback mode
Logs have been pasted in the description of the bug. ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed ** Changed in: linux (Ubuntu Focal) Status: Incomplete => Confirmed ** Changed in: linux (Ubuntu Jammy) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1980925 Title: [SRU] bcache deadlock during read IO in writeback mode Status in linux package in Ubuntu: Confirmed Status in linux source package in Focal: Confirmed Status in linux source package in Jammy: Confirmed Bug description: SRU Justification: [Impact] This bug in bcache affects focal and jammy releases. When Random Read I/O is started with a test like - fio --name=read_iops --directory=/home/ubuntu/bcache_mount/ --size=16G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --randrepeat=0 or random read-writes with a test like, fio --filename=/home/ubuntu/bcache_mount/cachedfile --size=15GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=128 --name=iops-test-job --randrepeat=0 traces are seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] The bug exists till kernel 5.15.50-051550-generic, and is fixed through upstream commit commit b57a534df246999bc09a510dc0e0125cacd1724c (HEAD) Author: Coly Li Date: Fri May 27 23:28:16 2022 +0800 bcache: memset on stack variables in bch_btree_check() and bch_sectors_dirty_init() The local variables check_state (in bch_btree_check()) and state (in bch_sectors_dirty_init()) should be fully filled by 0, because before allocating them on stack, they were dynamically allocated by kzalloc(). Signed-off-by: Coly Li Link: https://lore.kernel.org/r/20220527152818.27545-2-col...@suse.de Signed-off-by: Jens Axboe The reproducer is pasted below: # uname -a Linux bronzor 5.15.50-051550-generic #202206251445 SMP Sat Jun 25 14:51:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux NAMEMAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sdd 8:48 0 279.4G 0 disk └─sdd18:49 060G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount nvme0n1 259:00 372.6G 0 disk └─nvme0n1p1 259:2015G 0 part └─bcache0 252:0060G 0 disk /home/ubuntu/bcache_mount fio --name=read_iops --directory=/home/ubuntu/bcache_mount --size=12G --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=128 --rw=randread --group_reporting=1 read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.28 Starting 1 process read_iops: Laying out IO file (1 file / 12288MiB) The test does not progress beyond a few minutes, and this trace is then seen in the kernel log, [ 4473.699902] INFO: task bcache_writebac:1835 blocked for more than 120 seconds. [ 4474.050921] Not tainted 5.15.50-051550-generic #202206251445 [ 4474.350883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4474.731391] task:bcache_writebac state:D stack:0 pid: 1835 ppid: 2 flags:0x4000 [ 4474.731408] Call Trace: [ 4474.731411] [ 4474.731413] __schedule+0x23d/0x5a0 [ 4474.731433] schedule+0x4e/0xb0 [ 4474.731436] rwsem_down_write_slowpath+0x220/0x3d0 [ 4474.731441] down_write+0x43/0x50 [ 4474.731446] bch_writeback_thread+0x78/0x320 [bcache] [ 4474.731471] ? read_dirty_submit+0x70/0x70 [bcache] [ 4474.731487] kthread+0x12a/0x150 [ 4474.731491] ? set_kthread_struct+0x50/0x50 [ 4474.731494] ret_from_fork+0x22/0x30 [ 4474.731499] [ 4474.731735] INFO: task fio:16626 blocked for more than 121 seconds. [ 4475.035858] Not tainted 5.15.50-051550-generic #202206251445 [ 4475.335859] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4475.716526] task:fio state:D stack:0 pid:16626 ppid: 1715 flags:0x4002 [ 4475.716531] Call Trace: [ 4475.716546] [ 4475.716549]
