This bug was fixed in the package linux - 6.8.0-41.41
---------------
linux (6.8.0-41.41) noble; urgency=medium
* noble/linux: 6.8.0-41.41 -proposed tracker (LP: #2075611)
* Packaging resync (LP: #1786013)
- [Packaging] debian.master/dkms-versions -- update from kernel-versions
(main/s2024.07.08)
* md: nvme over tcp with a striped underlying md raid device leads to data
corruption (LP: #2075110)
- md/md-bitmap: fix writing non bitmap pages
* Linux 6.8 fails to boot on ARM64 if any param is more than 146 chars
(LP: #2069534)
- SAUCE: arm64: v6.8: cmdline param >= 146 chars kills kernel
* CVE-2024-39484
- mmc: davinci: Don't strip remove function when driver is builtin
* CVE-2024-39292
- um: Add winch to winch_handlers before registering winch IRQ
-- Manuel Diewald <[email protected]> Fri, 02 Aug 2024
16:15:19 +0200
** Changed in: linux (Ubuntu Noble)
Status: Fix Committed => Fix Released
** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2024-39292
** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2024-39484
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/2075110
Title:
md: nvme over tcp with a striped underlying md raid device leads to
data corruption
Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Noble:
Fix Released
Bug description:
BugLink: https://bugs.launchpad.net/bugs/2075110
[Impact]
There is a fault in the md subsystem where __write_sb_page() will
round the io size up to the optimal size, but it doesn't check to see
if the final io size exceeds the bitmap length.
This gets us into a situation where if we have 256K of io to submit,
64 pages are needed. md_bitmap_storage_alloc() allocates 1 page, and
63 are allocated afterward.
When we send md writes over the network, e.g. with nvme over tcp, the
network subsystem checks the first page which is sendpage_ok(), but
not the other 63, which might not be sendpage_ok(), and will get
stuck, causing a hang and data corruption.
If you trigger the issue, you get the following oops in dmesg:
WARNING: CPU: 0 PID: 83 at net/core/skbuff.c:6995
skb_splice_from_iter+0x139/0x370
CPU: 0 PID: 83 Comm: kworker/0:1H Not tainted 6.8.0-39-generic #39-Ubuntu
Workqueue: nvme_tcp_wq nvme_tcp_io_work [nvme_tcp]
RIP: 0010:skb_splice_from_iter+0x139/0x370
CR2: 000072dab83e5f84
Call Trace:
<TASK>
? show_regs+0x6d/0x80
? __warn+0x89/0x160
? skb_splice_from_iter+0x139/0x370
? report_bug+0x17e/0x1b0
? handle_bug+0x51/0xa0
? exc_invalid_op+0x18/0x80
? asm_exc_invalid_op+0x1b/0x20
? skb_splice_from_iter+0x139/0x370
tcp_sendmsg_locked+0x352/0xd70
? tcp_push+0x159/0x190
? tcp_sendmsg_locked+0x9c4/0xd70
tcp_sendmsg+0x2c/0x50
inet_sendmsg+0x42/0x80
sock_sendmsg+0x118/0x150
nvme_tcp_try_send_data+0x18b/0x4c0 [nvme_tcp]
? __tcp_cleanup_rbuf+0xc5/0xe0
nvme_tcp_try_send+0x23c/0x300 [nvme_tcp]
nvme_tcp_io_work+0x40/0xe0 [nvme_tcp]
process_one_work+0x16c/0x350
worker_thread+0x306/0x440
? _raw_spin_unlock_irqrestore+0x11/0x60
? __pfx_worker_thread+0x10/0x10
kthread+0xef/0x120
? __pfx_kthread+0x10/0x10
ret_from_fork+0x44/0x70
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1b/0x30
</TASK>
nvme nvme1: failed to send request -5
nvme nvme1: I/O tag 125 (307d) type 4 opcode 0x0 (I/O Cmd) QID 1 timeout
nvme nvme1: starting error recovery
block nvme1n1: no usable path - requeuing I/O
nvme nvme1: Reconnecting in 10 seconds...
There is no workaround.
[Fix]
This was fixed in the below commit in 6.11-rc1:
commit ab99a87542f194f28e2364a42afbf9fb48b1c724
Author: Ofir Gal <[email protected]>
Date: Fri Jun 7 10:27:44 2024 +0300
Subject: md/md-bitmap: fix writing non bitmap pages
Link:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ab99a87542f194f28e2364a42afbf9fb48b1c724
This is a clean cherry-pick to the Noble tree.
[Testcase]
This can be reproduced by running blktests md/001 [1], which the
author of the fix created to act as a regression test for this issue.
[1]
https://github.com/osandov/blktests/commit/a24a7b462816fbad7dc6c175e53fcc764ad0a822
Deploy a fresh Noble VM, that has a scratch NVME disk.
$ sudo apt install build-essential fio
$ git clone https://github.com/osandov/blktests.git
$ cd blktests
$ make
$ echo "TEST_DEVS=(/dev/nvme0n1)" > config
$ sudo ./check md/001
The md/001 test will hang an affected system, and the above oops
message will be visible in dmesg.
A test kernel is available in the following ppa:
https://launchpad.net/~mruffell/+archive/ubuntu/sf390669-test
If you install the test kernel, the md/001 test will complete
successfully, and the issue will no longer appear.
[Where problems could occur]
We are changing how the md subsystem calculates final IO sizes, and
taking the smaller value of the size or the bitmap_limit. This makes
sure we don't leak the final page and corrupt data.
If a regression were to occur, it would likely affect all md users,
but would be more obvious to md users over the network, like nvme over
tcp.
There is no workaround. Users would have to downgrade their kernels if
a regression occurs.
[Other info]
I checked Jammy 5.15 and it works fine, so the issue must have been
introduced later on. It is not needed for Focal or Jammy.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2075110/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : [email protected]
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
