Your message dated Wed, 07 Aug 2024 05:32:33 +0000
with message-id <[email protected]>
and subject line Bug#1054631: fixed in libnvme 1.3-1+deb12u1
has caused the Debian Bug report #1054631,
regarding random memory corruption with some NVMe devices
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact [email protected]
immediately.)
--
1054631: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1054631
Debian Bug Tracking System
Contact [email protected] with problems
--- Begin Message ---
Package: libnvme1
Version: 1.3-1
Severity: important
Tags: patch
Dear Maintainer,
libnvme has a serious bug that, on some NVMe hardware, can trigger DMA
writes that overwrite memory of unrelated processes, resulting in random
crashes and other system stability issues. This can be caused by simply
running `nvme list`.
This was very recently fixed upstream in
https://github.com/linux-nvme/libnvme/commit/a2b8e52e46cfd888ac5a48d8ce632bd70a5caa93
and
https://github.com/linux-nvme/libnvme/commit/68c6ffb11d40a427fc1fd70ac2ac97fd01952913.
I've been able to reproduce this in multiple systems that have
SKHynix_HFS256GD9TNI-L2B0B SSDs. From recent commit descriptions in
libnvme and nvme-cli, it sounds like some NVMe devices DMA only in 4k
blocks, but libnvme would sometimes allocate a smaller buffer. Which
can result in the DMA operation clobbering unrelated memory.
To reproduce:
1. Make sure the kernel isn't using IOMMU (e.g., boot with
intel_iommu=off).
2. Run `while nvme list; do sleep 0.1; done`.
Generally the nvme process will segfault or abort with an error within a
very small number of iterations. Example dmesg output when this
happens:
[ 2238.591144] show_signal_msg: 6 callbacks suppressed
[ 2238.591150] nvme[1315]: segfault at 8 ip 00007fbf286748e9 sp
00007ffe4cbccb30 error 4 in libc.so.6[7fbf28603000+155000] likely on CPU
1 (core 1, socket 0)
[ 2238.591178] Code: 24 18 45 85 d2 0f 85 17 05 00 00 48 81 fb ff 03 00
00 76 20 43 8d 44 2d 0c 48 8d 44 c5 00 48 8b 10 48 8d 48 f0 48 39 ca 74
0a <48> 39 5a 08 0f 83 2b 05 00 00 41 8d 4d 01 43 8d 44 2d 0e 89 cf 48
If you keep running this, you'll also find that other processes start
crashing as well, usually with segfaults or weird shared library
failures. I've seen sshd crash, firefox crash, systemd segfault, etc.
As an example, I recently saw sshd failing with this error:
Oct 26 19:46:27 challenger sshd[1361]: /usr/sbin/sshd: error while
loading shared libraries: /lib/x86_64-linux-gnu/libnsl.so.2: unexpected
PLT reloc type 0x00
I was able to trivially apply the two git commits listed above to
libnvme 1.3 in Bookworm, and this resolved the crash and memory
corruption caused by `nvme list`. I'd recommend applying these
changes to libnvme in Bookworm, since the impact is pretty severe for
users who happen to own affected devices.
There have also been other recent memory alignment changes in libnvme
and nvme-cli. It may be worth trying to backport more of these to
the Bookworm packages to avoid memory corruption during other nvme
operations.
-- System Information:
Debian Release: 12.2
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 6.1.0-13-amd64 (SMP w/6 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages libnvme1 depends on:
ii libc6 2.36-9+deb12u3
ii libdbus-1-3 1.14.10-1~deb12u1
ii libjson-c5 0.16-2
ii libssl3 3.0.11-1~deb12u2
libnvme1 recommends no packages.
Versions of packages libnvme1 suggests:
ii nvme-cli 2.4+really2.3-3
-- no debconf information
--- End Message ---
--- Begin Message ---
Source: libnvme
Source-Version: 1.3-1+deb12u1
Done: Daniel Baumann <[email protected]>
We believe that the bug you reported is fixed in the latest version of
libnvme, which is due to be installed in the Debian FTP archive.
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to [email protected],
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Daniel Baumann <[email protected]> (supplier of updated libnvme
package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing [email protected])
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Format: 1.8
Date: Sun, 14 Apr 2024 08:57:21 +0200
Source: libnvme
Architecture: source
Version: 1.3-1+deb12u1
Distribution: bookworm
Urgency: medium
Maintainer: Daniel Baumann <[email protected]>
Changed-By: Daniel Baumann <[email protected]>
Closes: 1054631
Changes:
libnvme (1.3-1+deb12u1) bookworm; urgency=medium
.
* Uploading to bookworm.
* Cherry-picking upstream commits to fix buffer overflow during scanning
devices that do not support sub-4k reads (Closes: #1054631).
Checksums-Sha1:
ace41f3e9de98b95a3febf90862d00dd2b1e2e4e 2146 libnvme_1.3-1+deb12u1.dsc
d2460d16746ebdadb3c1a7b991ee5f1153a286dc 7632
libnvme_1.3-1+deb12u1.debian.tar.xz
a8c41ab410b4ed6c247d79e6a52fd2769619281a 9248
libnvme_1.3-1+deb12u1_amd64.buildinfo
Checksums-Sha256:
c8d65529c5ee8f61f84766661257c6f05525d0ed0b9684517b639b1e99522b9c 2146
libnvme_1.3-1+deb12u1.dsc
e29cbd8d526fc6166918e40a7430ee33eb0bfc0f3f2e2cb17afa5b2a4b11fb75 7632
libnvme_1.3-1+deb12u1.debian.tar.xz
fe57f39e287fa5744b915945a4af6db36002aa18450909082e88c0217abc00f6 9248
libnvme_1.3-1+deb12u1_amd64.buildinfo
Files:
76038433f57f06f3c9eea9c2e3c022ca 2146 libs optional libnvme_1.3-1+deb12u1.dsc
b3cd5bc9d26255f9a51bbff49ced9461 7632 libs optional
libnvme_1.3-1+deb12u1.debian.tar.xz
6f903bb67d032717c3804b46795ee9fb 9248 libs optional
libnvme_1.3-1+deb12u1_amd64.buildinfo
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEEgTbtJcfWfpLHSkKSVc8b+YaruccFAmah7AcACgkQVc8b+Yar
ucd7LhAAz8t5KA0UbruTEUfyB80PWBk8Gre94XnZLEKkCSpCuaJlQ6cke0YUaaRA
+zxgbZs1XFbUT+ws8ZCFAkCVKLLPNZuRr7wrnrah/H1n3Bju/sQkobAIXdavuXTU
b5NtoWy2c8DdGETboFYcW8Md6xD3BeOjRrc0gWw8iswzIfSNXOTGo9k9Y6rHpuqa
omnUe0sLUiMy7vrfMi0zN0hReveESdy9TFuxms9Uj9dVqFFujeiduvwBdXLP5IQq
57+18RYBBLTOEF9x9rfnUbmOzlNHQk1YaWF+DYmq8dWR/PRo0I/mp+2vatDSsc4b
NxGiW1bk94NNhaiwgui4YSxVl1lJ7ISPB1sf1e8FfKLqycAQKRxyA4JDpQ5y18j7
efml7aequisxpF9g8ymMEYawe54zsKOMoQlO0rn2qhJbnEiKj9uCsG5CriSPx+y7
71XudNy1bTcOX+LwfRKL19eQkV2hc2hRVXyx7O45L9q2VWZh091+DtK1+n3DjY9o
FFRTySgBBVDhQ6k0Y/ZIC/5dh5V2ItrF/PmsDzsb0ExP5LA9enFtyloPM1WBzplx
0yKH04DCsxGCiAf+4vefKqbUcq1+o054LdpJLeDAKFvK6XxcT3jiIYcPl+Wz/GRp
fYeAl9KtmjLKRIlyfYTEVWFrCauXS0h6Xy54BaZ7zJBSr8xuZZU=
=UowV
-----END PGP SIGNATURE-----
pgpyTMwtVRsUT.pgp
Description: PGP signature
--- End Message ---
