Verification done for jammy-proposed. TEST CASE: I did test the -proposed package on a clean install of jammy (without the HWE kernel) and a test pool.
Since you need a tool that uses lseek SEEK_HOLE/SEEK_DATA and the coreutils bundled with Ubuntu 22.04 are still at version 8.x, I manually downloaded the tarball of coreutils 9.1 from https://www.gnu.org/software/coreutils/ and installed the newly compiled version of "cp" that is used by the zhammer script. test@zfstest:/test$ cp --version cp (GNU coreutils) 9.1 Then I used the zhammer.sh script linked in #16 to test for data corruption: test@zfstest:~$ sudo parallel --lb --halt-on-error now,fail=1 ./zhammer.sh /test 10000000 16k 10000 ::: $(seq $(nproc)) ... As expected on an unpatched system I immediately got the error on the first run: [zhammer::765410] zhammer_765410_0 differed from zhammer_765410_5930! [zhammer::765410] Hexdump diff follows --- zhammer_765410_0.hex 2024-02-15 15:27:22.370732491 +0000 +++ zhammer_765410_5930.hex 2024-02-15 15:27:22.370732491 +0000 @@ -1,3 +1,3 @@ -00000000 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| +00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00004000 [zhammer::765410] Uname: Linux zfstest 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 GNU/Linux [zhammer::765410] ZFS userspace: zfs-2.1.5-1ubuntu6~22.04.2 [zhammer::765410] ZFS kernel: zfs-kmod-2.1.5-1ubuntu6~22.04.2 [zhammer::765410] Module: /lib/modules/5.15.0-94-generic/kernel/zfs/zfs.ko [zhammer::765410] Srcversion: 5A94B4662A7A991696CC35F [zhammer::765410] SHA256: d83e630d4e46280ba6b8bf922850899318ebd24dedd14ab5672574c4bd811ffc [zhammer::765391] checking 10000 files at iteration 0 parallel: This job failed: ./zhammer.sh /test 10000000 16k 10000 5 ... Then I enabled the -proposed repository, installed the patched zfs-version and enabled the dkms kernel module for zfs. After a reboot I re-ran the zhammer.sh and it didn't trigger the bug any more after 5 iterations (x10.000 files). [zhammer::4265] Uname: Linux zfstest 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 GNU/Linux [zhammer::4265] ZFS userspace: zfs-2.1.5-1ubuntu6~22.04.3 [zhammer::4265] ZFS kernel: zfs-kmod-2.1.5-1ubuntu6~22.04.3 [zhammer::4265] Module: /lib/modules/5.15.0-94-generic/updates/dkms/zfs.ko [zhammer::4265] Srcversion: 9CA2EC22B1B594D0C432666 [zhammer::4265] SHA256: f78e84e7f995c15b715f6010ec939ca4202a73654bfafd24572d214ebfcb6364 [zhammer::4265] Work dir: /test [zhammer::4265] Count: 10000000 files [zhammer::4265] Block size: 16k [zhammer::4265] Check every: 10000 files [zhammer::4265] writing 10000 files at iteration 0 [...] [zhammer::4253] writing 10000 files at iteration 40000 [zhammer::4248] writing 10000 files at iteration 40000 [zhammer::4246] writing 10000 files at iteration 40000 [zhammer::4256] writing 10000 files at iteration 40000 [zhammer::4252] writing 10000 files at iteration 40000 [zhammer::4280] writing 10000 files at iteration 40000 [zhammer::4271] writing 10000 files at iteration 40000 [zhammer::4265] writing 10000 files at iteration 40000 [zhammer::4246] checking 10000 files at iteration 40000 [zhammer::4253] checking 10000 files at iteration 40000 [zhammer::4265] checking 10000 files at iteration 40000 [zhammer::4248] checking 10000 files at iteration 40000 [zhammer::4252] checking 10000 files at iteration 40000 [zhammer::4280] checking 10000 files at iteration 40000 [zhammer::4256] checking 10000 files at iteration 40000 [zhammer::4271] checking 10000 files at iteration 40000 ... Therefore I can report that the proposed fix is also working for jammy. P.S.: I am not sure if I am supposed to change the verification-tag, but since there seems to be very little other activity here and the fix for jammy is still not released I just went ahead. Hopefully this helps to get the fix released in less then (already humiliating) three months. ** Tags removed: verification-needed-jammy ** Tags added: verification-done-jammy -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/2044657 Title: Multiple data corruption issues in zfs Status in zfs-linux package in Ubuntu: Fix Released Status in zfs-linux source package in Xenial: Confirmed Status in zfs-linux source package in Bionic: Confirmed Status in zfs-linux source package in Focal: Fix Committed Status in zfs-linux source package in Jammy: Fix Committed Status in zfs-linux source package in Lunar: Won't Fix Status in zfs-linux source package in Mantic: Fix Released Status in zfs-linux source package in Noble: Fix Released Bug description: [ Impact ] * Multiple data corruption issues have been identified and fixed in ZFS. Some of them, at varying real-life reproducibility frequency have been deterimed to affect very old zfs releases. Recommendation is to upgrade to 2.2.2 or 2.1.14 or backport dnat patch alone. This is to ensure users get other potentially related fixes and runtime tunables to possibly mitigate other bugs that are related and are being fixed upstream for future releases. * For jammy the 2.1.14 upgrade will bring HWE kernel support and also compatiblity/support for hardened kernel builds that mitigate SLS (straight-line-speculation). * In the absence of the upgrade a cherry-pick will address this particular popular issue alone - without addressing other issues w.r.t. Redbleed / SLS, bugfixes around trim support, and other related improvements that were discovered and fixed around the same time as this popular issue. [ Test Plan ] * !!! Danger !!! use reproducer from https://zfsonlinux.topicbox.com/groups/zfs-discuss/T12876116b8607cdb and confirm if that issue is resolved or not. Do not run on production ZFS pools / systems. * autopkgtest pass (from https://ubuntu-archive- team.ubuntu.com/proposed-migration/ ) * adt-matrix pass (from https://kernel.ubuntu.com/adt-matrix/ ) * kernel regression zfs testsuite pass (from Kernel team RT test results summary, private) * zsys integration test pass (upgrade of zsys installed systems for all releases) * zsys install test pass (for daily images of LTS releases only that have such installer support, as per iso tracker test case) * LXD (ping LXD team to upgrade vendored in tooling to 2.2.2 and 2.1.14, and test LXD on these updated kernels) [ Where problems could occur ] * Upgrade to 2.1.14 on jammy with SLS mitigations compatiblity will introduce slight slow down on amd64 (for hw accelerated assembly code- paths only in the encryption primitives) * Uncertain of the perfomance impact of the extra checks in dnat patch fix itself. Possibly affecting speed of operation, at the benefit of correctness. * The cherry-picked patch ("dnat"? dnode) changes the dirty data check, but only makes it stronger and not weaker, thus if it were incorrect, likely only performance would be impacted (and it is unlikely to be incorrect given upstream reviews and attention to data corruption issues; also, there are no additional changes to that function upstream) [ Other Info ] * https://github.com/openzfs/zfs/pull/15571 is most current consideration of affairs To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/2044657/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
