I can confirm 100% repro of this bug, on several systems. Data for a Xeon Silver 4215R on Supermicro X11SPi-TF. The only change between 5.15.0-37-generic and 5.15.0-27-generic is booting the same machine with a different kernel.
Write to encrypted ramdisk: - 5.15.0-37-generic: 186 MB/s - 5.15.0-27-generic: 1.2 GB/s (6.5x faster) Read from encrypted RAIDz1 on 8 x ST16000NM000J, arc cache cold: - 5.15.0-37-generic: 61.1 MB/s - 5.15.0-27-generic: 490 MB/s (8x faster) It only appears to affect ZFS when encryption is enabled: Write to unencrypted ramdisk: - 5.15.0-37-generic: 1.6 GB/s - 5.15.0-27-generic: 1.6 GB/s (identical) I am getting very similar data on Xeon E3-1240 v6 / Supermicro X11SSM-F and EPYC 7282 / Supermicro H12SSL-i I am attaching a test script and output from "apport-cli --save 5.15.0-37-generic.apport -p linux --file-bug" Methodology below: # uname -a Linux aero 5.15.0-37-generic #39-Ubuntu SMP Wed Jun 1 19:16:45 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux # Create encrypted ZFS to ramdisk using the the attached "zfs-test.sh enc" Write speed to encrypted ramdisk: # dd if=/dev/zero of=/tmp/mount/zero bs=1M dd: error writing '/tmp/mount/zero': No space left on device 7432+0 records in 7431+0 records out 7792885760 bytes (7.8 GB, 7.3 GiB) copied, 41.9707 s, 186 MB/s # Create unencrypted ZFS to ramdisk using the the attached "zfs-test.sh" Write speed to unencrypted ramdisk: # dd if=/dev/zero of=/tmp/mount/zero bs=1M dd: error writing '/tmp/mount/zero': No space left on device 7439+0 records in 7438+0 records out 7799308288 bytes (7.8 GB, 7.3 GiB) copied, 4.93472 s, 1.6 GB/s Read speed from encrypted RAIDz1 on 8 x ST16000NM000J, arc cache cold: # dd if=/storage/kits/ubuntu-20.04.1-desktop-amd64.iso of=/dev/null bs=1M 2656+0 records in 2656+0 records out 2785017856 bytes (2.8 GB, 2.6 GiB) copied, 45.6045 s, 61.1 MB/s # uname -a Linux aero 5.15.0-27-generic #28-Ubuntu SMP Thu Apr 14 04:55:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux # Create encrypted ZFS to ramdisk using the the attached "zfs-test.sh enc" Write speed to encrypted ramdisk: # dd if=/dev/zero of=/tmp/mount/zero bs=1M dd: error writing '/tmp/mount/zero': No space left on device 7433+0 records in 7432+0 records out 7793016832 bytes (7.8 GB, 7.3 GiB) copied, 6.28478 s, 1.2 GB/s # Create unencrypted ZFS to ramdisk using the the attached "zfs-test.sh" Write speed to unencrypted ramdisk: # dd if=/dev/zero of=/tmp/mount/zero bs=1M dd: error writing '/tmp/mount/zero': No space left on device 7439+0 records in 7438+0 records out 7799308288 bytes (7.8 GB, 7.3 GiB) copied, 4.76863 s, 1.6 GB/s Read speed from encrypted RAIDz1 on 8 x ST16000NM000J, arc cache cold: # dd if=/storage/kits/ubuntu-20.04.1-desktop-amd64.iso of=/dev/null bs=1M 2656+0 records in 2656+0 records out 2785017856 bytes (2.8 GB, 2.6 GiB) copied, 5.68203 s, 490 MB/s -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1978347 Title: horrible IO degradation with encrypted zfs root on kernels past 5.15.0-27 Status in linux package in Ubuntu: Confirmed Bug description: IO on encrypted zfs root has fallen off a cliff in kernel versions after 5.15.0-27 (the degradation is observed since version 5.15.0-30, also seen on -33 and -37, -25 and -27 work like a charm). Heavy usage almost hangs a new laptop (building large singularity images or synthetic testing with dd). I have confirmed that things are working as expected on a default + upgraded Ubuntu 22.04 desktop LVM+LUKS installation using another NVMe SSD on the same laptop. There seems to be a regression when using the native zfs encryption (did aes-ni acceleration get turned off?) How to reproduce: - install ubuntu 22.04 desktop from iso, don't install web updates, check use zfs and encryption - sudo apt update && apt install dstat htop - create a dataset with compression disabled so that dd actually writes things to disk * sudo zfs create rpool/dummy * sudo zfs set compress=off rpool/dummy * sudo chown -R myusername. /dummy - start dstat and htop in the background (show kernel threads in the htop config) - dd if=/dev/zero of=/dummy/bigfile bs=1M count=16384 - sudo apt upgrade and reboot on the latest kernel, repeat Expected: some cpu load, dstat reports write speeds about as much as the SSD can sustain (2.9-3GiB/s with a 2TiB Samsung 970 EVO Plus for a 16GiB write test, 1.4GiB/s for a few seconds then 800MiB/s sustained for whatever WD 512GiB model I had laying around). Observed on versions -30 and later: 700% or more system cpu load, mostly in z_wr_iss threads, writes top at around 150-180MiB/s, the system becomes somewhat unresponsive. Reads are also not good but I have not benchmarked. Booting the system and launching apps seems about normal due to the low IO load. ProblemType: Bug DistroRelease: Ubuntu 22.04 Package: linux-image-5.15.0-37-generic 5.15.0-37.39 ProcVersionSignature: Ubuntu 5.15.0-37.39-generic 5.15.35 Uname: Linux 5.15.0-37-generic x86_64 NonfreeKernelModules: zfs zunicode zcommon znvpair zavl icp ApportVersion: 2.20.11-0ubuntu82.1 Architecture: amd64 AudioDevicesInUse: USER PID ACCESS COMMAND /dev/snd/controlC0: laperlej 4100 F.... pulseaudio CRDA: N/A CasperMD5CheckResult: pass CurrentDesktop: ubuntu:GNOME Date: Fri Jun 10 15:52:12 2022 HibernationDevice: RESUME=none InstallationDate: Installed on 2022-05-10 (31 days ago) InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Release amd64 (20220419) MachineType: HP HP EliteBook 850 G8 Notebook PC ProcFB: 0 i915drmfb ProcKernelCmdLine: BOOT_IMAGE=/BOOT/ubuntu_rgwvzq@/vmlinuz-5.15.0-37-generic root=ZFS=rpool/ROOT/ubuntu_rgwvzq ro quiet splash vt.handoff=1 RelatedPackageVersions: linux-restricted-modules-5.15.0-37-generic N/A linux-backports-modules-5.15.0-37-generic N/A linux-firmware 20220329.git681281e4-0ubuntu3.2 SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 01/11/2022 dmi.bios.release: 8.0 dmi.bios.vendor: HP dmi.bios.version: T76 Ver. 01.08.00 dmi.board.name: 8846 dmi.board.vendor: HP dmi.board.version: KBC Version 30.37.00 dmi.chassis.type: 10 dmi.chassis.vendor: HP dmi.ec.firmware.release: 48.55 dmi.modalias: dmi:bvnHP:bvrT76Ver.01.08.00:bd01/11/2022:br8.0:efr48.55:svnHP:pnHPEliteBook850G8NotebookPC:pvr:rvnHP:rn8846:rvrKBCVersion30.37.00:cvnHP:ct10:cvr:sku4V1S3UP#ABL: dmi.product.family: 103C_5336AN HP EliteBook dmi.product.name: HP EliteBook 850 G8 Notebook PC dmi.product.sku: 4V1S3UP#ABL dmi.sys.vendor: HP To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1978347/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp