For more context, openssl using aes-256-cbc (not aes-256-gcm, which is unsupported by default) appears to behave identically between the two kernels, and faster than when disabling aes-ni, therefore aes-ni appears to be enabled in both.
# uname -a Linux aero 5.15.0-37-generic #39-Ubuntu SMP Wed Jun 1 19:16:45 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux # openssl speed aes-256-cbc Doing aes-256-cbc for 3s on 16 size blocks: 186683511 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 64 size blocks: 50188595 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 256 size blocks: 12770138 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 1024 size blocks: 3204571 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 8192 size blocks: 400190 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 16384 size blocks: 199936 aes-256-cbc's in 3.00s version: 3.0.2 built on: Thu May 5 08:04:52 2022 UTC options: bn(64,64) compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/openssl-Ke3YUO/openssl-3.0.2=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_TLS_SECURITY_LEVEL=2 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2 CPUINFO: OPENSSL_ia32cap=0x7ffef3ffffebffff:0x818d39ef7eb The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes aes-256-cbc 995645.39k 1070690.03k 1089718.44k 1093826.90k 1092785.49k 1091917.14k # uname -a Linux aero 5.15.0-27-generic #28-Ubuntu SMP Thu Apr 14 04:55:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux # openssl speed aes-256-cbc Doing aes-256-cbc for 3s on 16 size blocks: 186867669 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 64 size blocks: 50246758 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 256 size blocks: 12765344 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 1024 size blocks: 3204376 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 8192 size blocks: 399400 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 16384 size blocks: 200109 aes-256-cbc's in 3.00s version: 3.0.2 built on: Thu May 5 08:04:52 2022 UTC options: bn(64,64) compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/openssl-Ke3YUO/openssl-3.0.2=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_TLS_SECURITY_LEVEL=2 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2 CPUINFO: OPENSSL_ia32cap=0x7ffef3ffffebffff:0x818d39ef7eb The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes aes-256-cbc 996627.57k 1071930.84k 1089309.35k 1093760.34k 1090628.27k 1092861.95k # OPENSSL_ia32cap= openssl speed aes-256-cbc Doing aes-256-cbc for 3s on 16 size blocks: 20600986 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 64 size blocks: 5866219 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 256 size blocks: 1517023 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 1024 size blocks: 885706 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 8192 size blocks: 112410 aes-256-cbc's in 3.00s Doing aes-256-cbc for 3s on 16384 size blocks: 56135 aes-256-cbc's in 3.00s version: 3.0.2 built on: Thu May 5 08:04:52 2022 UTC options: bn(64,64) compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/openssl-Ke3YUO/openssl-3.0.2=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_TLS_SECURITY_LEVEL=2 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2 CPUINFO: OPENSSL_ia32cap=0x400:0x0 env: The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes aes-256-cbc 109871.93k 125146.01k 129452.63k 302320.98k 306954.24k 306571.95k -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1978347 Title: horrible IO degradation with encrypted zfs root on kernels past 5.15.0-27 Status in linux package in Ubuntu: Confirmed Bug description: IO on encrypted zfs root has fallen off a cliff in kernel versions after 5.15.0-27 (the degradation is observed since version 5.15.0-30, also seen on -33 and -37, -25 and -27 work like a charm). Heavy usage almost hangs a new laptop (building large singularity images or synthetic testing with dd). I have confirmed that things are working as expected on a default + upgraded Ubuntu 22.04 desktop LVM+LUKS installation using another NVMe SSD on the same laptop. There seems to be a regression when using the native zfs encryption (did aes-ni acceleration get turned off?) How to reproduce: - install ubuntu 22.04 desktop from iso, don't install web updates, check use zfs and encryption - sudo apt update && apt install dstat htop - create a dataset with compression disabled so that dd actually writes things to disk * sudo zfs create rpool/dummy * sudo zfs set compress=off rpool/dummy * sudo chown -R myusername. /dummy - start dstat and htop in the background (show kernel threads in the htop config) - dd if=/dev/zero of=/dummy/bigfile bs=1M count=16384 - sudo apt upgrade and reboot on the latest kernel, repeat Expected: some cpu load, dstat reports write speeds about as much as the SSD can sustain (2.9-3GiB/s with a 2TiB Samsung 970 EVO Plus for a 16GiB write test, 1.4GiB/s for a few seconds then 800MiB/s sustained for whatever WD 512GiB model I had laying around). Observed on versions -30 and later: 700% or more system cpu load, mostly in z_wr_iss threads, writes top at around 150-180MiB/s, the system becomes somewhat unresponsive. Reads are also not good but I have not benchmarked. Booting the system and launching apps seems about normal due to the low IO load. ProblemType: Bug DistroRelease: Ubuntu 22.04 Package: linux-image-5.15.0-37-generic 5.15.0-37.39 ProcVersionSignature: Ubuntu 5.15.0-37.39-generic 5.15.35 Uname: Linux 5.15.0-37-generic x86_64 NonfreeKernelModules: zfs zunicode zcommon znvpair zavl icp ApportVersion: 2.20.11-0ubuntu82.1 Architecture: amd64 AudioDevicesInUse: USER PID ACCESS COMMAND /dev/snd/controlC0: laperlej 4100 F.... pulseaudio CRDA: N/A CasperMD5CheckResult: pass CurrentDesktop: ubuntu:GNOME Date: Fri Jun 10 15:52:12 2022 HibernationDevice: RESUME=none InstallationDate: Installed on 2022-05-10 (31 days ago) InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Release amd64 (20220419) MachineType: HP HP EliteBook 850 G8 Notebook PC ProcFB: 0 i915drmfb ProcKernelCmdLine: BOOT_IMAGE=/BOOT/ubuntu_rgwvzq@/vmlinuz-5.15.0-37-generic root=ZFS=rpool/ROOT/ubuntu_rgwvzq ro quiet splash vt.handoff=1 RelatedPackageVersions: linux-restricted-modules-5.15.0-37-generic N/A linux-backports-modules-5.15.0-37-generic N/A linux-firmware 20220329.git681281e4-0ubuntu3.2 SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 01/11/2022 dmi.bios.release: 8.0 dmi.bios.vendor: HP dmi.bios.version: T76 Ver. 01.08.00 dmi.board.name: 8846 dmi.board.vendor: HP dmi.board.version: KBC Version 30.37.00 dmi.chassis.type: 10 dmi.chassis.vendor: HP dmi.ec.firmware.release: 48.55 dmi.modalias: dmi:bvnHP:bvrT76Ver.01.08.00:bd01/11/2022:br8.0:efr48.55:svnHP:pnHPEliteBook850G8NotebookPC:pvr:rvnHP:rn8846:rvrKBCVersion30.37.00:cvnHP:ct10:cvr:sku4V1S3UP#ABL: dmi.product.family: 103C_5336AN HP EliteBook dmi.product.name: HP EliteBook 850 G8 Notebook PC dmi.product.sku: 4V1S3UP#ABL dmi.sys.vendor: HP To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1978347/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
