Public bug reported:
IO on encrypted zfs root has fallen off a cliff in kernel versions after
5.15.0-27 (the degradation is observed since version 5.15.0-30, also
seen on -33 and -37, -25 and -27 work like a charm). Heavy usage almost
hangs a new laptop (building large singularity images or synthetic
testing with dd).
I have confirmed that things are working as expected on a default +
upgraded Ubuntu 22.04 desktop LVM+LUKS installation using another NVMe
SSD on the same laptop. There seems to be a regression when using the
native zfs encryption (did aes-ni acceleration get turned off?)
How to reproduce:
- install ubuntu 22.04 desktop from iso, don't install web updates, check use
zfs and encryption
- sudo apt update && apt install dstat htop
- create a dataset with compression disabled so that dd actually writes things
to disk
* sudo zfs create rpool/dummy
* sudo zfs set compress=off rpool/dummy
* sudo chown -R myusername. /dummy
- start dstat and htop in the background (show kernel threads in the htop
config)
- dd if=/dev/zero of=/dummy/bigfile bs=1M count=16384
- sudo apt upgrade and reboot on the latest kernel, repeat
Expected: some cpu load, dstat reports write speeds about as much as the
SSD can sustain (2.9-3GiB/s with a 2TiB Samsung 970 EVO Plus for a 16GiB
write test, 1.4GiB/s for a few seconds then 800MiB/s sustained for
whatever WD 512GiB model I had laying around).
Observed on versions -30 and later: 700% or more system cpu load, mostly
in z_wr_iss threads, writes top at around 150-180MiB/s, the system
becomes somewhat unresponsive. Reads are also not good but I have not
benchmarked. Booting the system and launching apps seems about normal
due to the low IO load.
ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: linux-image-5.15.0-37-generic 5.15.0-37.39
ProcVersionSignature: Ubuntu 5.15.0-37.39-generic 5.15.35
Uname: Linux 5.15.0-37-generic x86_64
NonfreeKernelModules: zfs zunicode zcommon znvpair zavl icp
ApportVersion: 2.20.11-0ubuntu82.1
Architecture: amd64
AudioDevicesInUse:
USER PID ACCESS COMMAND
/dev/snd/controlC0: laperlej 4100 F.... pulseaudio
CRDA: N/A
CasperMD5CheckResult: pass
CurrentDesktop: ubuntu:GNOME
Date: Fri Jun 10 15:52:12 2022
HibernationDevice: RESUME=none
InstallationDate: Installed on 2022-05-10 (31 days ago)
InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Release amd64 (20220419)
MachineType: HP HP EliteBook 850 G8 Notebook PC
ProcFB: 0 i915drmfb
ProcKernelCmdLine: BOOT_IMAGE=/BOOT/ubuntu_rgwvzq@/vmlinuz-5.15.0-37-generic
root=ZFS=rpool/ROOT/ubuntu_rgwvzq ro quiet splash vt.handoff=1
RelatedPackageVersions:
linux-restricted-modules-5.15.0-37-generic N/A
linux-backports-modules-5.15.0-37-generic N/A
linux-firmware 20220329.git681281e4-0ubuntu3.2
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 01/11/2022
dmi.bios.release: 8.0
dmi.bios.vendor: HP
dmi.bios.version: T76 Ver. 01.08.00
dmi.board.name: 8846
dmi.board.vendor: HP
dmi.board.version: KBC Version 30.37.00
dmi.chassis.type: 10
dmi.chassis.vendor: HP
dmi.ec.firmware.release: 48.55
dmi.modalias:
dmi:bvnHP:bvrT76Ver.01.08.00:bd01/11/2022:br8.0:efr48.55:svnHP:pnHPEliteBook850G8NotebookPC:pvr:rvnHP:rn8846:rvrKBCVersion30.37.00:cvnHP:ct10:cvr:sku4V1S3UP#ABL:
dmi.product.family: 103C_5336AN HP EliteBook
dmi.product.name: HP EliteBook 850 G8 Notebook PC
dmi.product.sku: 4V1S3UP#ABL
dmi.sys.vendor: HP
** Affects: linux (Ubuntu)
Importance: Undecided
Status: Confirmed
** Tags: amd64 apport-bug jammy wayland-session
** Attachment added: "dmesg-outputs.tar.gz"
https://bugs.launchpad.net/bugs/1978347/+attachment/5596539/+files/dmesg-outputs.tar.gz
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1978347
Title:
horrible IO degradation with encrypted zfs root on kernels past
5.15.0-27
Status in linux package in Ubuntu:
Confirmed
Bug description:
IO on encrypted zfs root has fallen off a cliff in kernel versions
after 5.15.0-27 (the degradation is observed since version 5.15.0-30,
also seen on -33 and -37, -25 and -27 work like a charm). Heavy usage
almost hangs a new laptop (building large singularity images or
synthetic testing with dd).
I have confirmed that things are working as expected on a default +
upgraded Ubuntu 22.04 desktop LVM+LUKS installation using another NVMe
SSD on the same laptop. There seems to be a regression when using the
native zfs encryption (did aes-ni acceleration get turned off?)
How to reproduce:
- install ubuntu 22.04 desktop from iso, don't install web updates, check use
zfs and encryption
- sudo apt update && apt install dstat htop
- create a dataset with compression disabled so that dd actually writes
things to disk
* sudo zfs create rpool/dummy
* sudo zfs set compress=off rpool/dummy
* sudo chown -R myusername. /dummy
- start dstat and htop in the background (show kernel threads in the htop
config)
- dd if=/dev/zero of=/dummy/bigfile bs=1M count=16384
- sudo apt upgrade and reboot on the latest kernel, repeat
Expected: some cpu load, dstat reports write speeds about as much as
the SSD can sustain (2.9-3GiB/s with a 2TiB Samsung 970 EVO Plus for a
16GiB write test, 1.4GiB/s for a few seconds then 800MiB/s sustained
for whatever WD 512GiB model I had laying around).
Observed on versions -30 and later: 700% or more system cpu load,
mostly in z_wr_iss threads, writes top at around 150-180MiB/s, the
system becomes somewhat unresponsive. Reads are also not good but I
have not benchmarked. Booting the system and launching apps seems
about normal due to the low IO load.
ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: linux-image-5.15.0-37-generic 5.15.0-37.39
ProcVersionSignature: Ubuntu 5.15.0-37.39-generic 5.15.35
Uname: Linux 5.15.0-37-generic x86_64
NonfreeKernelModules: zfs zunicode zcommon znvpair zavl icp
ApportVersion: 2.20.11-0ubuntu82.1
Architecture: amd64
AudioDevicesInUse:
USER PID ACCESS COMMAND
/dev/snd/controlC0: laperlej 4100 F.... pulseaudio
CRDA: N/A
CasperMD5CheckResult: pass
CurrentDesktop: ubuntu:GNOME
Date: Fri Jun 10 15:52:12 2022
HibernationDevice: RESUME=none
InstallationDate: Installed on 2022-05-10 (31 days ago)
InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Release amd64
(20220419)
MachineType: HP HP EliteBook 850 G8 Notebook PC
ProcFB: 0 i915drmfb
ProcKernelCmdLine: BOOT_IMAGE=/BOOT/ubuntu_rgwvzq@/vmlinuz-5.15.0-37-generic
root=ZFS=rpool/ROOT/ubuntu_rgwvzq ro quiet splash vt.handoff=1
RelatedPackageVersions:
linux-restricted-modules-5.15.0-37-generic N/A
linux-backports-modules-5.15.0-37-generic N/A
linux-firmware 20220329.git681281e4-0ubuntu3.2
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 01/11/2022
dmi.bios.release: 8.0
dmi.bios.vendor: HP
dmi.bios.version: T76 Ver. 01.08.00
dmi.board.name: 8846
dmi.board.vendor: HP
dmi.board.version: KBC Version 30.37.00
dmi.chassis.type: 10
dmi.chassis.vendor: HP
dmi.ec.firmware.release: 48.55
dmi.modalias:
dmi:bvnHP:bvrT76Ver.01.08.00:bd01/11/2022:br8.0:efr48.55:svnHP:pnHPEliteBook850G8NotebookPC:pvr:rvnHP:rn8846:rvrKBCVersion30.37.00:cvnHP:ct10:cvr:sku4V1S3UP#ABL:
dmi.product.family: 103C_5336AN HP EliteBook
dmi.product.name: HP EliteBook 850 G8 Notebook PC
dmi.product.sku: 4V1S3UP#ABL
dmi.sys.vendor: HP
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1978347/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
