** Description changed:
I noticed performance issues on the server running some backend
applications after updating OS packages with Ubuntu Jammy. After further
investigation, the slowness was caused by disk IO.
After the update, running on kernel 5.15.0-112, the disk writes take
about twice as long. I can reproduce this by changing the GRUB
configuration to boot into kernel 5.15.0-107, where the writes are
- consistently slower. See below.
+ consistently faster than on the newer kernel. See below.
# uname -a
Linux dev03 5.15.0-112-generic #122-Ubuntu SMP Thu May 23 07:48:21 UTC 2024
x86_64 x86_64 x86_64 GNU/Linux
# time dd if=/dev/urandom of=deleteme bs=1024 count=100
100+0 records in
100+0 records out
102400 bytes (1.0 GB, 977 MiB) copied, 6.54711 s, 156 MB/s
real 0m6.782s
user 0m0.192s
sys 0m5.968s
# uname -a
Linux dev03 5.15.0-107-generic #117-Ubuntu SMP Fri Apr 26 12:26:49 UTC 2024
x86_64 x86_64 x86_64 GNU/Linux
# time dd if=/dev/urandom of=deleteme bs=1024 count=100
100+0 records in
100+0 records out
102400 bytes (1.0 GB, 977 MiB) copied, 3.77936 s, 271 MB/s
real 0m3.781s
user 0m0.132s
sys 0m3.648s
-
These times are with NVMe drives on ext4 filesystem. I observe the same
performance difference for ZFS filesystem, and also with HDDs (spinning
drives). The disk performance is also worse for 6.5.0-41 kernel. I also
tried 6.5.0-35 and linux-image-generic-hwe-22.04 - same problem - writes
are slow.
ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: linux-image-5.15.0-112-generic 5.15.0-112.122
ProcVersionSignature: Ubuntu 5.15.0-112.122-generic 5.15.152
Uname: Linux 5.15.0-112-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
AlsaDevices:
- total 0
- drwxr-xr-x 2 root root 60 Jun 24 08:23 by-path
- crw-rw 1 root audio 116, 2 Jun 24 08:23 controlC0
- crw-rw 1 root audio 116, 1 Jun 24 08:23 seq
- crw-rw 1 root audio 116, 33 Jun 24 08:23 timer
+ total 0
+ drwxr-xr-x 2 root root 60 Jun 24 08:23 by-path
+ crw-rw 1 root audio 116, 2 Jun 24 08:23 controlC0
+ crw-rw 1 root audio 116, 1 Jun 24 08:23 seq
+ crw-rw 1 root audio 116, 33 Jun 24 08:23 timer
AlsaVersion: Advanced Linux Sound Architecture Driver Version
k5.15.0-112-generic.
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu82.5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path',
'/dev/snd/controlC0', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
Card0.Amixer.info: Error: [Errno 2] No such file or directory: 'amixer'
Card0.Amixer.values: Error: [Errno 2] No such file or directory: 'amixer'
CasperMD5CheckResult: unknown
Date: Mon Jun 24 10:24:44 2024
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb:
- Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
- Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
- Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
- Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
+ Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
+ Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
+ Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
+ Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Lsusb-t:
- /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 1M
- /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
- /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 1M
- /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 480M
+ /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 1M
+ /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
+ /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 1M
+ /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 480M
ProcEnviron:
- TERM=xterm-256color
- PATH=(custom, no user)
- XDG_RUNTIME_DIR=
- LANG=en_US.UTF-8
- SHELL=/bin/bash
+ TERM=xterm-256color
+ PATH=(custom, no user)
+ XDG_RUNTIME_DIR=
+ LANG=en_US.UTF-8
+ SHELL=/bin/bash
ProcFB:
-
+
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.15.0-112-generic
root=UUID=3bff164a-d728-43bd-a926-ad3be6ff4715 ro consoleblank=0
systemd.show_status=true nomodeset consoleblank=0
RelatedPackageVersions:
- linux-restricted-modules-5.15.0-112-generic N/A
- linux-backports-modules-5.15.0-112-generic N/A
- linux-firmware 20220329.git681281e4-0ubuntu3.31
+ linux-restricted-modules-5.15.0-112-generic N/A
+ linux-backports-modules-5.15.0-112-generic N/A
+ linux-firmware 20220329.git681281e4-0ubuntu3.31
RfKill: Error: [Errno 2] No suc
