We have additional technical details after investigating this with Microsoft Azure Linux team engineers.
The regression appears to be triggered by a change in Linux 6.17 that adjusts the default block-device readahead logic for devices reported as rotational. Relevant change: https://github.com/torvalds/linux/commit/459779d04ae8dfd4083679a7bf9d72af165d1023 On Azure, non-NVMe disks (in particular Standard SSD, which is where we are affected) are presented to the guest OS as rotational (`queue/rotational=1`, `ROTA=1` in `lsblk`). With 6.17, this results in a much larger default `read_ahead_kb` (we observe 4096 KiB). For certain random-access workloads (in our case RavenDB using memory-mapped files), this can lead to severe performance degradation because page faults pull in MiB-scale readahead and flood the page cache with unused data. Quick verification on affected VMs: - `lsblk -d -o NAME,ROTA` shows `ROTA=1` for the OS/data disks - `/sys/block/<dev>/queue/read_ahead_kb` defaults to 4096 on 6.17 Current mitigation is to force a lower `read_ahead_kb` via udev rule (applied at device add/change): ``` sudo tee /etc/udev/rules.d/99-readahead.rules << 'EOF' ACTION=="add|change", KERNEL=="sd*", ATTR{queue/read_ahead_kb}="128" EOF sudo udevadm control --reload-rules sudo udevadm trigger ``` This restores expected performance for our workload. From our understanding, the complete picture is an interaction between: - Linux 6.17 readahead defaulting logic for rotational devices, and - how the Azure hypervisor/storage stack reports non-NVMe disks to the guest OS (as rotational). For comparison, on other platforms we tested (AWS and GCP), SSD-backed disks are not reported as rotational to the guest (i.e. `ROTA=0`), and we do not observe the same default `read_ahead_kb` increase. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2143713 Title: Performance regression between 6.14.0-1017 and 6.17.0-1008.8 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+bug/2143713/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
