On Thu, Dec 07, 2023 at 02:04:26PM -0700, Jens Axboe wrote:
> On Mon, 14 Aug 2023 15:41:00 +0100, Matthew Wilcox (Oracle) wrote:
>> The special casing was originally added in pre-git history; reproducing
>> the commit log here:
>>
>>> commit a318a92567d77
>>> Author: Andrew Morton <a...@osdl.org>
>>> Date: Sun Sep 21 01:42:22 2003 -0700
>>>
>>> [PATCH] Speed up direct-io hugetlbpage handling
>>>
>>> This patch short-circuits all the direct-io page dirtying logic for
>>> higher-order pages. Without this, we pointlessly bounce BIOs up to
>>> keventd all the time.
>>
>> [...]
>
> Applied, thanks!
>
> [1/1] block: Remove special-casing of compound pages
> commit: 1b151e2435fc3a9b10c8946c6aebe9f3e1938c55
This commit results in a change of behavior for QEMU VMs backed by hugepages
that open their VM disk image file with O_DIRECT (QEMU cache=none or
cache.direct=on options). When the VM shuts down and the QEMU process exits,
one or two hugepages may fail to free correctly. It appears to be a race, as
it doesn't happen every time.
>From debugging on 6.8-rc6, when it occurs, the hugepage that fails to free has
a non-zero refcount when it hits the folio_put_testzero(folio) test in
release_pages(). On a failure test iteration with 1 GiB hugepages, the failing
folio had a mapcount of 0, refcount of 35, and folio_maybe_dma_pinned was true.
The problem only occurs when the VM disk image file is opened with O_DIRECT.
When using QEMU cache=writeback or cache.direct=off options, it does not occur.
We first noticed it on the 6.1.y stable kernel when this commit landed there
(6.1.75).
A very simple reproducer without KVM (just boot VM up, then shut it down):
echo 512 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
qemu-system-x86_64 \
-cpu qemu64 \
-m 1024 \
-nographic \
-mem-path /dev/hugepages/vm00 \
-mem-prealloc \
-drive file=test.qcow2,if=none,cache=none,id=drive0 \
-device virtio-blk-pci,drive=drive0,id=disk0,bootindex=1
rm -f /dev/hugepages/vm00
Some testing notes:
* occurs with 6.1.75, 6.6.14, 6.8-rc6, and linux-next-20240229
* occurs with 1 GiB and 2 MiB huge pages, with both hugetlbfs and memfd
* occurs with QEMU 8.0.y, 8.1.y, 8.2.y, and master
* occurs with (-enable-kvm -cpu host) or without (-cpu qemu64) KVM
Thanks for your time!
Greg
