On 6/16/26 3:25 PM, Gavin Shan wrote:
All ram device regions was turned to be indirectly accessible by commit 4a2e242bbb ("memory: Don't use memcpy for ram_device regions"). This leads to a hanged guest where a NVidia GH100 GPU is passed from host. The memory in its PCI BAR#4 can be allocated as DMA target buffer. qemu has to take DMA bounce buffer in address_space_map() to cover the DMA request. However, the bounce buffer size is 4096 bytes and we're overrunning it easily when the guest has significant disk activities on compiling 'cuda-samples'. The full log and problem description can be found from PATCH[1/2]'s commit log.Try to fix the issue handled in commit 4a2e242bbb by replacing memcopy()/ memmove() with newly added helpers qemu_ram_{copy, move}() that works on top of __builtin_{memcpy, memmove} or unaligned access friendly memory movement in the accessors to the ram device regions. With this, we can basically revert that commit to make ram device region directly accessible again and bypass the bounce buffer in address_space_map() where the guest hang is caused. PATCH[1] uses qemu_ram_{copy, move}() in ram device region accessors PATCH[2] makes ram device region directly accessible again
Michael asked to include below context in the cover letter in v3, but I didn't noticed that before I sent v3 series, appended with them. ---- The issues listed by Michael: 1. On x86, memcpy is different from __builtin_memcpy if one uses old 1.0 force-headers from 2019. Likely no longer relevant. 2. variable length memcpy can translate 2,4,8 byte guest access into multiple byte accesses. doing this for mmio is guaranteed to break devices. 3. (theoretical concern) also on x86, unaligned accesses are possible on guest and host, so converting an unaligned access to a series of aligned ones can in theory break devices. 4. also on x86, vector instructions for large (>16 byte) writes into pgprot_noncached memory are safe and faster than multiple 8 byte ones. 5. also on x86 it so happens that if you write a fixed-size memcpy this gets optimized to a single store/load and it works for aligned and unaligned addresses on that architecture. How to ensure this keeps being correct is left as an excerise for the reader. But qemu already relies on this and did for years. 6. on non-x86 both unaligned accesses and vector instructions for accessing UC memory are illegal. 7. standard vfio gives KVM VM_ALLOW_ANY_UNCACHED, so even on non x86 guest can map the memory as as pgprot_noncached/ioremap or pgprot_writecombine/ioremap_uc. If it does the second then it can use unaligned or vector for access. This is why normal passthrough tends to work - it never traps to qemu at all. But for qemu, vfio uses pgprot_noncached unconditionally so qemu can't use unaligned or vector instructions on non-x86. 8. But for nvgrace RAM, vfio has a driver that uses pgprot_writecombine/ioremap_uc. so qemu could safely use unaligned/vector instructioons even on non-x86. 9. Except sadly, vfio currently does not tell qemu how it maps the memory, so qemu can not know what is safe on non-x86. Now, what is to be done? A. on x86, we must avoid converting 2,4,8 byte accesses into byte accesses. At least for aligned, perferably for unaligned accesses too. Fixed width memcpy seems to work for this. Whether we should bother with __builtin to work around broken old fortify headers, I donnu. I do not have any answer how to check that compiler does this correctly. If anyone is motivated enough, adding a GCC builtin could be possible. Given qemu did this for years, I think we can leave solving this for another day. B. Also on x86, I do not see why we should not use memcpy for large accesses if we can. Better perf. C. on non-x86, we currently must not memcpy since we do not know if it is pgprot_noncached. yes, performance will be bad for DMA into device RAM. D. It goes without saying that casting an unaligned address to unint32_t (be it for qatomic_set or whatever) is undefined behaviour in C and so a bad idea on any architecture. E. also for non-x86, we really should teach vfio to tell qemu whether it maps device pgprot_noncached or pgprot_writecombine. we will then be able to use memcpy for >8 accesses. Anyone, correct me if I'm wrong? Maybe I should start a new thread with this summary? Thanks, Gavin
