Adding qemu-block list. I'll summarize the question for qemu folks at the end.
On Fri, Dec 12, 2025 at 10:00:17 -0300, João Jandre wrote: > Hi, all > > I've been reading the QEMU documentation, the Libvirt documentation, as well > as doing some testing and code investigation regarding disk cache modes. > > In the Libvirt documentation, it is stated that the disk driver cache modes > accepted are "default", "none", "writethrough", "writeback", "directsync" > and "unsafe". Analyzing the code, we can see that this is somewhat reflected > on the cache modes that QEMU documents on the -drive option > (https://qemu-project.gitlab.io/qemu/system/invocation.html): > https://github.com/libvirt/libvirt/blob/04c1f458313e9001f5a804a898408e1f4985262b/src/qemu/qemu_domain.c#L10020-L10028 The intention behind that function was to preserve the behaviour from the time when libvirt was using -drive when I was converting libvirt to use -blockdev instead where the cache mode is configured by the 3 bool options. > However, the default option (when we do not define the "cache" option) is > different from other options and does not seem to map to any option on the While I no longer remeber the specifics (it's beel already almost 8 years since I've wrote the code) the code was designed to avoid setting any of the cache options in qemu (thus honour the 'hypervisor default'-type behaviour). Specifically the 'write-cache' option for the disk frontend device is defined as an "OnOffAuto" option and defaults to 'auto', which means we do set it to "default", whatever that means. Now the comment in the aforementioned function which I've copied from the qemu documentation doesn't have the line: "The default mode is cache=writeback." I unfortunately don't remember why thoug. It was present at the time I was copying it from there based on the commit history. > Qemu side. I see that, from testing and analyzing the code, if the default > (not defining the "cache" options) is used, cache.direct and cache.no-flush > are not informed; also, the write-cache option is not informed. > > Just for comparison, here is a disk defined with cache=writeback: > > XML: > > |<disk type='file' device='disk'> <driver name='qemu' type='qcow2' > cache='writeback'/> <source > file='/mnt/a66f48f5-13c6-3676-b320-a15d32bbf32f/c0e925a9-6697-4ee9-8abf-d36c7a95cea4' > index='2'/> <backingStore type='file' index='3'> <format type='qcow2'/> > <source > file='/mnt/a66f48f5-13c6-3676-b320-a15d32bbf32f/0db52809-9f33-43ac-8151-5452e9dee330'/> > <backingStore/> </backingStore> <target dev='vda' bus='virtio'/> > <serial>c0e925a966974ee98abf</serial> <alias name='virtio-disk0'/> <address > type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </disk> | > > Generated QEMU command: > > |-blockdev > {"driver":"file","filename":"/mnt/a66f48f5-13c6-3676-b320-a15d32bbf32f/c0e925a9-6697-4ee9-8abf-d36c7a95cea4","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"} > -blockdev > {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":"libvirt-3-format"} > -device > virtio-blk-pci,bus=pci.0,addr=0x5,drive=libvirt-2-format,id=virtio-disk0,bootindex=2,write-cache=on,serial=c0e925a966974ee98abf > | > > And here is a disk that does not have the cache option: > > XML: > > |<disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source > file='/mnt/b59eff0f-5b97-37ed-a513-9e1983d1d19b/b430296b-0924-412e-a7b1-ddc3d4c90f83' > index='2'/> <backingStore/> <target dev='vdc' bus='virtio'/> > <serial>b430296b0924412ea7b1</serial> <alias name='virtio-disk2'/> <address > type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </disk> | > > Generated QEMU command: > > |blockdev > {"driver":"file","filename":"/mnt/b59eff0f-5b97-37ed-a513-9e1983d1d19b/b430296b-0924-412e-a7b1-ddc3d4c90f83","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"} > -blockdev > {"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":null} > -device > virtio-blk-pci,bus=pci.0,addr=0x8,drive=libvirt-2-format,id=virtio-disk2,serial=b430296b0924412ea7b1 > | Indeed the differences in the commandlines what you see here are "itended" (in terms that I've likely wanted qemu to use the 'auto' mode of 'write-cache' whatever it is.). > The QEMU documentation states in the -drive option that the default cache > mode is |Writeback| (see > https://qemu-project.gitlab.io/qemu/system/invocation.html), so I expected > that setting |Writeback| or |Default| as the cache mode on Libvirt would > also be the same. I ran some performance tests using both options to try and > verify if they were the same, but they are clearly different. For each of > the configuration profiles below, I created a VM and ran some fio tests: > > Combination Disk cache mode Disk Controller IO thread > IO policy > hds0i default virtio-scsi 0 iouring > hds0t default virtio-scsi 0 threads > hds1i default virtio-scsi 1 iouring > hds1t default virtio-scsi 1 threads > hdv0i default virtio 0 iouring > hdv0t default virtio 0 threads > hdv1i default virtio 1 iouring > hdv1t default virtio 1 threads > wbs0i writeback virtio-scsi 0 iouring > wbs0t writeback virtio-scsi 0 threads > wbs1i writeback virtio-scsi 1 iouring > wbs1t writeback virtio-scsi 1 threads > wbv0i writeback virtio 0 iouring > wbv0t writeback virtio 0 threads > wbv1i writeback virtio 1 iouring > wbv1t writeback virtio 1 threads > > The fio test results are in a html attached to the email. > > I would like to understand what the practical difference is between not > informing the cache and any other option. I can see that the disk is defined > in different way, but just by analyzing the code I was not able to > understand what that leads to; also, looking at the performance tests, we > clearly see a difference in the results. Would anyone be able to explain > what is the expected behavior when the disk is defined in such a way? > > By the way, I'm using Libvirt 8.0.0 and Qemu 6.2.0. Libvirt uses the '-blockdev' mode and thus the above logic to pick the cache mode starting with qemu-4.2. In fact the support for '-drive' mode is no longer present in current upstream as we no-longer support anything older than qemu-6.2 thus always use the -blockdev mode.a While the _DEFAULT mode surely gives us the possibility to pick another combination of parameters, qemu-4.2 and thus our switch to -blockdev was 6 years ago. This gives a very long time when this combination of parameters was used and so I'm wary about changing the mode, unless it's bad for some reason. I'll thus CC the qemu folks with this message. TL;DR for qemu-block: When libvirt switched to '-blockdev' mode I've added a function which maps the cache modes used with '-drive' to the 3 options with blockdev. I for some reason chose that the "VIR_DOMAIN_DISK_CACHE_DEFAULT" mode is that *all* of the cache mode flags are just omitted relying on the default in qemu rather than going with the documented 'writeback' mode setting as default. Is the above combination problematic in any way? Should libvirt change the "default"?
