Hi Kevin,
Am 21.05.26 um 3:46 PM schrieb Kevin Wolf:
> Am 21.05.2026 um 14:12 hat Fiona Ebner geschrieben:
>> Am 27.04.26 um 7:04 PM schrieb Kevin Wolf:
>>> Most code in qcow2 that accesses (and potentially modifies) L2 tables
>>> does so while holding s->lock.
>>>
>>> There is one exception, which is allocating writes. They hold the lock
>>> initially while allocating clusters, but drop it for writing the guest
>>> payload before taking the lock again for updating the L2 tables. This
>>> allows concurrent requests that touch other parts of the image file to
>>> continue in parallel and is an important performance optimisation.
>>>
>>> However, this means that other requests that run while the lock is
>>> dropped for writing guest data must synchronise with the list of
>>> allocating requests in s->cluster_allocs and wait if they would overlap.
>>> For writes, this is done in handle_dependencies(), but discard and write
>>> zeros operations neglect to synchronise with s->cluster_allocs.
>>>
>>> This means that discard can free a cluster whose L2 entry will already
>>> be modified in qcow2_alloc_cluster_link_l2() by a previously started
>>> write. In the case of a pre-allocated zero cluster that is in the
>>> process of being overwritten, this means that discard can lead to a
>>> situation where the cluster is still mapped (because the write will
>>> restore the L2 entry just without the zero flag), but its refcount has
>>> been decreased, resulting in a corrupted image.
>>>
>>> Add the missing synchronisation to qcow2_cluster_discard() and
>>> qcow2_subcluster_zeroize() to fix the problem.
>>>
>>> Cc: [email protected]
>>> Reported-by: Denis V. Lunev <[email protected]>
>>> Signed-off-by: Kevin Wolf <[email protected]>
>>
>> we had started rolling out a build of QEMU 11 with this patch already
>> included. However, some of our users reported issues with VMs using
>> qcow2 disks soon after [0][1]. I was able to reproduce the in-guest
>> segfaults from [1] in a memory-constrained Debian 12 guest when using a
>> swap partition on the same disk. Thanks to Thomas for the hunch with
>> swap! After reverting this patch, I wasn't able to reproduce the issue
>> anymore. I do not have a better reproducer yet and am not sure about the
>> exact pattern causing the issue. It's related to the
>> wait_for_dependencies() call in qcow2_subcluster_zeroize(), because if I
>> revert just the one in qcow2_cluster_discard(), the issue still reproduces.
>>
>> Commandline for my reproducer VM [2]. The issue does not happen if I
>> drop "detect-zeroes":"unmap". Note that I don't have discard-no-unref
>> for the qcow2 image, so in zero_in_l2_slice(), the branch with
>> qcow2_free_any_cluster() is taken. Could the conflict be related to that?
>>
>> I'm still trying to figure things out and come up with a better
>> reproducer, but wanted to let you know early, also because of the
>> upcoming stable releases. Of course, I'd also be happy for hints/hunches
>> and am happy to test suggestions!
>
> Do you have any information about the options used with the image file?
> In particular, is it using subclusters? Maybe just the 'qemu-img info'
> output would already give a bit more context.
No subclusters if I'm not missing anything. When I created the image the
output was:
Formatting '/mnt/pve/dir/images/300/vm-300-disk-0.qcow2', fmt=qcow2
cluster_size=65536 extended_l2=off preallocation=metadata
compression_type=zlib size=4510973952 lazy_refcounts=off refcount_bits=16
Our management layer doesn't log the command itself, but doing the same
operation with logging added (and 301 instead of 300):
/usr/bin/qemu-img create -o preallocation=metadata -f qcow2
/mnt/pve/dir/images/301/vm-301-disk-0.qcow2 4405248K
qemu-img info gives:
{
"children": [
{
"name": "file",
"info": {
"children": [
],
"virtual-size": 6018105344,
"filename": "/mnt/pve/dir/images/300/vm-300-disk-0.qcow2",
"format": "file",
"actual-size": 3794231296,
"format-specific": {
"type": "file",
"data": {
"extent-size-hint": 1048576
}
},
"dirty-flag": false
}
}
],
"snapshots": [
{
"icount": 0,
"vm-clock-nsec": 0,
"name": "s0",
"date-sec": 1779354489,
"date-nsec": 415278000,
"vm-clock-sec": 0,
"id": "1",
"vm-state-size": 0
}
],
"virtual-size": 4510973952,
"filename": "/mnt/pve/dir/images/300/vm-300-disk-0.qcow2",
"cluster-size": 65536,
"format": "qcow2",
"actual-size": 3794231296,
"format-specific": {
"type": "qcow2",
"data": {
"compat": "1.1",
"compression-type": "zlib",
"lazy-refcounts": false,
"refcount-bits": 16,
"corrupt": false,
"extended-l2": false
}
},
"dirty-flag": false
}
> Could you already locate the actual corruption and check what the
> pattern looks like? Something like zeros where we would expect data or
> the other way around? Or something less clear? (If you don't know,
> that's a good answer too. I know well that this kind of things is hard
> to debug.)
Unfortunately not. I can only see the symptom of memory swapped back in
being corrupt (at least that's what happens AFAIU), leading to segfaults
in various processes as well as issues with heap allocations, e.g.:
corrupted double-linked list
free(): invalid pointer
I'll write a small program which allocates memory with a fixed pattern
and regularly dumps it, maybe that works to get an idea about the
corruption.
Best Regards,
Fiona
