Am 05.02.26 um 5:26 PM schrieb Kevin Wolf: > Am 05.02.2026 um 11:45 hat Fiona Ebner geschrieben: >> Am 05.02.26 um 10:28 AM schrieb Kevin Wolf: >>> Am 05.02.2026 um 09:59 hat Fiona Ebner geschrieben: >>>> Hi, >>>> >>>> Am 03.02.26 um 3:24 PM schrieb Jean-Louis Dupond: >>>>> Hi, >>>>> >>>>> Since some months we were observing disk corruption within the VM when >>>>> enabling backups (which triggers snapshots). >>>>> After a lot of troubleshooting, we were able to track down the commit >>>>> that caused it: >>>>> https://gitlab.com/qemu-project/qemu/-/ >>>>> commit/058cfca5645a9ed7cb2bdb77d15f2eacaf343694 >>>>> >>>>> More info in the issue: >>>>> https://gitlab.com/qemu-project/qemu/-/issues/3273 >>>>> >>>>> Now this seems to be caused by a race between disabling the >>>>> dirty_bitmaps and the tracking implemented in the mirror top layer. >>>>> Kevin shared me a possible solution: >>>>> >>>>> diff --git a/block/mirror.c b/block/mirror.c >>>>> index b344182c747..f76e43f22c1 100644 >>>>> --- a/block/mirror.c >>>>> +++ b/block/mirror.c >>>>> @@ -1122,6 +1122,9 @@ static int coroutine_fn mirror_run(Job *job, Error >>>>> **errp) >>>>> * accessing it. >>>>> */ >>>>> mirror_top_opaque->job = s; >>>>> + if (s->copy_mode != MIRROR_COPY_MODE_WRITE_BLOCKING) { >>>>> + bdrv_disable_dirty_bitmap(s->dirty_bitmap); >>>>> + } >>>>> >>>>> assert(!s->dbi); >>>>> s->dbi = bdrv_dirty_iter_new(s->dirty_bitmap); >>>>> @@ -2018,7 +2021,9 @@ static BlockJob *mirror_start_job( >>>>> * The dirty bitmap is set by bdrv_mirror_top_do_write() when not in >>>>> active >>>>> * mode. >>>>> */ >>>>> - bdrv_disable_dirty_bitmap(s->dirty_bitmap); >>>>> + if (s->copy_mode == MIRROR_COPY_MODE_WRITE_BLOCKING) { >>>>> + bdrv_disable_dirty_bitmap(s->dirty_bitmap); >>>>> + } >>>>> >>>>> bdrv_graph_wrlock_drained(); >>>>> ret = block_job_add_bdrv(&s->common, "source", bs, 0, >>>>> >>>>> >>>>> Running this for some hours already, and it seems to fix the issue. >>>>> >>>>> Let's open up the discussion if this is the proper way to fix it, or if >>>>> there are better alternatives :) >>>> >>>> Yes, I do think this is the proper solution. I ended up with essentially >>>> the same yesterday (see the Gitlab issue). I moved the disabling >>>> unconditionally, but there should be no need to delay the disabling if >>>> using write-blocking mode. I would suggest moving the comment on top of >>>> the changed hunk in mirror_start_job() along to the new hunk in >>>> mirror_run(). >>> >>> I didn't actually write this as a proper patch, but just as a quick >>> thing Jean-Louis could test, so yes, all of the details are up for >>> discussion. >>> >>> You're right that there is no need to delay disabling the bitmap in >>> active mode, but it probably also doesn't hurt to keep it enabled for a >>> little while? Maybe we should do it like you did just to keep the code a >>> little simpler. >> >> Yes, it shouldn't hurt. Looking at the code again, >> should_copy_to_target() will evaluate to false before the job is set, so >> it might be necessary for write-blocking mode too? > > I didn't check that part yet, but that is actually a very good point! > > Yes, I agree, so your version of the fix is even more correct. It seems > write blocking mode is not actually write blocking until the very same > point in the code. (We should mention that in the comment as well.) > >>> There is another approach I had in mind, but wasn't as sure about, so I >>> didn't suggest it for the testing to see if this race is even the >>> problem we're seeing in practice: Is there a reason why populating the >>> initial dirty bitmap (i.e. the second part of mirror_dirty_init()) can't >>> just come after setting 'mirror_top_opaque->job'? Then we could simply >>> leave the bitmap disabled all the time and rely solely on the mirror >>> job's own tracking. That would feel a little more consistent than using >>> the block layer just to plug a small race window during startup. >> >> Interesting idea! In write-blocking mode, it will lead to writes during >> dirty bitmap initialization already being synced to the target, so >> leading to a bit of duplicate work. > > That's true, and I don't see an easy way to avoid it if we do things in > this order. Probably not a big deal in practice, but also a bit ugly.
We do support changing the copy mode after the job is started (see mirror_change()). If we really want to avoid the duplicate work, we could always start in background mode and switch to write-blocking only after initializing the bitmap, if that's what the user wanted. There is a comment in mirror_change() stating: The implementation relies on the fact that copy_mode is only written under the BQL. Otherwise, further synchronization would be required. so we'd need to be careful if going for this. Discussion from back then: https://lore.kernel.org/qemu-devel/[email protected]/ > >> Git history tells me why setting the job ended up where it is now: >> >>> commit 32125b14606a454ed109ea6d9da32c747e94926f >>> Author: Kevin Wolf <[email protected]> >>> Date: Fri Feb 3 16:21:41 2023 +0100 >>> >>> mirror: Fix access of uninitialised fields during start >>> >>> bdrv_mirror_top_pwritev() accesses the job object when active mirroring >>> is enabled. It disables this code during early initialisation while >>> s->job isn't set yet. >>> >>> However, s->job is still set way too early when the job object isn't >>> fully initialised. For example, &s->ops_in_flight isn't initialised yet >>> and the in_flight bitmap doesn't exist yet. This causes crashes when a >>> write request comes in too early. >>> >>> Move the assignment of s->job to when the mirror job is actually fully >>> initialised to make sure that the mirror_top driver doesn't access it >>> too early. >> >> Right before initializing the dirty bitmap, the other fields of the job >> object are already initialized, so the problem described in that commit >> message should not resurface. >> >> That said, not sure I'm not missing something. > > Yes, I think it requires splitting mirror_dirty_init() as I said, but > otherwise I don't see a problem at the moment. I could still be missing > something, of course. > >> Something else I'm wondering about: do we need to use atomics for >> setting and accessing MirrorBDSOpaque.job? At least with virtio-blk >> using iothread-vq-mapping, mirror_run() and bdrv_mirror_top_do_write() >> might be called in different threads. > > This might not even be enough. At first I thought that barriers might be > enough to make sure that MirrorBDSOpaque.job is set before the bitmap is > disabled. But bdrv_set_dirty() with its check if the dirty bitmap is > enabled happens only in bdrv_co_write_req_finish() after the request > completes. So we have a race where the switch could happen while a > request is in the middle between mirror_top_bs and bdrv_set_dirty() and > neither place would mark the request dirty. This is for the approach with setting job and then disabling the bitmap in mirror_run(). With your suggestion of having the bitmap disabled from the beginning and setting the job before initializing the bitmap, using barriers would be enough, or? How should we proceed, i.e. which approach and who should write the patch? Best Regards, Fiona
