On 10/27/2026 2:54 PM, Fabiano Rosas wrote: > Pierrick Bouvier <[email protected]> writes: > >> On 10/27/2026 7:36 AM, Fabiano Rosas wrote: >>> Pierrick Bouvier <[email protected]> writes: >>> >>>> On 10/26/2026 8:32 AM, Fabiano Rosas wrote: >>>>> Pierrick Bouvier <[email protected]> writes: >>>>> >>>>>> On 10/23/2026 8:33 AM, Fabiano Rosas wrote: >>>>>>> Pierrick Bouvier <[email protected]> writes: >>>>>>> >>>>>>>> This series was inspired by a recent post from Fabiano to provide >>>>>>>> container >>>>>>>> images for migration tests: >>>>>>>> https://lore.kernel.org/qemu-devel/[email protected]/T/#m353f119dc1a24fa437dde8a96d1891629e64303c >>>>>>>> >>>>>>>> The idea to be able to build and run any QEMU (past) release with >>>>>>>> containers was >>>>>>>> in my mind for some time, so I gave it a shot. Finally, it was easier >>>>>>>> than >>>>>>>> expected to obtain a single dockerfile (with a few hacks) that works >>>>>>>> over all >>>>>>>> our releases. >>>>>>>> >>>>>>> >>>>>>> Good stuff, thanks for this work. I'm just not sure how much of it fits >>>>>>> upstream in the current shape, let's discuss... >>>>>>> >>>>>>> You say this was in your mind for a while, could you elaborate a bit on >>>>>>> how you think this could benefit the project? >>>>>>> >>>>>> >>>>>> First usage that comes to my mind is that it would greatly help for >>>>>> migration, allowing automatic testing in CI, by just downloading any >>>>>> previous QEMU, dump a machine, and restore it on current version. >>>>> >>>>> Don't we also build QEMU several times over in CI? Maybe I'm wrong, but >>>>> it might allow us to decouple a few tests from the build steps. >>>>> >>>> >>>> This is for current version, not past ones. I'm not sure it's worth >>>> adding an identical build for past versions on every pipeline. >>>> >>> >>> I thought we re-built current qemu a few times, no? We have containers >>> for various distros and then make; make check inside of them. The >>> question was whether a step 0 to build the binary container for this >>> release would allow some builds to be skipped. >>> >> >> I see. To some extent, yes. However, if you take a look more closely, >> you'll notice most of the builds differ by: >> - set of targets built >> - configure options >> - distro target >> >> While we could maybe optimize some parts, that still requires duplication. >> >>>>>> We could do it for last 10 releases to ensure migration from packaged >>>>>> versions in all supported distros (red hat, alpine, ubuntu, ...) work >>>>>> as expected. >>>>>> >>>>> >>>>> It's a nice idea but potentially a costly one. I don't think there is a >>>>> guarantee that the downstream package is migration-compatible with the >>>>> upstream package. >>>>> >>>> >>>> What would be costly? It seems it would just be a download + savevm >>>> through monitor, and a loadvm from current version tested. >>>> >>> >>> Setting it up would be simple. Working with it, not so much. It's a test >>> that once it fails, then it becomes a tedious job of figuring out >>> whether the issue is a failure on our side or if it's merely an >>> incompatibility between upstream vs. downstream. The packaged QEMU can >>> have all sorts of tweaks, extra patches, fewer patches, different code >>> entirely, different machine type ABI, etc (their own QMP commands >>> even). The distro can guarantee it all works inside their environment. >>> >>> When a problem happens on the intersection between the two binaries, if >>> it's on our side, we'd need to determine whether to fix it or tell the >>> distro "don't do that". If it's on their side then we open a bug report >>> and hope that this is even a priority for them. >>> >>> So overall I'm in favor, but I don't expect it to be game changing. >>> >> >> Do we have a concrete example in recent migration history of distro >> doing a custom downstream patch that broke migration? >> > > No because no one tries to migrate from the distro packaged QEMU to the > upstream QEMU (i.e. apt install qemu vs. git clone qemu). > > The downstream package contains whatever upstream version the downstream > maintainer wanted to include (e.g. a stable branch) with their own > customizations added on top. These are experienced people that make > sensible choices, so things are not wild, but there's still a solid > layer of separation. > > The (mostly enterprise) distros will also potentially have more than one > stream, e.g. qemu-11.0.0 and qemu-11.0.0-A, where this "A" is something > that fits their own policies (e.g. a version that only receives updates > for CVE fixes; or a version with backports that only ships as part of a > particular product). > > The downstream companies might also have their own special sauce to deal > with migration. Red Hat for instance defines machine types that don't > even exist upstream. > > The general point here is that it's already a lot of work for the > downstream, it might not be a good use of the community time to have us > also taking a look at all those details. And of course in some cases > there's a conflict of interest. >
Right, I see your point. We indeed can't do this job for the distros. >> This would be series author's job/time, not maintainer's. >> > > It should be, yes. More complex problems or ones that for some reason > weren't detected right away will still take other people's time. > >> Also, I think it's precisely why it's interesting to investigate those. >> What is the point to maintain any kind of compatibility, if it's not for >> our main users? >> Preserving compatibility for versions built manually from source is a >> niche use case. >> > > You're thinking in the correct direction, however migration specifically > is very dependent on hardware, both emulated and real. The right people > to test compatibility between distro version X vs. distro version X+1 is > the downstream QA team. Companies have access to hardware that closely > matches their customers', usually outside of the range of what the > community can get their hands on. > > We could maybe attempt a joint effort where upstream provides the CI and > tests while the downstreams add their own hardware and test jobs. Take > git HEAD, package it into a .rpm/.deb, etc and run tests on it. > > Note that there's: > > A) the migration/ subsystem, which performs the migration and provides > the APIs and core code. > > This is where we test features and compatibility mechanisms. > I.e. qtest/migration-test, the migration-compat job in CI and what I > test in every PR. This is well covered. We don't add new features or > major fixes without tests. > > B) the vmstate code for a device, IOW, the thing being migrated. > > This is where compatibility issues are more likely to occur (think > developer adds new emulation feature and forgets to update vmstate > code). > > There are some tests for this under qemu-iotests, there's static > checking using -dump-vmstate, some parts in functional tests, some in > qtests/qom-test. > > This is pooly covered for a bunch of reasons: > > One needs to think carefully about the device, craft tests that > include it, configure it and do something that is prone to be affected > by migration. Some bugs will only trigger at the OS layer, so guest > kernel and drivers come into play. > > There's a spec for the hardware, there's no spec for what happens when > the device state and OS magically change because it's actually QEMU > running v8.2.0 and migration took 1s more to converge than usual. So > writing the device and tests for it tends to be quicker than writing a > good migration test; most people don't bother. > > We also like to keep backwards migration compatibility because distros > usually provide that. It's a completely different test and sometimes > hard to justify when the device writer doesn't particularly care about > the cloud computing scenarios where migration is used. > Thanks for the insight, I understand better why migration is poorly covered in our test suite. Regards, Pierrick
