Peter Xu <[email protected]> writes: > On Mon, Sep 14, 2026 at 08:58:09PM -0300, Fabiano Rosas wrote: >> Hi, I just wanted to share and ask for feedback on some container images >> I've been building as part of my testing of migration pull-requests. >> >> From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the >> following: >> >> $ docker images >> qemu-build-base:11.0.0 b6efa198e93a 5.8GB >> qemu-rootfs-base:11.0.0 72bcb5418f4d 4.08GB >> qemu:11.0.0 018a3f0f4dcc 2GB >> >> This last image is stripped (for size) of everything but a few libs and >> binaries (qemu, gdb, migration-test, sh) and has an entrypoint of: >> >> CMD ["qemu-system-x86_64"] >> >> The point is that I can wrap it in a script called qemu-system-x86_64 >> and run it interchangeably with any other qemu binary during >> migration-test: >> >> QTEST_QEMU_BINARY_SRC=./build-11.0.0/qemu-system-x86_64 \ >> QTEST_QEMU_BINARY=./build-master/qemu-system-x86_64 \ >> ./tests/qtest/migration-test >> >> (note that migration-test spawns the qemu binaries itself) >> >> I run tests against the previous 3 QEMU releases. At every release I >> build the container for the new release and at every PR I build the >> container for that git hash using the latest tag as a base. > > I'm not a container person,
Me neither, lol. Last container work I did was back when multiarch manifests were being added to docker. Maybe 2016? I (clearly) haven't kept up with the progress in the area. > e.g. I did test my recent channel fix on > qemu-7.2 by simply building it with build-7.2/.. but after learning it the > idea using a link to kickoff the container is very interesting on its own > already. > Yeah, the issue is that `git checkout <old version>` and try to build it is pretty hit or miss. >> >> The reasons that lead me to this solution are: >> >> 1) Migration compat testing is special, the two migration peers can run >> in different OS versions, so their QEMU builds can be quite different. I >> cannot test using *just* my dev machine, neither using *just* a >> container. I need two hosts. Or two containers running different OSes >> that can both be reachable by the migration-test binary running in >> either of the OSes. > > True.. One thing I want to make us on the same page is, this is about > distro, not OS? Say, the kernel is still the same. > Yes, it's about what it takes to get a working QEMU binary. I'm not trying to cover a realistic migration scenario, I'm simply trying to test across QEMU versions *somehow*. > So we have three things (1) QEMU (2) distro (3) kernel/OS. Here the extra > coverage is (2) only. > It's not really about extra coverage, it's about (what I think should be) the minimum migration testing done by upstream, i.e. ensuring one binary we release is migration-compatible with another binary we release. Today we test n vs. n-1 in the CI and I think that's not enough. > My gut feeling is distro specific thing causing migration failure is > extremely rare; I can't even think of any in my work history. Meanwhile, > it's still not the "real migration", because kernel is shared. Anything > real will need to happen on two real and different hosts, ultimately, to > make it cover all things. It's more likely to me that the kernel > difference may fail a migration than distro difference. > You right, but that feels like another problem for another day. It starts to get away from what the QEMU community can do. I've seen a lot of cpu-specific issues in the field actually, so even swapping kernels would miss a few bugs. > So I still agree having such variance is better, though the hope is what we > currently do e.g. with compat migration test is good enough. > > Said that, if part of this RFC is to see if we want to introduce that too > for the compat test, no objections either. Really, it's a matter of if it > will help us find more bugs. > Right, I'm not sure yet. Just sharing because I feel like keeping this stuff private is a bad approach. In general, if a maintainer relies on something to do their work, it should be under the community umbrella. Of course there are matters of _workflow_, and every person has a preference, but maybe the tooling could be shared. >> >> 2) Neither the QEMU build nor the distro repos are reliable enough to >> allow me to just "go back in time" and rebuild at any moment, I need to >> keep the artifacts cached to avoid rebuilding. Otherwise it comes time >> to send a PR and I discover that some python piece has changed or the >> distro messed up the older repository. > > This is definitely true. While studying your test approach, I also noticed > that there's a "toolbox" tool that should have done many of what your > script did: "enter" the toolbox will transparently map many things (files > under /dev/, or $HOME) automatically. At least when thinking this as > something I use personally, toolbox seems working. But again I'm not > container person, maybe there's some difference, just FYI. > I had no idea this existed, I'll take a look, thanks! >> >> Any thoughts on this? Could it be useful to anyone? Maybe the stable >> branch has some similar issues? > > I'd be surprised if Michael would need to test stable branch migrations! > That's IMHO too much work for him.. if it needs to be more than what CI > covers. After all, migration is only a submodule in stable maint pov. > Oh, not migration, but the idea of building an older codebase. I know he's dealt with distro repos being broken as well. > > That's a good approach. > > If you want to go further, you could directly host it on dockerhub, for > debian or fedora hosts, so other don't need to rebuild anything. > And everytime there is a new release, we would just add the new image. > It would save storage as well. I'm currently keeping a 7.2 build just because Peter found a bug in it. > $ ./scripts/qemu-time-machine.sh qemu-system-aarch64 11.1.2 "$@" > podman run -it --rm \ > -w $(pwd) -v $(pwd):$(pwd) \ > docker.io/qemu-releases/qemu-system-aarch64:11.1.2 \ > /usr/bin/qemu-system-aarch64 "$@" > > You can drastically optimize size by using multi-stage builds. > All images can be based on a common deps image, which contain all > libraries qemu depend on (~140MB from what I can see on my debian > system), and then all qemu-* images would be FROM:deps, and they only > add a single binary inside. > Ah true, so I probably don't need the full distro build. And keeping the build around is also kinda pointless if it can just be rebuilt. I haven't yet encountered a scenario where a rebuild was necessary actually. > Final result would be 140MB for base deps + binary size per image on > top, (from 3MB to 40MB) per binary. And we could have the whole > collection of qemu binaries available for everyone with a single script. > Good to know, I haven't put too much effort in reducing the size of the images. Now I'm thinking that if they get quite small, we could also host a image just with the (non-unit) tests and maybe use it for some bisecting once a test fails. > Regards, > Pierrick
