On Mon, Sep 14, 2026 at 08:58:09PM -0300, Fabiano Rosas wrote:
> Hi, I just wanted to share and ask for feedback on some container images
> I've been building as part of my testing of migration pull-requests.
>
> From a Dockerfile from ./tests/docker/dockerfiles/ I'm producing the
> following:
>
> $ docker images
> qemu-build-base:11.0.0 b6efa198e93a 5.8GB
> qemu-rootfs-base:11.0.0 72bcb5418f4d 4.08GB
> qemu:11.0.0 018a3f0f4dcc 2GB
>
> This last image is stripped (for size) of everything but a few libs and
> binaries (qemu, gdb, migration-test, sh) and has an entrypoint of:
>
> CMD ["qemu-system-x86_64"]
>
> The point is that I can wrap it in a script called qemu-system-x86_64
> and run it interchangeably with any other qemu binary during
> migration-test:
>
> QTEST_QEMU_BINARY_SRC=./build-11.0.0/qemu-system-x86_64 \
> QTEST_QEMU_BINARY=./build-master/qemu-system-x86_64 \
> ./tests/qtest/migration-test
>
> (note that migration-test spawns the qemu binaries itself)
>
> I run tests against the previous 3 QEMU releases. At every release I
> build the container for the new release and at every PR I build the
> container for that git hash using the latest tag as a base.
I'm not a container person, e.g. I did test my recent channel fix on
qemu-7.2 by simply building it with build-7.2/.. but after learning it the
idea using a link to kickoff the container is very interesting on its own
already.
>
> The reasons that lead me to this solution are:
>
> 1) Migration compat testing is special, the two migration peers can run
> in different OS versions, so their QEMU builds can be quite different. I
> cannot test using *just* my dev machine, neither using *just* a
> container. I need two hosts. Or two containers running different OSes
> that can both be reachable by the migration-test binary running in
> either of the OSes.
True.. One thing I want to make us on the same page is, this is about
distro, not OS? Say, the kernel is still the same.
So we have three things (1) QEMU (2) distro (3) kernel/OS. Here the extra
coverage is (2) only.
My gut feeling is distro specific thing causing migration failure is
extremely rare; I can't even think of any in my work history. Meanwhile,
it's still not the "real migration", because kernel is shared. Anything
real will need to happen on two real and different hosts, ultimately, to
make it cover all things. It's more likely to me that the kernel
difference may fail a migration than distro difference.
So I still agree having such variance is better, though the hope is what we
currently do e.g. with compat migration test is good enough.
Said that, if part of this RFC is to see if we want to introduce that too
for the compat test, no objections either. Really, it's a matter of if it
will help us find more bugs.
>
> 2) Neither the QEMU build nor the distro repos are reliable enough to
> allow me to just "go back in time" and rebuild at any moment, I need to
> keep the artifacts cached to avoid rebuilding. Otherwise it comes time
> to send a PR and I discover that some python piece has changed or the
> distro messed up the older repository.
This is definitely true. While studying your test approach, I also noticed
that there's a "toolbox" tool that should have done many of what your
script did: "enter" the toolbox will transparently map many things (files
under /dev/, or $HOME) automatically. At least when thinking this as
something I use personally, toolbox seems working. But again I'm not
container person, maybe there's some difference, just FYI.
>
> Any thoughts on this? Could it be useful to anyone? Maybe the stable
> branch has some similar issues?
I'd be surprised if Michael would need to test stable branch migrations!
That's IMHO too much work for him.. if it needs to be more than what CI
covers. After all, migration is only a submodule in stable maint pov.
So, I'm not sure whether some of those should be in upstream repo, let me
know your thoughts. Besides, I'll definitely play more with toolbox and
your scripts some day.
Thanks for sharing!
>
> I have a messy script that builds everything, but here are the important
> pieces:
>
> - Dockerfile.rootfs.base ==> qemu-rootfs-base-11.1.0
> This is just ./tests/docker/dockerfiles/opensuse-leap.docker with a sed
> to remove ncurses-devel (suse repo issue)
>
>
> - Dockerfile.build.base ==> qemu-build-base:11.1.0
> FROM qemu-rootfs-base:11.1.0
>
> # actual source, mounted RO
> RUN mkdir /src
>
> # not the actual build dir, only the artifacts that are needed
> RUN mkdir /build
>
> WORKDIR /build-tmp-11.1.0
>
> RUN --mount=type=cache,target=/build-tmp-11.1.0,rw \
> --mount=type=bind,from=qemu-src-11.1.0,target=/src,ro \
> /src/configure \
> --target-list=x86_64-softmmu,aarch64-softmmu,ppc64-softmmu,s390x-softmmu \
> --disable-plugins --disable-modules --enable-werror --enable-debug \
> --disable-docs --disable-tools && \
> make -j14 && { echo 'build ok'; } || { echo 'build failed'; exit 1; }; \
> cp -rL qemu-bundle /build && \
> cp -rL scripts /build && \
> find . -maxdepth 1 -name "qemu-system-*" -type f -exec cp {} /build \; &&
> \
> find . -maxdepth 1 -name "tests" -type d -exec cp -r {} /build/tests \; ;
>
> RUN --mount=type=cache,target=/build-tmp-11.1.0,rw \
> find /build -maxdepth 1 -name "qemu-system-*" -type f -exec cp -s {} /bin
> \; && \
> find /build/tests/qtest -maxdepth 1 -name "migration-test" -type f -exec
> ln -sfT {} /bin/migration-test \; ;
> ---
>
>
> - Dockerfile ==> qemu:11.1.0
> FROM qemu-build-base:11.1.0 AS runtime
>
> WORKDIR /build
>
> # isolate the library dependencies of qemu, python and gdb
> RUN mkdir /libs64 && \
> cp -L $(ldd qemu-system-x86_64 | awk '{print $3}' | xargs) /libs64 && \
> cp -L $(ldd /usr/bin/gdb | awk '{print $3}' | xargs) /libs64 && \
> cp -L $(ldd /usr/bin/python3 | awk '{print $3}' | xargs) /libs64 && \
> cp -L $(ldd /bin/sh | awk '{print $3}' | xargs) /libs64 && \
> cp -L $(ldd /bin/sh | grep ld-linux | awk '{print $1}' | xargs) /libs64
> && \
> cp -L $(ldd /usr/lib64/$(readlink $(which
> python3))/site-packages/rpm/_rpm*.so | awk '{print $3}' | xargs) /libs64
> # rpm stuff above is because gdb has its own python runtime
>
> RUN mkdir -p /slash/usr/share && \
> cp -r /usr/lib64/python* /libs64 && \
> ln -sfT /lib64 /slash/usr/lib64
>
> RUN mkdir /libs && \
> cp -r /usr/lib/rpm* /libs
>
> # only these binaries in the image to keep it small
> RUN mkdir /bins && \
> cp -L /bin/sh /bins/ && \
> cp -L /bin/ls /bins/ && \
> cp -L /bin/echo /bins/ && \
> cp -L /usr/bin/cat /bins/ && \
> cp -L /usr/bin/gdb /bins/ && \
> cp -L /usr/bin/python3 /bins/ && \
> cp -L /usr/bin/docker /bins/ && \
> cp -L /usr/bin/env /bins/ && \
> cp -rL /usr/share/gdb /slash/usr/share
>
> # symlinks
> RUN cp -d /bin/migration-test /bins/ && \
> cp -d /bin/qemu-system-* /bins/
>
> FROM scratch
> COPY --from=runtime /slash /
> COPY --from=runtime /libs64 /lib64
> COPY --from=runtime /libs /usr/lib
> COPY --from=runtime /bins /bin
> COPY --from=runtime /build /build
> ENV PYTHONPATH="/lib64/python3.13"
> ENV TMPDIR="/tmp/migtmp"
> WORKDIR /build
>
> # -version helps debugging, this is overriden when calling qemu with
> # proper args
> CMD ["qemu-system-x86_64", "-version"]
> ---
>
>
> - The wrappers. One for each of qemu-system-x86_64, sh, migration-test,
> these live in a build-<version> directory.
> ---
> #!/bin/sh
>
> # pass the environment into the container
> echo > "/tmp/migtmp/envfile"
> env | while IFS= read -r line; do
> case "$line" in
> QTEST*|QEMU*|PYTHON*)
> echo "$line" >> "/tmp/migtmp/envfile"
> ;;
> esac
> done
>
> bin=qemu-system-x86_64
> exec docker run --rm -w /build \
> -v /tmp/migtmp:/tmp/migtmp \
> -v /home/farosas/archive:/archive \
> -v /var/run/docker.sock:/var/run/docker.sock \
> --network host \
> --env-file /tmp/migtmp/envfile \
> --cap-add=SYS_PTRACE \
> --security-opt seccomp=unconfined \
> --device /dev/kvm \
> --device /dev/userfaultfd:/dev/userfaultfd \
> -it qemu:11.1.0 $bin $(printf "%s\n" "$@")
> ---
>
> - The snippet of the test script that uses all of this:
>
> cd ./archive/build-11.0.0
>
> # container-relative
> CURRENT_QEMU=/archive/build-latest
>
> if [ $1 = "src" ]; then
> info "forward..."
> run_qtest PYTHON=/bin/python3 TMPDIR=/tmp/migtmp \
> QTEST_QEMU_BINARY=${CURRENT_QEMU}/qemu-system-${ARCH} \
> QTEST_QEMU_BINARY_SRC=./qemu-system-${ARCH} ./tests/qtest/${TEST}
> else
> info "backward..."
> run_qtest PYTHON=/bin/python3 TMPDIR=/tmp/migtmp \
> QTEST_QEMU_BINARY=${CURRENT_QEMU}/qemu-system-${ARCH} \
> QTEST_QEMU_BINARY_DST=./qemu-system-${ARCH} ./tests/qtest/${TEST}
> fi
> ---
>
--
Peter Xu
