Daniel P. Berrangé <berra...@redhat.com> writes:
> On Mon, Jul 05, 2021 at 02:44:34PM +0100, Alex Bennée wrote: >> >> Daniel P. Berrangé <berra...@redhat.com> writes: >> >> > This introduces >> > >> > https://gitlab.com/libvirt/libvirt-ci >> > >> > as a git submodule at tests/docker/libvirt-ci >> > >> > This submodule only needs to be checked out when needing to re-generate >> > the files in tests/docker/dockerfiles. >> > >> > When a new build pre-requisite is needed for QEMU, it should be added to >> > the libvirt-ci project 'qemu.yml' file, and the submodule updated to the >> > new commit. >> >> It seems a bit weird to have the canonical description of QEMU >> dependencies live in another project does it not? > > Yes, this is something I've been struggling with, since there > are varying use cases here. > > The "lcitool" command was originally written to automate the > provisioning of virtual machines, suitable to act as runners > for a CI tool. > > The VMs would be suitable for building a variety of projects, > so would need to be installed with the dependancies of all > projects. It makes sense to have the list of dependancies > in one central place. If you have them kept in each respective > project's git repo, then you have to checkout 20 git repos to > get their dependancies, before you can provision the VM. > > It still supports VM provisioning, but now also supports > the Dockerfile generation too in parallel. With the > dockerfiles, you still have a need to access the dependancy > information from outside the main project. For example, > when building libvirt-perl.git, it wants to know the > dependancies needed by libvirt.git, so that it can do > a chained build of the two. > > Potentially libvirt would also want to build qemu.git > first, so it can then test libvirt with latest QEMU. > > So these things all end up driving towards the idea of > storing the build dependancies separate from the project. > > It is definitely sub-optimal though, in that it introduces > a synchronization problem between the 2 respective git > repos for changes. > > For libvirt we've mostly just accepted that pain of needing > to merge stuff in lock-step, but I think it is worse when > dealing with QEMU becasue the subsystem maintainer model > means stuff merges in 2 phases, so there's not a ideal > synchronization point. > >> > The 'make docker-refresh' target will then re-create all the >> > dockerfiles with updated package lists. This ensures that all the >> > containers get exactly the same build pre-requisite packages installed. >> > >> > It also facilitates the addition of containers targetting new distros >> > or updating existing containers to new versions of the same distro, >> > where packages might have been renamed. >> > >> > Signed-off-by: Daniel P. Berrangé <berra...@redhat.com> >> > --- >> > docs/devel/testing.rst | 34 ++++++++++++++++-- >> > tests/docker/Makefile.include | 12 +++++++ >> > tests/docker/dockerfiles-refresh.py | 56 +++++++++++++++++++++++++++++ >> > 3 files changed, 100 insertions(+), 2 deletions(-) >> > create mode 100755 tests/docker/dockerfiles-refresh.py >> > >> > diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst >> > index 4e42392810..7882db85d4 100644 >> > --- a/docs/devel/testing.rst >> > +++ b/docs/devel/testing.rst >> > @@ -372,8 +372,38 @@ Along with many other images, the ``centos8`` image >> > is defined in a Dockerfile >> > in ``tests/docker/dockerfiles/``, called ``centos8.docker``. ``make >> > docker-help`` >> > command will list all the available images. >> > >> > -To add a new image, simply create a new ``.docker`` file under the >> > -``tests/docker/dockerfiles/`` directory. >> > +Most of the existing Dockerfiles were written by hand, simply by creating >> > a >> > +a new ``.docker`` file under the ``tests/docker/dockerfiles/`` directory. >> > +This has led to an inconsistent set of packages being present across the >> > +different containers. >> > + >> > +Thus going forward, QEMU is aiming to automatically generate the >> > Dockerfiles >> > +using the ``lcitool`` program provided by the ``libvirt-ci`` project: >> > + >> > + https://gitlab.com/libvirt/libvirt-ci >> > + >> > +In that project, there is a ``qemu.yml`` file defining the list of build >> > +pre-requisites needed by QEMU. This is processed together with the >> > +``mappings.yml`` file to compute the distro specific list of package >> > names. >> > +The package names are then fed into a generator which emits a well >> > structured >> > +dockerfile. The set of dockerfiles which are auto-generated is defined in >> > +the ``tests/docker/dockerfiles-refresh.py`` script. >> > + >> > +When preparing a patch series that changes dockerfiles managed by >> > ``libvirt-ci`` >> > +tools, the following steps should be takenL >> > + >> > + * Fork the ``libvirt-ci`` project on gitlab >> > + >> > + * Prepare changes to its ``qemu.yml`` file and optionally >> > ``mappings.yml`` >> > + to define the packages to be added to QEMU's dockerfiles. >> > + >> > + * In QEMU run ``make docker-refresh LCITOOL=/path/to/libvirt-ci/lcitool`` >> > + to re-create the dockerfiles in ``tests/docker/dockerfiles`` >> >> If lcitool could be a pre-requisite (even as a developer only one) >> should we consider having a submodule and QEMU mirror of it? > > I did have a submodule in the previous posting, but that creates its > own pain, because there's a chicken and egg problem to updates. Stuff > won't want to be merged into libvirt-ci.git until it is accepted by > a QEMU maintainer, but we need the submodule update ready before > it can be accepted by the QEMU maintainer. There's no nice way to > break that cycle without introducing a bit of manual work and > synchoronization at time of merge to master, which is not desirable > for QEMU IMHO Can't lcitool be improved to accept out-of-its-tree metadata? >> > + * Submit your changes to QEMU in the normal manner >> > + >> > + * Submit ``libvirt-ci`` changes as a merge request, linking to the >> > + QEMU patch series that uses them. >> >> This just seems clunky and likely to therefor not get used. I would >> prefer keeping the meta-data within the project, maybe with a check that >> ensures the dockerfiles have not gone out of sync with their "idealised" >> form. > > > Regards, > Daniel -- Alex Bennée