Re: minimal "zero conf" build dockerfiles for fedora:latest and alpine:latest
On 1/13/21 9:20 AM, Paolo Bonzini wrote:
> On 12/01/21 23:37, John Snow wrote:
>> - our configure file suggests bzip2 is an optional dependency (It's
>> set to 'auto') but meson will error out if it is not present at
>> configuration time:
>>
>> ../pc-bios/meson.build:5:2: ERROR: Program 'bzip2' not found
>
> Yes, the configure option is for libbzip2, not bzip2.
>
> Perhaps bzip2 could be required only if get_option('install_blobs') is
> true, I don't know.
Yes, OK.
Re: minimal "zero conf" build dockerfiles for fedora:latest and alpine:latest
On 1/14/21 6:10 AM, Andrea Bolognani wrote: On Wed, 2021-01-13 at 13:31 -0500, John Snow wrote: On 1/13/21 5:09 AM, Gerd Hoffmann wrote: I don't like Perl really, but there's a chicken-and-egg problem between detecting Python and using it to print the configure help script. For configure-time tasks, Perl has the advantage that "#! /usr/bin/env perl" just works. Assuming perl is actually installed, the world seems to shift to python. On a minimal fedora install python is present but perl is not ... On the other hand git depends on perl, so it is probably pretty hard to find a developer workstation without perl installed, so maybe that doesn't matter much for the time being. I agree that it doesn't matter much right now, Though I don't always have git installed in containers when I am doing builds. It will become more common to encounter environments that are missing "obvious" dependencies. Note that Fedora has a git-core package that doesn't depend on Perl while still providing more than enough git for something like a CI build job. Good to know. Another point against perl necessarily existing. As a data point, the libvirt project has made it an explicit goal[1] to remove all usage of Perl in favor of Python. We're not quite there yet, but at this point there are only a very tiny handful of Perl scripts remaining in the repository. [1] https://libvirt.org/strategy.html It's a good long term goal, I think. I'm biased, but: (1) I understand Python very well (2) I find perl difficult to work with. I assume that long-term it will be easier to use Python instead of Perl for most of our precompiler scripts. Based also somewhat on everyone's reaction when someone suggests changes to the checkpatch script, which is "Oh, sure, if YOU write it ..." --js
Re: minimal "zero conf" build dockerfiles for fedora:latest and alpine:latest
On Wed, 2021-01-13 at 13:31 -0500, John Snow wrote: > On 1/13/21 5:09 AM, Gerd Hoffmann wrote: > > > I don't like Perl really, but there's a chicken-and-egg problem between > > > detecting Python and using it to print the configure help script. For > > > configure-time tasks, Perl has the advantage that "#! /usr/bin/env perl" > > > just works. > > > > Assuming perl is actually installed, the world seems to shift to python. > > On a minimal fedora install python is present but perl is not ... > > > > On the other hand git depends on perl, so it is probably pretty hard to > > find a developer workstation without perl installed, so maybe that > > doesn't matter much for the time being. > > I agree that it doesn't matter much right now, Though I don't always > have git installed in containers when I am doing builds. It will become > more common to encounter environments that are missing "obvious" > dependencies. Note that Fedora has a git-core package that doesn't depend on Perl while still providing more than enough git for something like a CI build job. As a data point, the libvirt project has made it an explicit goal[1] to remove all usage of Perl in favor of Python. We're not quite there yet, but at this point there are only a very tiny handful of Perl scripts remaining in the repository. [1] https://libvirt.org/strategy.html -- Andrea Bolognani / Red Hat / Virtualization
Re: minimal "zero conf" build dockerfiles for fedora:latest and alpine:latest
On 1/13/21 3:20 AM, Paolo Bonzini wrote:
On 12/01/21 23:37, John Snow wrote:
- our configure file suggests bzip2 is an optional dependency (It's
set to 'auto') but meson will error out if it is not present at
configuration time:
../pc-bios/meson.build:5:2: ERROR: Program 'bzip2' not found
Yes, the configure option is for libbzip2, not bzip2.
Perhaps bzip2 could be required only if get_option('install_blobs') is
true, I don't know.
Oh, right. Library vs tool. Good point.
Yes, we should make the tool required somewhere in the configure jungle.
FROM alpine:latest
ENV PACKAGES \
wget \
xz \
python3 \
ninja \
gcc \
musl-dev \
pkgconfig \
glib-dev \
pixman-dev \
make \
bash \
perl
ENV QEMU_CONFIGURE_OPTS --disable-linux-user
RUN apk add $PACKAGES
RUN wget https://download.qemu.org/qemu-5.2.0.tar.xz && \
tar xvJf qemu-5.2.0.tar.xz
WORKDIR /qemu-5.2.0
RUN ./configure $QEMU_CONFIGURE_OPTS && \
make -j9
This should be added to the CI!
Apparently someone else is working on an Alpine linux patchset, so I
suppose something like this will be included there.
If not, feel free to take it:
Signed-off-by: John Snow
However, yes, I would like to include some build smoke tests for CI,
which just cover "configure && make" with the absolute minimum set of
dependencies possible. We could rebase our more fully fledged tests on
top of them to share the common layer.
Not a priority for me, but I'm keeping these little dinky dockerfiles
around to use as smoke tests against fedora:latest, alpine:latest, etc.
I should likely expand to CentOS stream, debian, and so on to catch
early breakages.
- diffutils is required for the qapi-schema test, which runs at build
time.
This is not required by meson because technically it is not needed
except for "make check".
Is that true? I was just running "make". qapi-schema test does run at
build time to make sure that the schema pre-processing occurred correctly.
Perhaps we could do
-if build_docs
+if build_docs and diff.found()
in tests/qapi-schema/meson.build.
Paolo
I wasn't building docs, I don't think. (No sphinx and no --enable-docs.)
--js
Re: minimal "zero conf" build dockerfiles for fedora:latest and alpine:latest
On 1/13/21 5:09 AM, Gerd Hoffmann wrote: Hi, I don't like Perl really, but there's a chicken-and-egg problem between detecting Python and using it to print the configure help script. For configure-time tasks, Perl has the advantage that "#! /usr/bin/env perl" just works. Assuming perl is actually installed, the world seems to shift to python. On a minimal fedora install python is present but perl is not ... On the other hand git depends on perl, so it is probably pretty hard to find a developer workstation without perl installed, so maybe that doesn't matter much for the time being. take care, Gerd I agree that it doesn't matter much right now, Though I don't always have git installed in containers when I am doing builds. It will become more common to encounter environments that are missing "obvious" dependencies. This was just an "FYI" experiment. :) --js
Re: minimal "zero conf" build dockerfiles for fedora:latest and alpine:latest
On 1/13/21 1:48 AM, Thomas Huth wrote: On 12/01/2021 23.37, John Snow wrote: I wanted to know what the minimal setup required was to replicate the compilation instructions featured on https://www.qemu.org/download/#source [...] > pixman-devel \ pixman is only required for the softmmu and tools targets. If you just build the linux-user targets, you can even get rid of this. Sure; the intent was to figure out what happens if you run the instructions from the website without explicitly disabling anything. What will you get by default? softmmu by default is probably a pretty reasonable thing to want to build, so I'd say pixman will have to stick around in our minimal dependency list for now; though if we were to translate this experiment to the website, it'd be worth noting that pixman is only required for some, but not all, binaries like you say. (If there was some way to build softmmu without pixman by default, that would be pretty interesting, though. I suppose it would involve disabling all spice/vnc/graphics devices entirely? It doesn't seem tremendously important, but pixman does stick out as the lone very particular dependency for a minimal build.) [...] Notes: - our configure file suggests bzip2 is an optional dependency (It's set to 'auto') but meson will error out if it is not present at configuration time: ../pc-bios/meson.build:5:2: ERROR: Program 'bzip2' not found IIRC it's required for compressing the edk2 firmware images, so if you compile without x86 and arm, you don't need it. Maybe it would be good to add a check for this to the configure script, too? It already is erroring out when meson runs, so I think it's probably fine. Just something to document. - diffutils is required for the qapi-schema test, which runs at build time. We should maybe add a check for "diff" to the configure script? It's in tests/qapi-schema/meson.build already, so I think it's sufficient. - early on in the build process, an error "bash: find: command not found" can be seen, but it doesn't seem to cause a failure otherwise. - perl is not declared as a hard pre-requisite during configure time, but the build will error out if it is not present: [254/8314] Generating texture-blit-frag.h with a meson_exe.py custom command FAILED: ui/shader/texture-blit-frag.h /usr/bin/python3 /qemu-5.2.0/meson/meson.py --internal exe --capture ui/shader/texture-blit-frag.h -- /usr/bin/env perl /qemu-5.2.0/scripts/shaderinclude.pl ../ui/shader/texture-blit.frag /usr/bin/env: ‘perl’: No such file or directory shaderinclude.pl seems to be pretty small, maybe it could be rewritten in python? Maybe; Paolo has replied to you as well, but this is just the first perl script that so happens to run. There might be others. Drawing down on perl long-term is probably nice to do, but it's probably not a priority. For now, we have both perl and python as build dependencies. Let's document and move on. - bash has to be installed explicitly. configure/meson do not check for it, but the build will fail if they aren't present. IIRC we were able to compile without bash before the meson conversion, just some parts like the iotests needed the bash (at least that's why we have a check for bash in tests/check-block.sh for example). Where is it failing now? - musl seems to work alright, but does throw a ton of warnings. I didn't actually run any tests, since they require more dependencies. - linux-user binaries can't be compiled because alpine's usage of musl; I didn't look much more closely. There were some related patches on the list recently, look for the "Alpine Linux build fix and CI pipeline" patch series. Glad to see someone else thought it would be worthwhile to support Alpine. There may be interesting use cases for building alpine QEMU containers that are configured for extremely particular use cases, to be deployed as a utility. --js
Re: minimal "zero conf" build dockerfiles for fedora:latest and alpine:latest
On 13/01/21 11:29, Philippe Mathieu-Daudé wrote: On 1/13/21 11:09 AM, Gerd Hoffmann wrote: Hi, I don't like Perl really, but there's a chicken-and-egg problem between detecting Python and using it to print the configure help script. For configure-time tasks, Perl has the advantage that "#! /usr/bin/env perl" just works. Assuming perl is actually installed, the world seems to shift to python. On a minimal fedora install python is present but perl is not ... On the other hand git depends on perl, so it is probably pretty hard to find a developer workstation without perl installed, so maybe that doesn't matter much for the time being. There is also the new configure-parse-buildoptions.pl script Paolo wants to add: https://www.mail-archive.com/qemu-devel@nongnu.org/msg770651.html Yes, that's what I was referring to. shaderinclude.pl could be converted easily to Python, and I do support in general moving from Perl to Python. For configure-parse-buildoptions.pl I viewed Perl as more of a necessary evil. In the case of scripts/tap-driver.pl, the plan is to use "meson test" instead as soon as it becomes featureful enough. I have already switched my private branch to it, the upstream meson work is tracked at https://github.com/mesonbuild/meson/issues/7830. Paolo
Re: minimal "zero conf" build dockerfiles for fedora:latest and alpine:latest
On 1/13/21 11:09 AM, Gerd Hoffmann wrote: > Hi, > >> I don't like Perl really, but there's a chicken-and-egg problem between >> detecting Python and using it to print the configure help script. For >> configure-time tasks, Perl has the advantage that "#! /usr/bin/env perl" >> just works. > > Assuming perl is actually installed, the world seems to shift to python. > On a minimal fedora install python is present but perl is not ... > > On the other hand git depends on perl, so it is probably pretty hard to > find a developer workstation without perl installed, so maybe that > doesn't matter much for the time being. There is also the new configure-parse-buildoptions.pl script Paolo wants to add: https://www.mail-archive.com/qemu-devel@nongnu.org/msg770651.html
Re: minimal "zero conf" build dockerfiles for fedora:latest and alpine:latest
Hi, > I don't like Perl really, but there's a chicken-and-egg problem between > detecting Python and using it to print the configure help script. For > configure-time tasks, Perl has the advantage that "#! /usr/bin/env perl" > just works. Assuming perl is actually installed, the world seems to shift to python. On a minimal fedora install python is present but perl is not ... On the other hand git depends on perl, so it is probably pretty hard to find a developer workstation without perl installed, so maybe that doesn't matter much for the time being. take care, Gerd
Re: minimal "zero conf" build dockerfiles for fedora:latest and alpine:latest
On 13/01/21 07:48, Thomas Huth wrote: [254/8314] Generating texture-blit-frag.h with a meson_exe.py custom command FAILED: ui/shader/texture-blit-frag.h /usr/bin/python3 /qemu-5.2.0/meson/meson.py --internal exe --capture ui/shader/texture-blit-frag.h -- /usr/bin/env perl /qemu-5.2.0/scripts/shaderinclude.pl ../ui/shader/texture-blit.frag /usr/bin/env: ‘perl’: No such file or directory shaderinclude.pl seems to be pretty small, maybe it could be rewritten in python? Probably, but "make check" also requires Perl for the TAP driver. I do have plans for using "meson test" instead, *however* there's also the other idea I've floated of parsing the command line with Perl: https://patchew.org/QEMU/20210107140039.467969-1-pbonz...@redhat.com/20210107140039.467969-9-pbonz...@redhat.com/ (reviews welcome by the way). I don't like Perl really, but there's a chicken-and-egg problem between detecting Python and using it to print the configure help script. For configure-time tasks, Perl has the advantage that "#! /usr/bin/env perl" just works. - bash has to be installed explicitly. configure/meson do not check for it, but the build will fail if they aren't present. IIRC we were able to compile without bash before the meson conversion, just some parts like the iotests needed the bash (at least that's why we have a check for bash in tests/check-block.sh for example). Where is it failing now? It's failing due to SHELL = /usr/bin/env bash -o pipefail introduced in 5.2 (but not related to the Meson conversion) commit 3bf4583580ab705de1beff6222e934239c3a0356 Author: Paolo Bonzini Date: Wed Oct 14 07:35:13 2020 -0400 make: run shell with pipefail Without pipefail, it is possible to miss failures if the recipes include pipes. Reviewed-by: Daniel P. Berrangé Signed-off-by: Paolo Bonzini Paolo
Re: minimal "zero conf" build dockerfiles for fedora:latest and alpine:latest
On 12/01/21 23:37, John Snow wrote:
- our configure file suggests bzip2 is an optional dependency (It's set
to 'auto') but meson will error out if it is not present at
configuration time:
../pc-bios/meson.build:5:2: ERROR: Program 'bzip2' not found
Yes, the configure option is for libbzip2, not bzip2.
Perhaps bzip2 could be required only if get_option('install_blobs') is
true, I don't know.
FROM alpine:latest
ENV PACKAGES \
wget \
xz \
python3 \
ninja \
gcc \
musl-dev \
pkgconfig \
glib-dev \
pixman-dev \
make \
bash \
perl
ENV QEMU_CONFIGURE_OPTS --disable-linux-user
RUN apk add $PACKAGES
RUN wget https://download.qemu.org/qemu-5.2.0.tar.xz && \
tar xvJf qemu-5.2.0.tar.xz
WORKDIR /qemu-5.2.0
RUN ./configure $QEMU_CONFIGURE_OPTS && \
make -j9
This should be added to the CI!
- diffutils is required for the qapi-schema test, which runs at build time.
This is not required by meson because technically it is not needed
except for "make check".
Perhaps we could do
-if build_docs
+if build_docs and diff.found()
in tests/qapi-schema/meson.build.
Paolo
Re: minimal "zero conf" build dockerfiles for fedora:latest and alpine:latest
On 12/01/2021 23.37, John Snow wrote: I wanted to know what the minimal setup required was to replicate the compilation instructions featured on https://www.qemu.org/download/#source [...] > pixman-devel \ pixman is only required for the softmmu and tools targets. If you just build the linux-user targets, you can even get rid of this. [...] Notes: - our configure file suggests bzip2 is an optional dependency (It's set to 'auto') but meson will error out if it is not present at configuration time: ../pc-bios/meson.build:5:2: ERROR: Program 'bzip2' not found IIRC it's required for compressing the edk2 firmware images, so if you compile without x86 and arm, you don't need it. Maybe it would be good to add a check for this to the configure script, too? - diffutils is required for the qapi-schema test, which runs at build time. We should maybe add a check for "diff" to the configure script? - early on in the build process, an error "bash: find: command not found" can be seen, but it doesn't seem to cause a failure otherwise. - perl is not declared as a hard pre-requisite during configure time, but the build will error out if it is not present: [254/8314] Generating texture-blit-frag.h with a meson_exe.py custom command FAILED: ui/shader/texture-blit-frag.h /usr/bin/python3 /qemu-5.2.0/meson/meson.py --internal exe --capture ui/shader/texture-blit-frag.h -- /usr/bin/env perl /qemu-5.2.0/scripts/shaderinclude.pl ../ui/shader/texture-blit.frag /usr/bin/env: ‘perl’: No such file or directory shaderinclude.pl seems to be pretty small, maybe it could be rewritten in python? - bash has to be installed explicitly. configure/meson do not check for it, but the build will fail if they aren't present. IIRC we were able to compile without bash before the meson conversion, just some parts like the iotests needed the bash (at least that's why we have a check for bash in tests/check-block.sh for example). Where is it failing now? - musl seems to work alright, but does throw a ton of warnings. I didn't actually run any tests, since they require more dependencies. - linux-user binaries can't be compiled because alpine's usage of musl; I didn't look much more closely. There were some related patches on the list recently, look for the "Alpine Linux build fix and CI pipeline" patch series. Thomas
minimal "zero conf" build dockerfiles for fedora:latest and alpine:latest
I wanted to know what the minimal setup required was to replicate the compilation instructions featured on https://www.qemu.org/download/#source > wget https://download.qemu.org/qemu-5.2.0.tar.xz > tar xvJf qemu-5.2.0.tar.xz > cd qemu-5.2.0 > ./configure > make For fedora:latest, I found that to be: FROM fedora:latest ENV PACKAGES \ wget \ xz \ ninja-build \ gcc \ glib2-devel \ pixman-devel \ bzip2 \ diffutils \ perl ENV QEMU_CONFIGURE_OPTS "" RUN dnf install -y $PACKAGES && \ rpm -q $PACKAGES | sort > /packages.txt RUN wget https://download.qemu.org/qemu-5.2.0.tar.xz && \ tar xvJf qemu-5.2.0.tar.xz WORKDIR /qemu-5.2.0 RUN ./configure $QEMU_CONFIGURE_OPTS && \ make -j9 Notes: - our configure file suggests bzip2 is an optional dependency (It's set to 'auto') but meson will error out if it is not present at configuration time: ../pc-bios/meson.build:5:2: ERROR: Program 'bzip2' not found - diffutils is required for the qapi-schema test, which runs at build time. - early on in the build process, an error "bash: find: command not found" can be seen, but it doesn't seem to cause a failure otherwise. - perl is not declared as a hard pre-requisite during configure time, but the build will error out if it is not present: [254/8314] Generating texture-blit-frag.h with a meson_exe.py custom command FAILED: ui/shader/texture-blit-frag.h /usr/bin/python3 /qemu-5.2.0/meson/meson.py --internal exe --capture ui/shader/texture-blit-frag.h -- /usr/bin/env perl /qemu-5.2.0/scripts/shaderinclude.pl ../ui/shader/texture-blit.frag /usr/bin/env: ‘perl’: No such file or directory I wanted to try with alpine for the sake of a dependency audit. It isn't quite "zero conf", but I did get it working by disabling linux-user: FROM alpine:latest ENV PACKAGES \ wget \ xz \ python3 \ ninja \ gcc \ musl-dev \ pkgconfig \ glib-dev \ pixman-dev \ make \ bash \ perl ENV QEMU_CONFIGURE_OPTS --disable-linux-user RUN apk add $PACKAGES RUN wget https://download.qemu.org/qemu-5.2.0.tar.xz && \ tar xvJf qemu-5.2.0.tar.xz WORKDIR /qemu-5.2.0 RUN ./configure $QEMU_CONFIGURE_OPTS && \ make -j9 Notes: - "ninja" actually installs "samurai", but it appears to work. - musl seems to work alright, but does throw a ton of warnings. I didn't actually run any tests, since they require more dependencies. - bash has to be installed explicitly. configure/meson do not check for it, but the build will fail if they aren't present. - linux-user binaries can't be compiled because alpine's usage of musl; I didn't look much more closely. Takeaways: - You really don't need a lot to build a minimal QEMU. Even the alpine package list is pretty small. - meson seems to be handling "absolutely everything is missing" environments pretty well. - There are a scant handful of dependencies that could be added to configure, but you are very likely not to be missing them, so it's low priority. --js
