Re: [Qemu-devel] [PATCH 3/7] configure: integrate Meson in the build system
On Thu, Jun 27, 2019 at 03:16:01PM +0200, Gerd Hoffmann wrote: > Hi, > > > Ok, I can understand that. I've been thinking about how we can switch > > libvirt to use meson too, and trying to decide between meson being the > > owner, calling out to make vs keeping make as the owner and calling > > out to meson. Ultimately to entirely banish make, autoconf, automake, > > libtool, m4 & shell from our build system :-) > > > > Despite thinking about an incremental conversion though, I was still > > hoping libvirt would just have a single (largish) patch series to > > do a complete conversion at a specific point in time. > > Another possible approach would be to have two build systems. > The traditional configure & make and the new meson & ninja. > > Advantage is we don't have to worry about the transition and mixing & > make + meson at all. > > Disadvantage is the duplication. That wouldn't be forever though. > I'd expect we'll have one or maybe two releases with both build systems, > then delete the make & configure. Personally I strongly wish to avoid having two parallel build systems. In several projects I've encountered with that I've ended up hitting bugs where each build system produced different binary results due to the developers not putting the same changes into both build systems. Prefer to only merge a new build system once there is confidence that it is working, and then immediately delete the old one so as to avoid bitrot & concentrate any bug fixing on the modern stuff. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH 3/7] configure: integrate Meson in the build system
On 27/06/19 15:16, Gerd Hoffmann wrote: > Hi, > >> Ok, I can understand that. I've been thinking about how we can switch >> libvirt to use meson too, and trying to decide between meson being the >> owner, calling out to make vs keeping make as the owner and calling >> out to meson. Ultimately to entirely banish make, autoconf, automake, >> libtool, m4 & shell from our build system :-) >> >> Despite thinking about an incremental conversion though, I was still >> hoping libvirt would just have a single (largish) patch series to >> do a complete conversion at a specific point in time. > > Another possible approach would be to have two build systems. > The traditional configure & make and the new meson & ninja. > > Advantage is we don't have to worry about the transition and mixing & > make + meson at all. > > Disadvantage is the duplication. That wouldn't be forever though. > I'd expect we'll have one or maybe two releases with both build systems, > then delete the make & configure. Yeah, I have seen other projects do this. On the other hand I'm not sure how you avoid bitrot in one of the two, and having to convert the huge configure script from the beginning seems like a daunting task. Paolo
Re: [Qemu-devel] [PATCH 3/7] configure: integrate Meson in the build system
Hi, > Ok, I can understand that. I've been thinking about how we can switch > libvirt to use meson too, and trying to decide between meson being the > owner, calling out to make vs keeping make as the owner and calling > out to meson. Ultimately to entirely banish make, autoconf, automake, > libtool, m4 & shell from our build system :-) > > Despite thinking about an incremental conversion though, I was still > hoping libvirt would just have a single (largish) patch series to > do a complete conversion at a specific point in time. Another possible approach would be to have two build systems. The traditional configure & make and the new meson & ninja. Advantage is we don't have to worry about the transition and mixing & make + meson at all. Disadvantage is the duplication. That wouldn't be forever though. I'd expect we'll have one or maybe two releases with both build systems, then delete the make & configure. cheers, Gerd
Re: [Qemu-devel] [PATCH 3/7] configure: integrate Meson in the build system
On 27/06/19 14:23, Markus Armbruster wrote: > Paolo Bonzini writes: >> On 27/06/19 11:03, Daniel P. Berrangé wrote: >> There are two parts of this. One is practical and has to do with >> supporting a step-by-step transition. Using ninja2make makes it trivial >> to have make build products that depend on meson build products, and >> this way bottom up is a natural direction to do the conversion, which is >> bottom up. You'd start from libqemuutil.a and code generators (tracing >> + QAPI), then go to the tools and the emulators. > > *If* the conversion is too big a task to permit doing it all at once, > then the step by step strategy you describe makes sense to me. > > The trouble with step by step is running out of steam before the final > step. That would leave us worse off. Even an overly protracted > conversion would be bad. Agreed. But hey, Makefile.objs is 2000 lines of lists of files, it is boring but it cannot take that long to convert. So it's not really about being _possible_ to do it all at once, it's mostly about the manageability of conflicts. > Thus, my standard question on any proposed step-by-step conversion: > commitment to finishing it? I'd be quite happy to take your word for > it. I cannot really make such a commitment now, but perhaps I can (or I and someone else can) once a little more of the conversion is ready. Especially QAPI and trace generators, since those are where most of the magic resides. I can try that since your reaction wasn't of total disgust. :) >> The second is a design decision that simplifying the Make/meson >> integration is *not* a goal. Rather the goals are: 1) making the >> transition easier on developers; 2) avoiding magic in meson.build at all >> costs. More specifically: >> > Your plan confines new magic to "Makefile rules and external scripts". > We'll get actual reduction only if we can retire or at least radically > simplify them at some point. But even before that, we'll get *practical* reduction if it hacking the Makefile rules and external scripts becomes rare enough. See for example kconfig, where # of people hacking minikconf.py is much less than # of people hacking default-configs. > I'm more ambivalent on (1). Yes, making the transition easier for > developers is worth hiding a certain amount of magic out of their way. It's especially worth during the transition. Once Makefile.objs is gone I agree we can afford more radical changes, but let's cross that bridge when we get there. >> I expect testing might also require some hand-holding, because "meson >> test" does not integrate with "make -j" and to keep supporting our "make >> check-*" targets. However, until the make->ninja flag day we could >> generate tap-driver Makefile rules from "meson introspect --tests" >> output. Basically I'm dropping Makefile magic in favor of build rule >> generators are written in high-level languages. > > A PoC for selected tests would be nice. Fair enough. > Ignorant question: could the switch to Meson enable doing less in > configure? It's big and slw. It would be smaller since the DSL is more compact than shell scripts, probably not much faster as the same tasks still have to be done. But perhaps in the future Meson can parallelize them, and then we'd get that for free. Paolo
Re: [Qemu-devel] [PATCH 3/7] configure: integrate Meson in the build system
On Thu, Jun 27, 2019 at 12:16:05PM +0200, Paolo Bonzini wrote: > On 27/06/19 11:03, Daniel P. Berrangé wrote: > > On Mon, Jun 10, 2019 at 01:14:57PM +0200, Paolo Bonzini wrote: > >> The Meson build system is integrated in the existing configure/make steps > >> by invoking Meson from the configure script and converting Meson's > >> build.ninja > >> rules to an included Makefile. > > > > Why did you take the route of converting ninja rules into makefile > > rules, as opposed to just having some stub makefile rules which > > directly invoke ninja where needed ? > > There are two parts of this. One is practical and has to do with > supporting a step-by-step transition. Using ninja2make makes it trivial > to have make build products that depend on meson build products, and > this way bottom up is a natural direction to do the conversion, which is > bottom up. You'd start from libqemuutil.a and code generators (tracing > + QAPI), then go to the tools and the emulators. Ok, I can understand that. I've been thinking about how we can switch libvirt to use meson too, and trying to decide between meson being the owner, calling out to make vs keeping make as the owner and calling out to meson. Ultimately to entirely banish make, autoconf, automake, libtool, m4 & shell from our build system :-) Despite thinking about an incremental conversion though, I was still hoping libvirt would just have a single (largish) patch series to do a complete conversion at a specific point in time. > > Obviously this series is just some initial integration, but eventually > > when we get a complete conversion, I think it will look pretty wierd > > if we're still converting ninja to make. > > I agree; once all the build rules are converted the Makefile could be as > simple as > > all: > include config.mak > include tests/docker/Makefile.include > include tests/vm/Makefile.include > .NOTPARALLEL: > %: > ninja $@ > > subdir-%-softmmu: > ninja qemu-system-$* > > subdir-%-linux-user: > ninja qemu-$* > > check: > $(MESON) test > > check-%: > $(MESON) test --suite $* > > etc. (and likewise the configure script could just translate the command > line options before invoking meson). But for now, since rules are > written half in meson and half in make, ninja2make seems the most > transparent way to integrate the two. Ok, I think it understand the motiviation better now that its obviously just a short term expedient step to convert ninja to make. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH 3/7] configure: integrate Meson in the build system
On Thu, Jun 27, 2019 at 02:23:52PM +0200, Markus Armbruster wrote: > Paolo Bonzini writes: > > But for now, since rules are > > written half in meson and half in make, ninja2make seems the most > > transparent way to integrate the two. > > > >> Part of the downside of our current build system is that although it > >> uses make, the usage is wierd and QEMU specific structure. It would > >> be a shame to pick meson and then again use it is a way that is wierd > >> and QEMU specific. > > > > I agree, this is why it's important to have at least a standard meson.build. > > > > Some knowledge of config-host.mak is needed, because meson.build uses > > declare_dependency() instead of dependency() to link with libraries that > > were already found by configure, but that's it. > > Ignorant question: could the switch to Meson enable doing less in > configure? It's big and slw. At a conceptual level meson is supposed to take over the role of both configure and automake, so you have a single domain specific language covering all aspects of your build system (though sometimes needs some external python helper scripts to cover missing features in meson). IOW, in a proper conversion, I'd expect configure to go away. I can imagine that being one of the last steps in QEMU's conversion though Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH 3/7] configure: integrate Meson in the build system
Paolo Bonzini writes: > On 27/06/19 11:03, Daniel P. Berrangé wrote: >> On Mon, Jun 10, 2019 at 01:14:57PM +0200, Paolo Bonzini wrote: >>> The Meson build system is integrated in the existing configure/make steps >>> by invoking Meson from the configure script and converting Meson's >>> build.ninja >>> rules to an included Makefile. >> >> Why did you take the route of converting ninja rules into makefile >> rules, as opposed to just having some stub makefile rules which >> directly invoke ninja where needed ? > > There are two parts of this. One is practical and has to do with > supporting a step-by-step transition. Using ninja2make makes it trivial > to have make build products that depend on meson build products, and > this way bottom up is a natural direction to do the conversion, which is > bottom up. You'd start from libqemuutil.a and code generators (tracing > + QAPI), then go to the tools and the emulators. *If* the conversion is too big a task to permit doing it all at once, then the step by step strategy you describe makes sense to me. The trouble with step by step is running out of steam before the final step. That would leave us worse off. Even an overly protracted conversion would be bad. Thus, my standard question on any proposed step-by-step conversion: commitment to finishing it? I'd be quite happy to take your word for it. > The second is a design decision that simplifying the Make/meson > integration is *not* a goal. Rather the goals are: 1) making the > transition easier on developers; 2) avoiding magic in meson.build at all > costs. More specifically: > > - it should remain trivial to do things that used to be trivial, and > most "make" invocations should be kept the same at least until > everything is converted and we can perhaps declare a flag day. People > are used to "make check" or "make subdir-x86_64-softmmu", those should > continue to work while the transition is in progress. > > - it should be possible to modify meson.build without knowing QEMU > specific details, and that should be _already_ possible now at the > beginning of the transition (to avoid creating technical debt). This > means keeping the magic confined in Makefile rules and external scripts, > while having a pretty plain meson.build. I wholeheartedly agree with (2). Reducing magic is the whole point. Your plan confines new magic to "Makefile rules and external scripts". We'll get actual reduction only if we can retire or at least radically simplify them at some point. I'm more ambivalent on (1). Yes, making the transition easier for developers is worth hiding a certain amount of magic out of their way. But not any amount. We should try to preserve the targets printed by "make help". > I expect testing might also require some hand-holding, because "meson > test" does not integrate with "make -j" and to keep supporting our "make > check-*" targets. However, until the make->ninja flag day we could > generate tap-driver Makefile rules from "meson introspect --tests" > output. Basically I'm dropping Makefile magic in favor of build rule > generators are written in high-level languages. A PoC for selected tests would be nice. >> Obviously this series is just some initial integration, but eventually >> when we get a complete conversion, I think it will look pretty wierd >> if we're still converting ninja to make. > > I agree; once all the build rules are converted the Makefile could be as > simple as > > all: > include config.mak > include tests/docker/Makefile.include > include tests/vm/Makefile.include > .NOTPARALLEL: > %: > ninja $@ > > subdir-%-softmmu: > ninja qemu-system-$* > > subdir-%-linux-user: > ninja qemu-$* > > check: > $(MESON) test > > check-%: > $(MESON) test --suite $* > > etc. (and likewise the configure script could just translate the command > line options before invoking meson). I wouldn't bother keeping around such an empty shell of a Makefile. Keeping configure as a wrapper could be marginally more useful, mostly for people trying to build QEMU for the first time. > But for now, since rules are > written half in meson and half in make, ninja2make seems the most > transparent way to integrate the two. > >> Part of the downside of our current build system is that although it >> uses make, the usage is wierd and QEMU specific structure. It would >> be a shame to pick meson and then again use it is a way that is wierd >> and QEMU specific. > > I agree, this is why it's important to have at least a standard meson.build. > > Some knowledge of config-host.mak is needed, because meson.build uses > declare_dependency() instead of dependency() to link with libraries that > were already found by configure, but that's it. Ignorant question: could the switch to Meson enable doing less in configure? It's big and sl
Re: [Qemu-devel] [PATCH 3/7] configure: integrate Meson in the build system
On 27/06/19 11:03, Daniel P. Berrangé wrote: > On Mon, Jun 10, 2019 at 01:14:57PM +0200, Paolo Bonzini wrote: >> The Meson build system is integrated in the existing configure/make steps >> by invoking Meson from the configure script and converting Meson's >> build.ninja >> rules to an included Makefile. > > Why did you take the route of converting ninja rules into makefile > rules, as opposed to just having some stub makefile rules which > directly invoke ninja where needed ? There are two parts of this. One is practical and has to do with supporting a step-by-step transition. Using ninja2make makes it trivial to have make build products that depend on meson build products, and this way bottom up is a natural direction to do the conversion, which is bottom up. You'd start from libqemuutil.a and code generators (tracing + QAPI), then go to the tools and the emulators. The second is a design decision that simplifying the Make/meson integration is *not* a goal. Rather the goals are: 1) making the transition easier on developers; 2) avoiding magic in meson.build at all costs. More specifically: - it should remain trivial to do things that used to be trivial, and most "make" invocations should be kept the same at least until everything is converted and we can perhaps declare a flag day. People are used to "make check" or "make subdir-x86_64-softmmu", those should continue to work while the transition is in progress. - it should be possible to modify meson.build without knowing QEMU specific details, and that should be _already_ possible now at the beginning of the transition (to avoid creating technical debt). This means keeping the magic confined in Makefile rules and external scripts, while having a pretty plain meson.build. I expect testing might also require some hand-holding, because "meson test" does not integrate with "make -j" and to keep supporting our "make check-*" targets. However, until the make->ninja flag day we could generate tap-driver Makefile rules from "meson introspect --tests" output. Basically I'm dropping Makefile magic in favor of build rule generators are written in high-level languages. > Obviously this series is just some initial integration, but eventually > when we get a complete conversion, I think it will look pretty wierd > if we're still converting ninja to make. I agree; once all the build rules are converted the Makefile could be as simple as all: include config.mak include tests/docker/Makefile.include include tests/vm/Makefile.include .NOTPARALLEL: %: ninja $@ subdir-%-softmmu: ninja qemu-system-$* subdir-%-linux-user: ninja qemu-$* check: $(MESON) test check-%: $(MESON) test --suite $* etc. (and likewise the configure script could just translate the command line options before invoking meson). But for now, since rules are written half in meson and half in make, ninja2make seems the most transparent way to integrate the two. > Part of the downside of our current build system is that although it > uses make, the usage is wierd and QEMU specific structure. It would > be a shame to pick meson and then again use it is a way that is wierd > and QEMU specific. I agree, this is why it's important to have at least a standard meson.build. Some knowledge of config-host.mak is needed, because meson.build uses declare_dependency() instead of dependency() to link with libraries that were already found by configure, but that's it. Paolo
Re: [Qemu-devel] [PATCH 3/7] configure: integrate Meson in the build system
Markus Armbruster writes:
> Paolo Bonzini writes:
>
>> The Meson build system is integrated in the existing configure/make steps
>> by invoking Meson from the configure script and converting Meson's
>> build.ninja
>> rules to an included Makefile.
>>
>> Signed-off-by: Paolo Bonzini
>> ---
>> Makefile | 9 +
>> configure| 30 ++
>> meson.build | 9 +
>> scripts/ninjatool.py | 964
>> +++
>
> Uff.
>
>> 4 files changed, 1012 insertions(+)
>> create mode 100644 meson.build
>> create mode 100644 scripts/ninjatool.py
>>
>> diff --git a/Makefile b/Makefile
>> index 8e2fc66..b8f802c 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -48,6 +48,15 @@ git-submodule-update:
>> endif
>> endif
>>
>> +export NINJA=./ninjatool
>> +Makefile.ninja: build.ninja ninjatool
>> +./ninjatool -t ninja2make --omit dist uninstall < $< > $@
>> +-include Makefile.ninja
>> +
>> +ninjatool: $(SRC_PATH)/scripts/ninjatool.py
>> +sed -e '1c\' -e '#! $(PYTHON)' $< > $@
>> +chmod +x $@
>
> Why do we need this here, but not for other Python scripts?
>
> We have 39 Python scripts with #!/usr/bin/env python, one with
> #!/usr/bin/env python2, and 12 with #!/usr/bin/python. The Makefiles
> generally use $(PYTHON) SCRIPT ARGS...
>
>> +
>> .git-submodule-status: git-submodule-update config-host.mak
>>
>> # Check that we're not trying to do an out-of-tree build from
>> diff --git a/configure b/configure
>> index 0814a5f..b8c3c58 100755
>> --- a/configure
>> +++ b/configure
>> @@ -493,6 +493,7 @@ docker="no"
>> debug_mutex="no"
>> libpmem=""
>> default_devices="yes"
>> +meson=meson
>>
>> # cross compilers defaults, can be overridden with --cross-cc-ARCH
>> cross_cc_aarch64="aarch64-linux-gnu-gcc"
>> @@ -983,6 +984,8 @@ for opt do
>>;;
>>--python=*) python="$optarg"
>>;;
>> + --meson=*) meson="$optarg"
>> + ;;
>>--gcov=*) gcov_tool="$optarg"
>>;;
>>--smbd=*) smbd="$optarg"
>> @@ -1685,6 +1688,7 @@ Advanced options (experts only):
>>--make=MAKE use specified make [$make]
>>--install=INSTALLuse specified install [$install]
>>--python=PYTHON use specified python [$python]
>> + --meson=PYTHON use specified meson [$meson]
Typo: --meson=MESON
>>--smbd=SMBD use specified smbd [$smbd]
>>--with-git=GIT use specified git [$git]
>>--static enable static build [$static]
>> @@ -1850,6 +1854,11 @@ then
>> error_exit "Python not found. Use --python=/path/to/python"
>> fi
>>
>> +if ! has "$meson"
>> +then
>> +error_exit "Meson not found. Use --meson=/path/to/meson"
>> +fi
>> +
>> # Note that if the Python conditional here evaluates True we will exit
>> # with status 1 which is a shell 'false' value.
>> if ! $python -c 'import sys; sys.exit(sys.version_info < (2,7))'; then
>> @@ -7983,6 +7992,27 @@ echo "# Automatically generated by configure - do not
>> modify" > "$iotests_common
>> echo >> "$iotests_common_env"
>> echo "export PYTHON='$python'" >> "$iotests_common_env"
>>
>> +# bootstrap ninjatool, we need it before Make runs
>> +if ! test -x ninjatool; then
>> + sed -e '1c\' -e "#! $python" ${source_path}/scripts/ninjatool.py >
>> ninjatool
>> + chmod +x ninjatool
>> +fi
>> +rm -rf meson-private meson-info meson-logs
>
> Ignorant question: why do we need configure remove this stuff?
>
>> +NINJA=$PWD/ninjatool $python $meson setup \
>
> This prints
>
> /usr/bin/python3: can't open file 'meson': [Errno 2] No such file or
> directory
>
> for me, then goes on happily.
>
> For what it's worth:
>
> $ type meson
> meson is /usr/bin/meson
>
> Are you sure you want to override /usr/bin/meson's #! line?
>
> If I drop $python, I get
>
> meson.build:1:0: ERROR: Meson version is 0.50.1 but project requires
> >=0.50.999.
>
> which is expected.
>
> It's too hot right for me now to figure out how to obtain a suitable
> version.
>
>> +--prefix "$prefix" \
>> +--libdir "$libdir" \
>> +--libexecdir "$libexecdir" \
>> +--bindir "$bindir" \
>> +--includedir "$includedir" \
>> +--datadir "$datadir" \
>> +--mandir "$mandir" \
>> +--sysconfdir "$sysconfdir" \
>> +--localstatedir "$local_statedir" \
>> +$(test "$strip_opt" = yes && echo --strip) \
>> +--buildtype $(if test "$debug" = yes; then echo debug; else echo
>> release; fi) \
>> +"$PWD" "$source_path"
>> +
>> +
>> # Save the configure command line for later reuse.
>> cat > #!/bin/sh
>> diff --git a/meson.build b/meson.build
>> new file mode 100644
>> index 000..b683d70
>> --- /dev/null
>> +++ b/meson.build
>> @@ -0,0 +1,9 @@
>> +project('qemu', 'c', meson_version: '>=0.50.999')
>> +
>> +kconfig = import('unstable-kconfig')
>> +config_host = kconfig.load(meson.current_build_dir() / 'config-host.mak')
>> +
>> +add_project_arguments(config_host['QEMU_CFLAGS'].split(),
>> +
Re: [Qemu-devel] [PATCH 3/7] configure: integrate Meson in the build system
On Mon, Jun 10, 2019 at 01:14:57PM +0200, Paolo Bonzini wrote: > The Meson build system is integrated in the existing configure/make steps > by invoking Meson from the configure script and converting Meson's build.ninja > rules to an included Makefile. Why did you take the route of converting ninja rules into makefile rules, as opposed to just having some stub makefile rules which directly invoke ninja where needed ? Obviously this series is just some initial integration, but eventually when we get a complete conversion, I think it will look pretty wierd if we're still converting ninja to make. Part of the downside of our current build system is that although it uses make, the usage is wierd and QEMU specific structure. It would be a shame to pick meson and then again use it is a way that is wierd and QEMU specific. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH 3/7] configure: integrate Meson in the build system
On 26/06/19 19:34, Markus Armbruster wrote:
> Paolo Bonzini writes:
>
>> The Meson build system is integrated in the existing configure/make steps
>> by invoking Meson from the configure script and converting Meson's
>> build.ninja
>> rules to an included Makefile.
>>
>> Signed-off-by: Paolo Bonzini
>> ---
>> Makefile | 9 +
>> configure| 30 ++
>> meson.build | 9 +
>> scripts/ninjatool.py | 964
>> +++
>
> Uff.
What, the 9+30+9 or the 964? :)
>> +export NINJA=./ninjatool
>> +Makefile.ninja: build.ninja ninjatool
>> +./ninjatool -t ninja2make --omit dist uninstall < $< > $@
>> +-include Makefile.ninja
>> +
>> +ninjatool: $(SRC_PATH)/scripts/ninjatool.py
>> +sed -e '1c\' -e '#! $(PYTHON)' $< > $@
>> +chmod +x $@
>
> Why do we need this here, but not for other Python scripts?
>
> We have 39 Python scripts with #!/usr/bin/env python, one with
> #!/usr/bin/env python2, and 12 with #!/usr/bin/python. The Makefiles
> generally use $(PYTHON) SCRIPT ARGS...
Because this script is invoked by meson too. While my patch to support
a NINJA environment variable was accepted (and we point it to
ninjatool), meson invokes it as "$(NINJA)" --foo with no splitting, so
we need the above change.
But I can do better than this. I can generate the file directly from
meson.build:
configure_file(input: files('scripts/ninjatool.py'),
output: 'ninjatool',
configuration: config_host)
and avoid code duplication between configure and Makefile.
>> +rm -rf meson-private meson-info meson-logs
>
> Ignorant question: why do we need configure remove this stuff?
By default, reinvoking meson complains that you probably want to
reconfigure (a la config.status) because there is already a build tree
here. Our configure script always reconfigures the tree from scratch.
So I just cause meson to fail that check.
>> +NINJA=$PWD/ninjatool $python $meson setup \
>
> This prints
>
> /usr/bin/python3: can't open file 'meson': [Errno 2] No such file or
> directory
>
> for me, then goes on happily.
>
> For what it's worth:
>
> $ type meson
> meson is /usr/bin/meson
>
> Are you sure you want to override /usr/bin/meson's #! line?
No I'm not. It should be just $meson, as you do below, though it could
also be $python "$(type -p meson)" if I wanted to override---but
certainly not $python $meson. It worked for me because I was pointing
$meson to an absolute path under my home directory.
There is actually one reason to override it, which is why I did it in
the patch. Meson has an automagic way to invoke all Python programs
with the same interpreter that was used to run meson itself (you make
the script non-executable and add "#! /usr/bin/env python3" as the first
line). If we want to use that feature, we should invoke meson with $python.
> If I drop $python, I get
>
> meson.build:1:0: ERROR: Meson version is 0.50.1 but project requires
> >=0.50.999.
>
> which is expected.
>
> It's too hot right for me now to figure out how to obtain a suitable
> version.
FWIW 0.51.0 has now been released so you could grab it from Koji.
>> diff --git a/scripts/ninjatool.py b/scripts/ninjatool.py
>> new file mode 100644
>> index 000..6d90919
>> --- /dev/null
>> +++ b/scripts/ninjatool.py
> [Lots of code...]
>
> Did you write ninjatool.py specifically for QEMU, or did you steal it
> (or parts) somewhere?
I wrote it specifically for QEMU.
Paolo
Re: [Qemu-devel] [PATCH 3/7] configure: integrate Meson in the build system
Paolo Bonzini writes:
> The Meson build system is integrated in the existing configure/make steps
> by invoking Meson from the configure script and converting Meson's build.ninja
> rules to an included Makefile.
>
> Signed-off-by: Paolo Bonzini
> ---
> Makefile | 9 +
> configure| 30 ++
> meson.build | 9 +
> scripts/ninjatool.py | 964
> +++
Uff.
> 4 files changed, 1012 insertions(+)
> create mode 100644 meson.build
> create mode 100644 scripts/ninjatool.py
>
> diff --git a/Makefile b/Makefile
> index 8e2fc66..b8f802c 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -48,6 +48,15 @@ git-submodule-update:
> endif
> endif
>
> +export NINJA=./ninjatool
> +Makefile.ninja: build.ninja ninjatool
> + ./ninjatool -t ninja2make --omit dist uninstall < $< > $@
> +-include Makefile.ninja
> +
> +ninjatool: $(SRC_PATH)/scripts/ninjatool.py
> + sed -e '1c\' -e '#! $(PYTHON)' $< > $@
> + chmod +x $@
Why do we need this here, but not for other Python scripts?
We have 39 Python scripts with #!/usr/bin/env python, one with
#!/usr/bin/env python2, and 12 with #!/usr/bin/python. The Makefiles
generally use $(PYTHON) SCRIPT ARGS...
> +
> .git-submodule-status: git-submodule-update config-host.mak
>
> # Check that we're not trying to do an out-of-tree build from
> diff --git a/configure b/configure
> index 0814a5f..b8c3c58 100755
> --- a/configure
> +++ b/configure
> @@ -493,6 +493,7 @@ docker="no"
> debug_mutex="no"
> libpmem=""
> default_devices="yes"
> +meson=meson
>
> # cross compilers defaults, can be overridden with --cross-cc-ARCH
> cross_cc_aarch64="aarch64-linux-gnu-gcc"
> @@ -983,6 +984,8 @@ for opt do
>;;
>--python=*) python="$optarg"
>;;
> + --meson=*) meson="$optarg"
> + ;;
>--gcov=*) gcov_tool="$optarg"
>;;
>--smbd=*) smbd="$optarg"
> @@ -1685,6 +1688,7 @@ Advanced options (experts only):
>--make=MAKE use specified make [$make]
>--install=INSTALLuse specified install [$install]
>--python=PYTHON use specified python [$python]
> + --meson=PYTHON use specified meson [$meson]
>--smbd=SMBD use specified smbd [$smbd]
>--with-git=GIT use specified git [$git]
>--static enable static build [$static]
> @@ -1850,6 +1854,11 @@ then
> error_exit "Python not found. Use --python=/path/to/python"
> fi
>
> +if ! has "$meson"
> +then
> +error_exit "Meson not found. Use --meson=/path/to/meson"
> +fi
> +
> # Note that if the Python conditional here evaluates True we will exit
> # with status 1 which is a shell 'false' value.
> if ! $python -c 'import sys; sys.exit(sys.version_info < (2,7))'; then
> @@ -7983,6 +7992,27 @@ echo "# Automatically generated by configure - do not
> modify" > "$iotests_common
> echo >> "$iotests_common_env"
> echo "export PYTHON='$python'" >> "$iotests_common_env"
>
> +# bootstrap ninjatool, we need it before Make runs
> +if ! test -x ninjatool; then
> + sed -e '1c\' -e "#! $python" ${source_path}/scripts/ninjatool.py >
> ninjatool
> + chmod +x ninjatool
> +fi
> +rm -rf meson-private meson-info meson-logs
Ignorant question: why do we need configure remove this stuff?
> +NINJA=$PWD/ninjatool $python $meson setup \
This prints
/usr/bin/python3: can't open file 'meson': [Errno 2] No such file or
directory
for me, then goes on happily.
For what it's worth:
$ type meson
meson is /usr/bin/meson
Are you sure you want to override /usr/bin/meson's #! line?
If I drop $python, I get
meson.build:1:0: ERROR: Meson version is 0.50.1 but project requires
>=0.50.999.
which is expected.
It's too hot right for me now to figure out how to obtain a suitable
version.
> + --prefix "$prefix" \
> + --libdir "$libdir" \
> + --libexecdir "$libexecdir" \
> + --bindir "$bindir" \
> + --includedir "$includedir" \
> + --datadir "$datadir" \
> + --mandir "$mandir" \
> + --sysconfdir "$sysconfdir" \
> + --localstatedir "$local_statedir" \
> + $(test "$strip_opt" = yes && echo --strip) \
> + --buildtype $(if test "$debug" = yes; then echo debug; else echo
> release; fi) \
> + "$PWD" "$source_path"
> +
> +
> # Save the configure command line for later reuse.
> cat #!/bin/sh
> diff --git a/meson.build b/meson.build
> new file mode 100644
> index 000..b683d70
> --- /dev/null
> +++ b/meson.build
> @@ -0,0 +1,9 @@
> +project('qemu', 'c', meson_version: '>=0.50.999')
> +
> +kconfig = import('unstable-kconfig')
> +config_host = kconfig.load(meson.current_build_dir() / 'config-host.mak')
> +
> +add_project_arguments(config_host['QEMU_CFLAGS'].split(),
> + language: 'c')
> +add_project_arguments(config_host['QEMU_INCLUDES'].split(),
> + language: 'c')
> diff --git a/scripts/ninjatool.py b/scripts/ninjatool.py
> new file mode 100644
>
[Qemu-devel] [PATCH 3/7] configure: integrate Meson in the build system
The Meson build system is integrated in the existing configure/make steps
by invoking Meson from the configure script and converting Meson's build.ninja
rules to an included Makefile.
Signed-off-by: Paolo Bonzini
---
Makefile | 9 +
configure| 30 ++
meson.build | 9 +
scripts/ninjatool.py | 964 +++
4 files changed, 1012 insertions(+)
create mode 100644 meson.build
create mode 100644 scripts/ninjatool.py
diff --git a/Makefile b/Makefile
index 8e2fc66..b8f802c 100644
--- a/Makefile
+++ b/Makefile
@@ -48,6 +48,15 @@ git-submodule-update:
endif
endif
+export NINJA=./ninjatool
+Makefile.ninja: build.ninja ninjatool
+ ./ninjatool -t ninja2make --omit dist uninstall < $< > $@
+-include Makefile.ninja
+
+ninjatool: $(SRC_PATH)/scripts/ninjatool.py
+ sed -e '1c\' -e '#! $(PYTHON)' $< > $@
+ chmod +x $@
+
.git-submodule-status: git-submodule-update config-host.mak
# Check that we're not trying to do an out-of-tree build from
diff --git a/configure b/configure
index 0814a5f..b8c3c58 100755
--- a/configure
+++ b/configure
@@ -493,6 +493,7 @@ docker="no"
debug_mutex="no"
libpmem=""
default_devices="yes"
+meson=meson
# cross compilers defaults, can be overridden with --cross-cc-ARCH
cross_cc_aarch64="aarch64-linux-gnu-gcc"
@@ -983,6 +984,8 @@ for opt do
;;
--python=*) python="$optarg"
;;
+ --meson=*) meson="$optarg"
+ ;;
--gcov=*) gcov_tool="$optarg"
;;
--smbd=*) smbd="$optarg"
@@ -1685,6 +1688,7 @@ Advanced options (experts only):
--make=MAKE use specified make [$make]
--install=INSTALLuse specified install [$install]
--python=PYTHON use specified python [$python]
+ --meson=PYTHON use specified meson [$meson]
--smbd=SMBD use specified smbd [$smbd]
--with-git=GIT use specified git [$git]
--static enable static build [$static]
@@ -1850,6 +1854,11 @@ then
error_exit "Python not found. Use --python=/path/to/python"
fi
+if ! has "$meson"
+then
+error_exit "Meson not found. Use --meson=/path/to/meson"
+fi
+
# Note that if the Python conditional here evaluates True we will exit
# with status 1 which is a shell 'false' value.
if ! $python -c 'import sys; sys.exit(sys.version_info < (2,7))'; then
@@ -7983,6 +7992,27 @@ echo "# Automatically generated by configure - do not
modify" > "$iotests_common
echo >> "$iotests_common_env"
echo "export PYTHON='$python'" >> "$iotests_common_env"
+# bootstrap ninjatool, we need it before Make runs
+if ! test -x ninjatool; then
+ sed -e '1c\' -e "#! $python" ${source_path}/scripts/ninjatool.py > ninjatool
+ chmod +x ninjatool
+fi
+rm -rf meson-private meson-info meson-logs
+NINJA=$PWD/ninjatool $python $meson setup \
+ --prefix "$prefix" \
+ --libdir "$libdir" \
+ --libexecdir "$libexecdir" \
+ --bindir "$bindir" \
+ --includedir "$includedir" \
+ --datadir "$datadir" \
+ --mandir "$mandir" \
+ --sysconfdir "$sysconfdir" \
+ --localstatedir "$local_statedir" \
+ $(test "$strip_opt" = yes && echo --strip) \
+ --buildtype $(if test "$debug" = yes; then echo debug; else echo
release; fi) \
+ "$PWD" "$source_path"
+
+
# Save the configure command line for later reuse.
cat =0.50.999')
+
+kconfig = import('unstable-kconfig')
+config_host = kconfig.load(meson.current_build_dir() / 'config-host.mak')
+
+add_project_arguments(config_host['QEMU_CFLAGS'].split(),
+ language: 'c')
+add_project_arguments(config_host['QEMU_INCLUDES'].split(),
+ language: 'c')
diff --git a/scripts/ninjatool.py b/scripts/ninjatool.py
new file mode 100644
index 000..6d90919
--- /dev/null
+++ b/scripts/ninjatool.py
@@ -0,0 +1,964 @@
+#! /usr/bin/env python3
+
+"""Python module for parsing and processing .ninja files.
+
+ Author: Paolo Bonzini
+
+ Copyright (C) 2019 Red Hat, Inc."""
+
+from collections import namedtuple, defaultdict
+import sys
+import os
+import re
+import json
+import argparse
+
+
+class InvalidArgumentError(Exception):
+pass
+
+
+# lexer and parser
+
+BUILD = 1
+POOL = 2
+RULE = 3
+DEFAULT = 4
+EQUALS = 5
+COLON = 6
+PIPE = 7
+PIPE2 = 8
+IDENT = 9
+INCLUDE = 10
+INDENT = 11
+EOL = 12
+
+
+class LexerError(Exception):
+pass
+
+
+class ParseError(Exception):
+pass
+
+
+class NinjaParserEvents(object):
+def __init__(self, parser):
+self.parser = parser
+
+def literal_token(self, word, in_path=False):
+word = word.replace('$', '$$')
+if in_path:
+word = word.replace(' ', '$ ').replace(':', '$:')
+return word
+
+def variable_expansion_token(self, varname):
+
