Re: [Qemu-devel] converting build system to Meson?

[Paolo Bonzini]

On 18/04/19 10:21, Markus Armbruster wrote:
> Paolo Bonzini  writes:
> 
>> Hi all,
>>
>> lately I have been thinking of converting the QEMU build system to
>> Meson.  Meson is a relatively new build system that can replace
>> Autotools or hand-written Makefiles such as QEMU; as a die-hard
>> Autotools fan, I must say that Meson is by far better than anything else
>> that has ever tried to replace Autotools, and actually has the potential
>> to do so.
>>
>> Advantages of Meson that directly matter for QEMU include:
>>
>> - build definitions in a very readable and user friendly DSL, which
>> supports looping and conditions.
>>
>> - ability to introspect the build definitions so that you can find out
>> what is built without building it (the DSL is not Turing complete and
>> most objects in it are immutable, so it cannot be abused that much :))
>>
>> - support for a non-recursive build from per-subdirectory input (similar
>> to Makefile.objs)
> 
> Could Meson build the $(TARGET_DIRS) non-recursively?

Yes.  It doesn't support recursive builds at all, in fact.

If you can stand the C++, you can find a "testcase slash example with
obvious QEMU inspiration" at
https://github.com/mesonbuild/meson/pull/5028/commits/9495ea241aaca5c2242dac1654ef3168bee1657a.
 In fact, that pull request is basically the last obvious missing piece
before one could seriously try Meson for QEMU.  Then of course there are
the non-obvious missing pieces, but those are more bugfixes than
anything else.

Paolo

> 
> I'm asking because I find the make recursion there annoying.  As usual
> with recursion, we have to dumb down dependencies: if anything made by
> the recursion needs X, then everything does.  If making X fails, we
> don't recurse.  Defeats -k.
> 
>> - ease of distributing a full copy of Meson to support distros that ship
>> an older version (no dependencies apart from Python 3.5).  At 4
>> lines of Python, Meson is relatively small.
> 
> [...]
> 

Re: [Qemu-devel] converting build system to Meson?

[Markus Armbruster]

Paolo Bonzini  writes:

> Hi all,
>
> lately I have been thinking of converting the QEMU build system to
> Meson.  Meson is a relatively new build system that can replace
> Autotools or hand-written Makefiles such as QEMU; as a die-hard
> Autotools fan, I must say that Meson is by far better than anything else
> that has ever tried to replace Autotools, and actually has the potential
> to do so.
>
> Advantages of Meson that directly matter for QEMU include:
>
> - build definitions in a very readable and user friendly DSL, which
> supports looping and conditions.
>
> - ability to introspect the build definitions so that you can find out
> what is built without building it (the DSL is not Turing complete and
> most objects in it are immutable, so it cannot be abused that much :))
>
> - support for a non-recursive build from per-subdirectory input (similar
> to Makefile.objs)

Could Meson build the $(TARGET_DIRS) non-recursively?

I'm asking because I find the make recursion there annoying.  As usual
with recursion, we have to dumb down dependencies: if anything made by
the recursion needs X, then everything does.  If making X fails, we
don't recurse.  Defeats -k.

> - ease of distributing a full copy of Meson to support distros that ship
> an older version (no dependencies apart from Python 3.5).  At 4
> lines of Python, Meson is relatively small.

[...]

Re: [Qemu-devel] converting build system to Meson?

[Paolo Bonzini]

On 10/03/19 15:33, Markus Armbruster wrote:
>>  * is not particularly hard to deal with for the common cases
>>("add new source file" is straightforward)
>>  * covers all our requirements as far as I'm aware
>>(whereas you've listed a couple of places where Meson
>>would need changes/extensions to support things we do already)
>>  * is generally flexible enough to be hackable to deal with odd
>>cases (it has escape mechanisms to generic-programmability,
>>even if they're ugly and awkward)
> 
> Yes, it's hackable, but it takes quite a hacker to hack it.  While it's
> reasonably easy to do simple things in it with basic voodoo skills, the
> learning curve goes up like the Zimbabwean inflation rate after that.  I
> got plenty of experience in Make, and consider myself pretty fluent, yet
> I find myself running to Paolo for help.

The build system should make it trivial to do trivial things; easy to do
things that are a matter of cut-and-paste from something that already
exist; possible to do everything else.

We are good at the first and barely acceptable at the second.  The third
depends on your definition of possible and on the effort you want to put in.

My hope with a switch to Meson would be to keep the first just as
trivial as it is now (commas and quotes do add visual weight but do not
make things any less trivial; they are a nuisance but not a blocker);
make the second easier; make the third hopefully much easier and at
least more predictable.

There is a possibility that sooner or later we'll find out something
else that requires more Meson patches.  Our build system is already
quite mature, so that's at least not too likely, but it's obviously
impossible to rule it out definitively.

Paolo

Re: [Qemu-devel] converting build system to Meson?

[Daniel P . Berrangé]

On Sun, Mar 10, 2019 at 05:28:44PM +0100, Markus Armbruster wrote:
> Paolo Bonzini  writes:
> 
> > Hi all,
> >
> > lately I have been thinking of converting the QEMU build system to
> > Meson.  Meson is a relatively new build system that can replace
> > Autotools or hand-written Makefiles such as QEMU; as a die-hard
> > Autotools fan, I must say that Meson is by far better than anything else
> > that has ever tried to replace Autotools, and actually has the potential
> > to do so.
> >
> > Advantages of Meson that directly matter for QEMU include:
> [...]
> > - ease of distributing a full copy of Meson to support distros that ship
> > an older version (no dependencies apart from Python 3.5).  At 4
> > lines of Python, Meson is relatively small.
> 
> Meson is licensed under the Apache 2 license.  QEMU as a whole is
> licensed under GPLv2 (and no later versions).  From
> :
> 
> This is a free software license, compatible with version 3 of the
> GNU GPL.
> 
> Please note that this license is not compatible with GPL version 2,
> because it has some requirements that are not in that GPL version.
> These include certain patent termination and indemnification
> provisions.  The patent termination provision is a good thing, which
> is why we recommend the Apache 2.0 license for substantial programs
> over other lax permissive licenses.
> 
> Can we distribute Meson along with QEMU anyway?

Yes, it is not a problem (from a legal POV at least). A single source
tar.gz is fine to bundle code under a mixture of licenses, even if those
licenses are not compatible. What matters is what you then do with that
code & how it comes together into combined works.

Meson in this case is just part of the toolchain and none of its code
is integrated into the final QEMU binaries. As such its license does
not have impact on the licence of the QEMU binaries.

We could of course need to update our top level LICENSE file to make it
clear to users/downstream vendors that we've got new license included,
but this isn't much of a surprise as QEMU has many licenses already :-)

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Qemu-devel] converting build system to Meson?

[Thomas Huth]

On 10/03/2019 17.28, Markus Armbruster wrote:
> Paolo Bonzini  writes:
> 
>> Hi all,
>>
>> lately I have been thinking of converting the QEMU build system to
>> Meson.  Meson is a relatively new build system that can replace
>> Autotools or hand-written Makefiles such as QEMU; as a die-hard
>> Autotools fan, I must say that Meson is by far better than anything else
>> that has ever tried to replace Autotools, and actually has the potential
>> to do so.
>>
>> Advantages of Meson that directly matter for QEMU include:
> [...]
>> - ease of distributing a full copy of Meson to support distros that ship
>> an older version (no dependencies apart from Python 3.5).  At 4
>> lines of Python, Meson is relatively small.
> 
> Meson is licensed under the Apache 2 license.  QEMU as a whole is
> licensed under GPLv2 (and no later versions).  From
> :
> 
> This is a free software license, compatible with version 3 of the
> GNU GPL.
> 
> Please note that this license is not compatible with GPL version 2,
> because it has some requirements that are not in that GPL version.
> These include certain patent termination and indemnification
> provisions.  The patent termination provision is a good thing, which
> is why we recommend the Apache 2.0 license for substantial programs
> over other lax permissive licenses.
> 
> Can we distribute Meson along with QEMU anyway?

Sure, as long as we make it clear that it is a separate executable (with
its own license), and do not link the meson code into the QEMU binary.

 Thomas

Re: [Qemu-devel] converting build system to Meson?

[Neal Gompa]

On Thu, Mar 7, 2019 at 1:44 PM Paolo Bonzini  wrote:
>
> On 07/03/19 14:09, Peter Maydell wrote:
> > On Thu, 7 Mar 2019 at 12:56, Paolo Bonzini  wrote:
> >> In any case, this wouldn't change; as you suggest below, configure could
> >> remain as a front-end (well, in-srcdir builds are not supported by
> >> Meson, so "../configure && ninja" perhaps).
> >
> > As an aside, it might be a nice idea to drop the in-srcdir
> > build altogether for QEMU anyway -- it's not really a very
> > good idea and it means our build system has to cope with two
> > different ways of working to no particularly useful end.
>
> I was actually going to propose that, but I was afraid of throwing two
> bombs in the same day. :)
>
> > Of the various hosts I do builds on:
> >  * OSX is 2.7 only
> >  * the ppc64 box in the gcc compile farm is 2.7 and 3.4
> >("Centos 7.5.1804")
> >  * the aarch64 box in the gcc compile farm is 2.7 and 3.4
> >("Ubuntu 14.04.5 LTS", aka trusty)
>
> OS X will need to get Python 3 from Homebrew sooner or later anyway.
>
> Trusty does have a python3.5 package, perhaps you could ask the
> maintainers to install it.
>
> CentOS 7 doesn't have "native" Python 3 (even the 3.4 version you have
> there is probably coming from Fedora) but it has software collections,
> where you have to do "scl enable rh-python35 './configure && make'".
> You can check if scl and the rh-python35 software collections are
> already installed.
>

Most normal users of CentOS use it with EPEL. In this case, EPEL 7 is
transitioning to Python 3.6 (both are available in EPEL 7 today, and
Meson is built there against 3.6).

macOS already has Python 3 in Homebrew, as well as Meson itself.
Eventually Apple will change the native Python runtime to 3.x as well.

Ubuntu 14.04 LTS is EOL next month, and Ubuntu 16.04 LTS ships Python
3.5 and offers a recent Meson in backports.


-- 
真実はいつも一つ！/ Always, there's only one truth!

Re: [Qemu-devel] converting build system to Meson?

[Markus Armbruster]

Cleber Rosa  writes:

> On Thu, Mar 07, 2019 at 01:09:52PM +, Peter Maydell wrote:
>> On Thu, 7 Mar 2019 at 12:56, Paolo Bonzini  wrote:
>> > In any case, this wouldn't change; as you suggest below, configure could
>> > remain as a front-end (well, in-srcdir builds are not supported by
>> > Meson, so "../configure && ninja" perhaps).
>> 
>> As an aside, it might be a nice idea to drop the in-srcdir
>> build altogether for QEMU anyway -- it's not really a very
>> good idea and it means our build system has to cope with two
>> different ways of working to no particularly useful end.
>>
>
> THIS.  Please.

Seconded.

If nothing beyond abandoning in-tree builds should come out of this
thread, it will have been totally worth it for me.

Re: [Qemu-devel] converting build system to Meson?

[Markus Armbruster]

Paolo Bonzini  writes:

> Hi all,
>
> lately I have been thinking of converting the QEMU build system to
> Meson.  Meson is a relatively new build system that can replace
> Autotools or hand-written Makefiles such as QEMU; as a die-hard
> Autotools fan, I must say that Meson is by far better than anything else
> that has ever tried to replace Autotools, and actually has the potential
> to do so.
>
> Advantages of Meson that directly matter for QEMU include:
[...]
> - ease of distributing a full copy of Meson to support distros that ship
> an older version (no dependencies apart from Python 3.5).  At 4
> lines of Python, Meson is relatively small.

Meson is licensed under the Apache 2 license.  QEMU as a whole is
licensed under GPLv2 (and no later versions).  From
:

This is a free software license, compatible with version 3 of the
GNU GPL.

Please note that this license is not compatible with GPL version 2,
because it has some requirements that are not in that GPL version.
These include certain patent termination and indemnification
provisions.  The patent termination provision is a good thing, which
is why we recommend the Apache 2.0 license for substantial programs
over other lax permissive licenses.

Can we distribute Meson along with QEMU anyway?

[...]

Re: [Qemu-devel] converting build system to Meson?

[Markus Armbruster]

Peter Maydell  writes:

> On Thu, 7 Mar 2019 at 06:39, Thomas Huth  wrote:
>> On 06/03/2019 19.12, Paolo Bonzini wrote:
>> > lately I have been thinking of converting the QEMU build system to
>> > Meson.  Meson is a relatively new build system that can replace
>> > Autotools or hand-written Makefiles such as QEMU; as a die-hard
>> > Autotools fan, I must say that Meson is by far better than anything else
>> > that has ever tried to replace Autotools, and actually has the potential
>> > to do so.
>> >
>> > Advantages of Meson that directly matter for QEMU include:[...]
>>
>> I'm not objecting a new build system per se, but could you elaborate on
>>  problems of the current QEMU build system that will be fixed by this
>> change? Since apart from some minor glitches (with the *.mak file
>> dependencies for example), the current build system seems to work quite
>> well for me ... so at least I currently don't feel enough pain yet to do
>> such a big step, just because there is another new cool build system
>> around...
>
> Yes, that tends to be my view. Our current build system:
>  * has no dependencies that are problematic for older hosts
>(contrast Meson, which needs Python 3.5, even if we take
>the drastic step of shipping an entire build tool along
>with QEMU; OSX python is 2.7 still)

By the time Meson is ready for us, and we're ready for Meson, chances
are even OS-X has moved on from Python 2.

https://pythonclock.org/

>  * is not particularly hard to deal with for the common cases
>("add new source file" is straightforward)

Yes.  Quite an achievement.

>  * covers all our requirements as far as I'm aware
>(whereas you've listed a couple of places where Meson
>would need changes/extensions to support things we do already)
>  * is generally flexible enough to be hackable to deal with odd
>cases (it has escape mechanisms to generic-programmability,
>even if they're ugly and awkward)

Yes, it's hackable, but it takes quite a hacker to hack it.  While it's
reasonably easy to do simple things in it with basic voodoo skills, the
learning curve goes up like the Zimbabwean inflation rate after that.  I
got plenty of experience in Make, and consider myself pretty fluent, yet
I find myself running to Paolo for help.

> So I think we'd need a more compelling reason to move right now.
> (This might change in the future, eg if Meson catches on to the
> extent that everybody is using it and competitors like CMake are
> more obviously eclipsed by it, in the way that git took over
> from svn and relegated mercurial and bzr to obscurity.)
>
> thanks
> -- PMM

Re: [Qemu-devel] converting build system to Meson?

[Markus Armbruster]

Daniel P. Berrangé  writes:

[...]
> As long term contributors we've built enough enough knowledge to
> QEMU to consider our build system attractive or even "simple",

The words that come to my mind aren't "attractive" or "simple", but
"impressively clever", "brittle", and "uh, can I hack on something else
today?"

I've obviously failed to develop enough of a Stockholm Syndrome in 8+
years.

> which skews our view of the benefits of alternatives to people
> without this tribal knowledge.

Re: [Qemu-devel] converting build system to Meson?

[Peter Maydell]

On Fri, 8 Mar 2019 at 16:32, Paolo Bonzini  wrote:
> Let's go and do it.  If somebody really wants to do in-tree builds they
> can use --disable-docs.

TBH I'd suggest postponing this to 4.1...

-- PMM

Re: [Qemu-devel] converting build system to Meson?

[Paolo Bonzini]

On 08/03/19 17:17, Eric Blake wrote:
> On 3/8/19 6:03 AM, Peter Maydell wrote:
>> On Fri, 8 Mar 2019 at 11:58, Gerd Hoffmann  wrote:
>>> On Fri, Mar 08, 2019 at 10:31:20AM +, Peter Maydell wrote:
 I think in fact I have just broken in-tree builds with the
 Sphinx support patchset :-/
>>>
>>> Great opportunity!  Stop supporting in-tree build fixes a bug,
>>> so this can be merged post soft-freeze.
>>
>> Joking aside, this is quite tempting, because Sphinx simply
>> does not support building into the same directory as its
>> source documentation files. So we need to either:
>>  * create an ad-hoc build directory (eg docs/built)
>>where Sphinx builds the docs to if we're doing an
>>in-tree build, and have the 'make install' phase pull
>>the docs out of there
> 
> Lots of work to just throw away if we plan on doing a flag day in 4.1.
> But necessary if you want 4.0 to still support in-tree builds for one
> last hurrah.
> 
>>  * declare in-tree builds to be no longer supported at all
> 
> Moves the flag day a bit sooner (today! surprise!), but I can live with
> this.
> 
> How long do we wait for objections, or should we pull the trigger?

Let's go and do it.  If somebody really wants to do in-tree builds they
can use --disable-docs.

The only possible negative effect could be that Coverity might show my
local paths - IIRC that's why I switched Coverity to use an in-tree
build - but we'll find a way around it.

Paolo



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] converting build system to Meson?

[Peter Maydell]

On Fri, 8 Mar 2019 at 16:17, Eric Blake  wrote:
>
> On 3/8/19 6:03 AM, Peter Maydell wrote:
> > On Fri, 8 Mar 2019 at 11:58, Gerd Hoffmann  wrote:
> >> On Fri, Mar 08, 2019 at 10:31:20AM +, Peter Maydell wrote:
> >>> I think in fact I have just broken in-tree builds with the
> >>> Sphinx support patchset :-/
> >>
> >> Great opportunity!  Stop supporting in-tree build fixes a bug,
> >> so this can be merged post soft-freeze.
> >
> > Joking aside, this is quite tempting, because Sphinx simply
> > does not support building into the same directory as its
> > source documentation files. So we need to either:
> >  * create an ad-hoc build directory (eg docs/built)
> >where Sphinx builds the docs to if we're doing an
> >in-tree build, and have the 'make install' phase pull
> >the docs out of there
>
> Lots of work to just throw away if we plan on doing a flag day in 4.1.
> But necessary if you want 4.0 to still support in-tree builds for one
> last hurrah.

It turned out not to be that much work:
https://patchew.org/QEMU/20190308135744.6480-1-peter.mayd...@linaro.org/20190308135744.6480-2-peter.mayd...@linaro.org/

thanks
-- PMM

Re: [Qemu-devel] converting build system to Meson?

[Eric Blake]

On 3/8/19 6:03 AM, Peter Maydell wrote:
> On Fri, 8 Mar 2019 at 11:58, Gerd Hoffmann  wrote:
>> On Fri, Mar 08, 2019 at 10:31:20AM +, Peter Maydell wrote:
>>> I think in fact I have just broken in-tree builds with the
>>> Sphinx support patchset :-/
>>
>> Great opportunity!  Stop supporting in-tree build fixes a bug,
>> so this can be merged post soft-freeze.
> 
> Joking aside, this is quite tempting, because Sphinx simply
> does not support building into the same directory as its
> source documentation files. So we need to either:
>  * create an ad-hoc build directory (eg docs/built)
>where Sphinx builds the docs to if we're doing an
>in-tree build, and have the 'make install' phase pull
>the docs out of there

Lots of work to just throw away if we plan on doing a flag day in 4.1.
But necessary if you want 4.0 to still support in-tree builds for one
last hurrah.

>  * declare in-tree builds to be no longer supported at all

Moves the flag day a bit sooner (today! surprise!), but I can live with
this.

How long do we wait for objections, or should we pull the trigger?

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] converting build system to Meson?

[Daniel P . Berrangé]

On Thu, Mar 07, 2019 at 01:24:07PM -0600, Eric Blake wrote:
> On 3/7/19 1:04 PM, Eric Blake wrote:
> 
> > I don't mind a switch to mandatory out-of-tree builds (I like the
> > convenience of in-tree for quick hacks, but VPATH is infinitely more
> > flexible when testing multiple build configurations in parallel, and
> > I've worked on other projects with mandatory VPATH builds without seeing
> > too many complaints there).
> 
> I will add: The GNU Coding Standards require both in-tree and
> out-of-tree builds to work (and you get that with Autotool'd projects).

You only get it with autotools if you are good at writing Makefile.ams :-)
libvirt frequently breaks VPATH by accident when using autotools by
missing $(srcdir) prefixes on paths.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Qemu-devel] converting build system to Meson?

[Daniel P . Berrangé]

On Thu, Mar 07, 2019 at 06:18:28PM +, Peter Maydell wrote:
> On Thu, 7 Mar 2019 at 18:17, Marc-André Lureau
>  wrote:
> > As someone who has also been annoyed having to support in-tree and
> > out-of-tree, I support that move. I used to be a pretty happy use of
> > in-tree builds, but out-of-tree his generally equally convenient.
> >
> > Who is volunteering to do patches to deprecate the support? warning in
> > 4.0, and error in 4.1?
> 
> Do we need to bother to deprecate it? It doesn't affect end-users,
> only those who build QEMU, and the change is easy...

Yeah, I don't think we need to deprecate it. If we get configure to
report an error straightaway it will be obviuous to developers what
they need to change to adapt.

Sufficient to just announce on the mailing list that we intend todo
it in the start of the next dev cycle. Gives developers plenty of
time to adapt, and it will be in the release notes for the next
release to alert downstream consumers.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Qemu-devel] converting build system to Meson?

[Peter Maydell]

On Fri, 8 Mar 2019 at 11:58, Gerd Hoffmann  wrote:
> On Fri, Mar 08, 2019 at 10:31:20AM +, Peter Maydell wrote:
> > I think in fact I have just broken in-tree builds with the
> > Sphinx support patchset :-/
>
> Great opportunity!  Stop supporting in-tree build fixes a bug,
> so this can be merged post soft-freeze.

Joking aside, this is quite tempting, because Sphinx simply
does not support building into the same directory as its
source documentation files. So we need to either:
 * create an ad-hoc build directory (eg docs/built)
   where Sphinx builds the docs to if we're doing an
   in-tree build, and have the 'make install' phase pull
   the docs out of there
 * declare in-tree builds to be no longer supported at all

thanks
-- PMM

Re: [Qemu-devel] converting build system to Meson?

[Gerd Hoffmann]

On Fri, Mar 08, 2019 at 10:31:20AM +, Peter Maydell wrote:
> On Fri, 8 Mar 2019 at 06:47, Gerd Hoffmann  wrote:
> > I stopped using in-tree builds years ago.  When test-building different
> > configurations using an build/$cfgname out-of-tree build directory is
> > the only sane approach.
> 
> Yes. The other big advantage for me is being able to blow away
> the whole build tree and recreate it if necessary.

Yes.  Easy way to force a full rebuild, which seems to be needed now and
then because the dependency tracking of our build system isn't perfect.

> > And IIRC I've managed to break in-tree builds
> > because of that.
> 
> I think in fact I have just broken in-tree builds with the
> Sphinx support patchset :-/

Great opportunity!  Stop supporting in-tree build fixes a bug,
so this can be merged post soft-freeze.

cheers,
  Gerd

Re: [Qemu-devel] converting build system to Meson?

[Peter Maydell]

On Fri, 8 Mar 2019 at 06:47, Gerd Hoffmann  wrote:
> I stopped using in-tree builds years ago.  When test-building different
> configurations using an build/$cfgname out-of-tree build directory is
> the only sane approach.

Yes. The other big advantage for me is being able to blow away
the whole build tree and recreate it if necessary.

>  And IIRC I've managed to break in-tree builds
> because of that.

I think in fact I have just broken in-tree builds with the
Sphinx support patchset :-/

thanks
-- PMM

Re: [Qemu-devel] converting build system to Meson?

[Thomas Huth]

On 08/03/2019 07.47, Gerd Hoffmann wrote:
>   Hi,
> 
>>> As an aside, it might be a nice idea to drop the in-srcdir
>>> build altogether for QEMU anyway -- it's not really a very
>>> good idea and it means our build system has to cope with two
>>> different ways of working to no particularly useful end.
>>
>> I was actually going to propose that, but I was afraid of throwing two
>> bombs in the same day. :)
> 
> I stopped using in-tree builds years ago.  When test-building different
> configurations using an build/$cfgname out-of-tree build directory is
> the only sane approach.  And IIRC I've managed to break in-tree builds
> because of that.

FWIW, I'm also only doing out-of-tree builds. Thus I'd support a patch,
too, which enforces out of tree builds.

 Thomas

Re: [Qemu-devel] converting build system to Meson?

[Gerd Hoffmann]

  Hi,

> > As an aside, it might be a nice idea to drop the in-srcdir
> > build altogether for QEMU anyway -- it's not really a very
> > good idea and it means our build system has to cope with two
> > different ways of working to no particularly useful end.
> 
> I was actually going to propose that, but I was afraid of throwing two
> bombs in the same day. :)

I stopped using in-tree builds years ago.  When test-building different
configurations using an build/$cfgname out-of-tree build directory is
the only sane approach.  And IIRC I've managed to break in-tree builds
because of that.

So, yes, please.

> CentOS 7 doesn't have "native" Python 3 (even the 3.4 version you have
> there is probably coming from Fedora) but it has software collections,
> where you have to do "scl enable rh-python35 './configure && make'".
> You can check if scl and the rh-python35 software collections are
> already installed.

Also:  epel offers a meson package.  It has python34 and python36
packages too.

cheers,
  Gerd

Re: [Qemu-devel] converting build system to Meson?

[Liviu Ionescu]

> On 7 Mar 2019, at 20:18, Peter Maydell  wrote:
> 
> On Thu, 7 Mar 2019 at 18:17, Marc-André Lureau
>> ... deprecate ... in-srcdir
> 
> Do we need to bother to deprecate it? It doesn't affect end-users,
> only those who build QEMU, and the change is easy...

no problems for the GNU MCU Eclipse QEMU distribution, only out-of-srcdir 
builds used.


regards,

Liviu

Re: [Qemu-devel] converting build system to Meson?

[Eric Blake]

On 3/7/19 1:23 PM, BALATON Zoltan wrote:
> On Thu, 7 Mar 2019, Peter Maydell wrote:
>> On Thu, 7 Mar 2019 at 18:17, Marc-André Lureau
>>  wrote:
>>> As someone who has also been annoyed having to support in-tree and
>>> out-of-tree, I support that move. I used to be a pretty happy use of
>>> in-tree builds, but out-of-tree his generally equally convenient.
>>>
>>> Who is volunteering to do patches to deprecate the support? warning in
>>> 4.0, and error in 4.1?
>>
>> Do we need to bother to deprecate it? It doesn't affect end-users,
>> only those who build QEMU, and the change is easy...
> 
> If you do this without warning and time for getting used to it, it would
> be nice to add some convenience functionality if possible to not upset
> people used to in-tree builds too much. That is if someone runs make in
> the top src dir it should create a build subdir and run make -C in there
> instead of just returning an error so people can still just run make as
> now instead of get frustrated every time until get used to use build
> dir. (I sometimes use build dir, sometimes in-tree build and the latter
> is often more convenient if I don't need multiple separate builds only
> one.)

You might be able to do something like (untested):

echo /GNUMakefile >> .git/info/exclude
cat > GNUMakefile <<\EOF
%: force
@$(MAKE) -C subdir $@
force: ;

so that 'make blah' in-tree becomes a shim to a default out-of-tree
build location. (You'd need to make it smarter than that to handle a
user invoking 'make -C tests ...', left as an exercise for the reader)

But yes, ideally we'd actually have a proper wrapper Makefile that does
this as part of qemu.git, instead of having to be reinvented by each
developer.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] converting build system to Meson?

[BALATON Zoltan]

On Thu, 7 Mar 2019, Peter Maydell wrote:

On Thu, 7 Mar 2019 at 18:17, Marc-André Lureau
 wrote:

As someone who has also been annoyed having to support in-tree and
out-of-tree, I support that move. I used to be a pretty happy use of
in-tree builds, but out-of-tree his generally equally convenient.

Who is volunteering to do patches to deprecate the support? warning in
4.0, and error in 4.1?


Do we need to bother to deprecate it? It doesn't affect end-users,
only those who build QEMU, and the change is easy...


If you do this without warning and time for getting used to it, it would 
be nice to add some convenience functionality if possible to not upset 
people used to in-tree builds too much. That is if someone runs make in 
the top src dir it should create a build subdir and run make -C in there 
instead of just returning an error so people can still just run make as 
now instead of get frustrated every time until get used to use build dir. 
(I sometimes use build dir, sometimes in-tree build and the latter is 
often more convenient if I don't need multiple separate builds only one.)


Regards,
BALATON Zoltan

Re: [Qemu-devel] converting build system to Meson?

[Cleber Rosa]

On Thu, Mar 07, 2019 at 01:09:52PM +, Peter Maydell wrote:
> On Thu, 7 Mar 2019 at 12:56, Paolo Bonzini  wrote:
> > In any case, this wouldn't change; as you suggest below, configure could
> > remain as a front-end (well, in-srcdir builds are not supported by
> > Meson, so "../configure && ninja" perhaps).
> 
> As an aside, it might be a nice idea to drop the in-srcdir
> build altogether for QEMU anyway -- it's not really a very
> good idea and it means our build system has to cope with two
> different ways of working to no particularly useful end.
>

THIS.  Please.

- Cleber.

Re: [Qemu-devel] converting build system to Meson?

[Eric Blake]

On 3/7/19 1:04 PM, Eric Blake wrote:

> I don't mind a switch to mandatory out-of-tree builds (I like the
> convenience of in-tree for quick hacks, but VPATH is infinitely more
> flexible when testing multiple build configurations in parallel, and
> I've worked on other projects with mandatory VPATH builds without seeing
> too many complaints there).

I will add: The GNU Coding Standards require both in-tree and
out-of-tree builds to work (and you get that with Autotool'd projects).
But since qemu is intentionally not using Autotools, and is not a GNU
project, it is merely an annoyance to me if in-tree builds are broken
(I'm much more annoyed by projects that don't support VPATH builds than
by projects that don't support in-tree builds).

And if switching to out-of-tree-only eases maintenance burdens (less
.gitignore patches from me, for starters :), then we can spend developer
time on more important stuff.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] converting build system to Meson?

[Eric Blake]

On 3/7/19 12:17 PM, Marc-André Lureau wrote:
> Hi
> 
> On Thu, Mar 7, 2019 at 7:13 PM Paolo Bonzini  wrote:
>>
>> On 07/03/19 14:09, Peter Maydell wrote:
>>> On Thu, 7 Mar 2019 at 12:56, Paolo Bonzini  wrote:
 In any case, this wouldn't change; as you suggest below, configure could
 remain as a front-end (well, in-srcdir builds are not supported by
 Meson, so "../configure && ninja" perhaps).
>>>
>>> As an aside, it might be a nice idea to drop the in-srcdir
>>> build altogether for QEMU anyway -- it's not really a very
>>> good idea and it means our build system has to cope with two
>>> different ways of working to no particularly useful end.
>>
>> I was actually going to propose that, but I was afraid of throwing two
>> bombs in the same day. :)
> 
> As someone who has also been annoyed having to support in-tree and
> out-of-tree, I support that move. I used to be a pretty happy use of
> in-tree builds, but out-of-tree his generally equally convenient.
> 
> Who is volunteering to do patches to deprecate the support? warning in
> 4.0, and error in 4.1?

How you build only affects developers, so I don't see this as needing to
go through the usual qemu-deprecated.texi rules - we just get enough
developers to agree to it, then have a flag day (and hopefully a sane
enough patch to make an incremental in-tree build complain loudly, but
with clear instructions that can be copied-and-pasted to turn into a
clean VPATH build).

I don't mind a switch to mandatory out-of-tree builds (I like the
convenience of in-tree for quick hacks, but VPATH is infinitely more
flexible when testing multiple build configurations in parallel, and
I've worked on other projects with mandatory VPATH builds without seeing
too many complaints there).

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] converting build system to Meson?

[Peter Maydell]

On Thu, 7 Mar 2019 at 18:17, Marc-André Lureau
 wrote:
> As someone who has also been annoyed having to support in-tree and
> out-of-tree, I support that move. I used to be a pretty happy use of
> in-tree builds, but out-of-tree his generally equally convenient.
>
> Who is volunteering to do patches to deprecate the support? warning in
> 4.0, and error in 4.1?

Do we need to bother to deprecate it? It doesn't affect end-users,
only those who build QEMU, and the change is easy...

thanks
-- PMM

Re: [Qemu-devel] converting build system to Meson?

[Peter Maydell]

On Thu, 7 Mar 2019 at 18:18, Peter Maydell  wrote:
>
> On Thu, 7 Mar 2019 at 18:17, Marc-André Lureau
>  wrote:
> > As someone who has also been annoyed having to support in-tree and
> > out-of-tree, I support that move. I used to be a pretty happy use of
> > in-tree builds, but out-of-tree his generally equally convenient.
> >
> > Who is volunteering to do patches to deprecate the support? warning in
> > 4.0, and error in 4.1?
>
> Do we need to bother to deprecate it? It doesn't affect end-users,
> only those who build QEMU, and the change is easy...

...and note that we don't have a "warning" phase for "we added
a new build dependency so you need to do something new to be
able to build this new QEMU version".

-- PMM

Re: [Qemu-devel] converting build system to Meson?

[Alex Bennée]

Paolo Bonzini  writes:

> On 07/03/19 14:09, Peter Maydell wrote:
>> On Thu, 7 Mar 2019 at 12:56, Paolo Bonzini  wrote:
>>> In any case, this wouldn't change; as you suggest below, configure could
>>> remain as a front-end (well, in-srcdir builds are not supported by
>>> Meson, so "../configure && ninja" perhaps).
>>
>> As an aside, it might be a nice idea to drop the in-srcdir
>> build altogether for QEMU anyway -- it's not really a very
>> good idea and it means our build system has to cope with two
>> different ways of working to no particularly useful end.
>
> I was actually going to propose that, but I was afraid of throwing two
> bombs in the same day. :)

I have just discovered that gcovr has issues with out-of-srcdir builds
because it has difficulty finding the generated trace header files:

  Gathered coveraged data for 41 files
  Traceback (most recent call last):
File "/usr/bin/gcovr", line 11, in 
  load_entry_point('gcovr==4.1', 'console_scripts', 'gcovr')()
File "/usr/lib/python3/dist-packages/gcovr/__main__.py", line 588, in main
  print_html_report(covdata, options)
File "/usr/lib/python3/dist-packages/gcovr/html_generator.py", line 275, in 
print_html_report
  errors='replace')
  FileNotFoundError: [Errno 2] No such file or directory: 
'builds/gprof/i386-softmmu/accel/tcg/trace-root.h'

So it would be useful to fix the runes for that as well before we
totally deprecate in-srcdir builds.

>
>> Of the various hosts I do builds on:
>>  * OSX is 2.7 only
>>  * the ppc64 box in the gcc compile farm is 2.7 and 3.4
>>("Centos 7.5.1804")
>>  * the aarch64 box in the gcc compile farm is 2.7 and 3.4
>>("Ubuntu 14.04.5 LTS", aka trusty)
>
> OS X will need to get Python 3 from Homebrew sooner or later anyway.
>
> Trusty does have a python3.5 package, perhaps you could ask the
> maintainers to install it.
>
> CentOS 7 doesn't have "native" Python 3 (even the 3.4 version you have
> there is probably coming from Fedora) but it has software collections,
> where you have to do "scl enable rh-python35 './configure && make'".
> You can check if scl and the rh-python35 software collections are
> already installed.
>
> Paolo


--
Alex Bennée

Re: [Qemu-devel] converting build system to Meson?

[Marc-André Lureau]

Hi

On Thu, Mar 7, 2019 at 7:13 PM Paolo Bonzini  wrote:
>
> On 07/03/19 14:09, Peter Maydell wrote:
> > On Thu, 7 Mar 2019 at 12:56, Paolo Bonzini  wrote:
> >> In any case, this wouldn't change; as you suggest below, configure could
> >> remain as a front-end (well, in-srcdir builds are not supported by
> >> Meson, so "../configure && ninja" perhaps).
> >
> > As an aside, it might be a nice idea to drop the in-srcdir
> > build altogether for QEMU anyway -- it's not really a very
> > good idea and it means our build system has to cope with two
> > different ways of working to no particularly useful end.
>
> I was actually going to propose that, but I was afraid of throwing two
> bombs in the same day. :)

As someone who has also been annoyed having to support in-tree and
out-of-tree, I support that move. I used to be a pretty happy use of
in-tree builds, but out-of-tree his generally equally convenient.

Who is volunteering to do patches to deprecate the support? warning in
4.0, and error in 4.1?

thanks

Re: [Qemu-devel] converting build system to Meson?

[Paolo Bonzini]

On 07/03/19 14:09, Peter Maydell wrote:
> On Thu, 7 Mar 2019 at 12:56, Paolo Bonzini  wrote:
>> In any case, this wouldn't change; as you suggest below, configure could
>> remain as a front-end (well, in-srcdir builds are not supported by
>> Meson, so "../configure && ninja" perhaps).
> 
> As an aside, it might be a nice idea to drop the in-srcdir
> build altogether for QEMU anyway -- it's not really a very
> good idea and it means our build system has to cope with two
> different ways of working to no particularly useful end.

I was actually going to propose that, but I was afraid of throwing two
bombs in the same day. :)

> Of the various hosts I do builds on:
>  * OSX is 2.7 only
>  * the ppc64 box in the gcc compile farm is 2.7 and 3.4
>("Centos 7.5.1804")
>  * the aarch64 box in the gcc compile farm is 2.7 and 3.4
>("Ubuntu 14.04.5 LTS", aka trusty)

OS X will need to get Python 3 from Homebrew sooner or later anyway.

Trusty does have a python3.5 package, perhaps you could ask the
maintainers to install it.

CentOS 7 doesn't have "native" Python 3 (even the 3.4 version you have
there is probably coming from Fedora) but it has software collections,
where you have to do "scl enable rh-python35 './configure && make'".
You can check if scl and the rh-python35 software collections are
already installed.

Paolo

Re: [Qemu-devel] converting build system to Meson?

[Daniel P . Berrangé]

On Thu, Mar 07, 2019 at 01:09:52PM +, Peter Maydell wrote:
> On Thu, 7 Mar 2019 at 12:56, Paolo Bonzini  wrote:
> > In any case, this wouldn't change; as you suggest below, configure could
> > remain as a front-end (well, in-srcdir builds are not supported by
> > Meson, so "../configure && ninja" perhaps).
> 
> As an aside, it might be a nice idea to drop the in-srcdir
> build altogether for QEMU anyway -- it's not really a very
> good idea and it means our build system has to cope with two
> different ways of working to no particularly useful end.

I'd be in favour of that. A huge amount of the pain I felt when
making previous changes to QEMU's makefiles has been caused by
trying to support both in-srcdir & vpath build with the same
set of rules. Fixing one often breaks the other, so you have
to cycle back & forth doing a fix in one, then testing the
other, repeat, repeat, scream, repeat.

Personally I always do in-srcdir builds, but I'd be fine to
switch to vpath builds to reduce our maint burden in the build
system as it would be a net win overall.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Qemu-devel] converting build system to Meson?

[Peter Maydell]

On Thu, 7 Mar 2019 at 12:56, Paolo Bonzini  wrote:
> In any case, this wouldn't change; as you suggest below, configure could
> remain as a front-end (well, in-srcdir builds are not supported by
> Meson, so "../configure && ninja" perhaps).

As an aside, it might be a nice idea to drop the in-srcdir
build altogether for QEMU anyway -- it's not really a very
good idea and it means our build system has to cope with two
different ways of working to no particularly useful end.

> > I suspect needing Python 3.5 means it will take a while before all our
> > build hosts will have it.
>
> Which ones don't?  (This is the information I might be missing and why I
> have started the thread). RHEL7 has it in software collections, OS X in
> Homebrew.   RHEL6 is not a supported host anymore.

Of the various hosts I do builds on:
 * OSX is 2.7 only
 * the ppc64 box in the gcc compile farm is 2.7 and 3.4
   ("Centos 7.5.1804")
 * the aarch64 box in the gcc compile farm is 2.7 and 3.4
   ("Ubuntu 14.04.5 LTS", aka trusty)

I haven't checked the BSD VM images.

thanks
-- PMM

Re: [Qemu-devel] converting build system to Meson?

[Paolo Bonzini]

On 07/03/19 12:54, Alex Bennée wrote:
> 
> Paolo Bonzini  writes:
> 
>> Hi all,
>>
>> lately I have been thinking of converting the QEMU build system to
>> Meson.  Meson is a relatively new build system that can replace
>> Autotools or hand-written Makefiles such as QEMU; as a die-hard
>> Autotools fan, I must say that Meson is by far better than anything else
>> that has ever tried to replace Autotools, and actually has the potential
>> to do so.
> 
> I'm confused by the description of QEMU's build system as Autotools.

It's not Autotools.  However, *I* am an Autotools fan and therefore have
always looked with suspicion at Autoconf replacements such as CMake or
SCons.  Meson is the first one which not only has pleasantly surprised
me, but also I would consider an improvement for QEMU (Autotools
wouldn't be enough of an improvement to suggest it).

> . We certainly don't currently have a M4 DSL that vomits out an
> impenetrable sets of Makefiles from a set of templates.> The configure script 
> is
> certainly large but portable and readable POSIX shell.

Technically, the M4 DSL spits out shell, not Makefiles (Autoconf can be
used with hand-written Makefiles).  Still,

if test "$sdl_image" != "no" ; then
if $pkg_config SDL2_image --exists; then
if test "$static" = "yes"; then
sdl_image_libs=$($pkg_config SDL2_image --libs --static
2>/dev/null)
else
sdl_image_libs=$($pkg_config SDL2_image --libs 2>/dev/null)
fi
sdl_image_cflags=$($pkg_config SDL2_image --cflags 2>/dev/null)
sdl_image=yes

sdl_cflags="$sdl_cflags $sdl_image_cflags"
sdl_libs="$sdl_libs $sdl_image_libs"
else
if test "$sdl_image" = "yes" ; then
feature_not_found "sdl_image" "Install SDL Image devel"
else
sdl_image=no
fi
fi
fi

is a bit more verbose than

sdl_image = dependency('SDL2_image',
   static: get_option('static'),
   required: get_option('sdl_image')):

We are used to cut-and-paste, it doesn't mean it's the best way to
approach the problem.

> Finally there is a lot to be said for following the de-facto standard
> ./configure && make pattern.

At the same time, we don't follow the standard fully, for example
--extra-cflags or --cc are QEMU-isms, --srcdir is replaced by
--search-path, and so on.

In any case, this wouldn't change; as you suggest below, configure could
remain as a front-end (well, in-srcdir builds are not supported by
Meson, so "../configure && ninja" perhaps).

>> - ease of distributing a full copy of Meson to support distros that ship
>> an older version (no dependencies apart from Python 3.5).  At 4
>> lines of Python, Meson is relatively small.
> 
> I suspect needing Python 3.5 means it will take a while before all our
> build hosts will have it.

Which ones don't?  (This is the information I might be missing and why I
have started the thread). RHEL7 has it in software collections, OS X in
Homebrew.   RHEL6 is not a supported host anymore.

> However I have watched many build-systems appear over the years, often
> with wonderful stories and ambitions to avoid all the mistakes and cruft
> of "legacy" make systems only for them to run into their own
> complexities. As someone who builds lots of random software my heart
> tends to sink when I come across a non-standard system because it
> becomes another thing I have to learn just to build something. If we do
> decide to shift I would strongly encourage the front end to still be
> configure/make just to keep the initial learning curve for people
> checking out the QEMU source code for the first time.

I agree.  Again,

Paolo

Re: [Qemu-devel] converting build system to Meson?

[Alex Bennée]

Paolo Bonzini  writes:

> Hi all,
>
> lately I have been thinking of converting the QEMU build system to
> Meson.  Meson is a relatively new build system that can replace
> Autotools or hand-written Makefiles such as QEMU; as a die-hard
> Autotools fan, I must say that Meson is by far better than anything else
> that has ever tried to replace Autotools, and actually has the potential
> to do so.

I'm confused by the description of QEMU's build system as Autotools. We
certainly don't currently have a M4 DSL that vomits out an impenetrable
sets of Makefiles from a set of templates. The configure script is
certainly large but portable and readable POSIX shell. While Makefiles
are old school I think QEMU's hold up as a reasonable example of how
clean modern Makefiles can be.

Finally there is a lot to be said for following the de-facto standard
./configure && make pattern.

> Advantages of Meson that directly matter for QEMU include:
>
> - build definitions in a very readable and user friendly DSL, which
> supports looping and conditions.
>
> - ability to introspect the build definitions so that you can find out
> what is built without building it (the DSL is not Turing complete and
> most objects in it are immutable, so it cannot be abused that much :))
>
> - support for a non-recursive build from per-subdirectory input (similar
> to Makefile.objs)
>
> - ease of distributing a full copy of Meson to support distros that ship
> an older version (no dependencies apart from Python 3.5).  At 4
> lines of Python, Meson is relatively small.

I suspect needing Python 3.5 means it will take a while before all our
build hosts will have it.

> Unlike Autoconf, meson.build files include both feature detection and
> build rules.  Also unlike Autoconf, Meson must be present on the system
> of whoever builds QEMU, which is why the last bullet above matters.
> However, it is possible to keep a configure script, if only to provide
> backwards command line compatibility and support a bundled copy of Meson.
>
> Of course a full conversion of QEMU's build system, including the 7k
> lines configure script is not easy.  In addition, even though Meson
> supports extension modules that are written in Python, those are really
> used only for namespacing purposes as the extensions need to live in the
> Meson tree and not in the QEMU tree.  Therefore I have tried to identify
> the parts of QEMU's Makefiles that could complicate the transition, and
> see in advance if they can be addressed:
>
> - --enable-modules's usage of special linker options -u SYMBOL.  For
> this I initially wrote patches to Meson that would make everything
> automatic, however they were rejected.  Instead, the Meson maintainer
> prodded me to find a way to implement the behavior in QEMU's build
> system without having to extend Meson, and we managed to find one.  This
> was a good exercise because it showed that, despite being very
> "opinionated", Meson manages to be pretty flexible too.
>
> - ease of use for test logs and the ability to cut and paste test
> invocations from the logs to the command line.  For this I have started
> "probing" how the Meson developers feel about this kind of change[1],
> and intend to follow up until the meson test driver is comparable in
> usability to QEMU's "make check",
>
> - ease of converting Makefile.objs files.  The Makefile.objs files are
> very nice to change for simple modifications, and any replacement should
> have the same feature.  This will require a Meson extension which I have
> proposed already[2], where adding a source file will look like
>
> obj.add(['CONFIG_VIRTIO'], files('virtio.c'),
> if_false: files('virtio-stub.c'))
> common_obj.add(['CONFIG_SDL'], files('sdl.c'],
>dependencies: sdl)
>
> This is a little more verbose than Makefile.objs, but should be okay
> assuming the extension is accepted.  Rejection of my extension would be
> a blocker though, because QEMU is quite special in how it builds dozens
> of large binaries, all *from the same sources* (unlike e.g. gstreamer
> which has >100 build targets but they are all independent plugins).

Personally these seems more verbose and more potential for subtle errors
to creep in because you've replaced simple textual expansion with code
that can now behave weirdly if you get the wrong , & ()s.

> Rules for generators like trace-tool or QAPI would becomes quite a bit
> more verbose.  Hopefully this is offset by increased clarity if we don't
> rely anymore on stuff like pattern rules and instead have foreach
> loops.

I think there is certainly scope for removing these complex generators
out into scripts and maybe even generating the make fragments for them.

>
> - unintended performance issues.  I wouldn't be surprised if QEMU
> stressed Meson more than many other projects did.  However, this should
> be mitigated by the fact that source selection is done in Python rather
> than in Meson's DSL.  Everything else is O

Re: [Qemu-devel] converting build system to Meson?

[Marc-André Lureau]

Hi

On Thu, Mar 7, 2019 at 12:29 PM Dr. David Alan Gilbert
 wrote:

> Having said that, the counterpoint in Meson is that it's build
> directories seem tobe exact-meson version dependent; doing a minor
> host update and you find you have to nuke your build directories.
>

I have heard from meson developers at FOSDEM that this kind
meson-version problems were recently addressed (sorry, I don't have
any pointers)

-- 
Marc-André Lureau

Re: [Qemu-devel] converting build system to Meson?

[Daniel P . Berrangé]

On Thu, Mar 07, 2019 at 10:13:13AM +, Peter Maydell wrote:
> So I think we'd need a more compelling reason to move right now.
> (This might change in the future, eg if Meson catches on to the
> extent that everybody is using it and competitors like CMake are
> more obviously eclipsed by it, in the way that git took over
> from svn and relegated mercurial and bzr to obscurity.)

I don't think we'll see the same level of domination as you see with
git, for a number of reasons.

It is compelling enough though that I think it will ultimately win
over autotools for newly started projects with little difficulty.
Everyone I know grumbles about autoconf & m4, and libtool, so they
won't be missed. 

Switching SCM is relatively straightforward as for most projects it
doesn't affect the way the code works. At simplest you can just copy
the code into git & away you go. If you want to preseve history there
are many convertors from SVN/CVS and of course free project hosting
on github just re-inforces the benefits.

Switching build systems by comparison requires careful technical
work whose burden increases as the complexity of the project increases.
This is a much larger barrier than seen with GIT conversions. So I think
it will have a harder time getting existing projects to convert than
was seen getting projeects from switch to GIT. This is true of any
build system conversion though, not just meson. There's no real scope
for automatic conversions to help her.

Most major modern programming languages comes some kind of standard
build system. So while meson can work with non-C languages it is not
likely to make much headway there IMHO.

Even if we don't switch build systems in the near term, I think it
is worth watching the space & being open to the idea at some point.

I think our current build system does put off contributors. I'm
personally put off touching any make based build system that is
not using autoconf/automake. Project specific hand crafted make
files have their own new way to doing things you have to learn
and so for a long time I dared not try changing anything complex
in QEMU's build.

As long term contributors we've built enough enough knowledge to
QEMU to consider our build system attractive or even "simple",
which skews our view of the benefits of alternatives to people
without this tribal knowledge.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Qemu-devel] converting build system to Meson?

[Peter Maydell]

On Thu, 7 Mar 2019 at 11:29, Dr. David Alan Gilbert  wrote:
> Pretty regularly I just give up on build directories and recreate
> them because of changes that our Makefiles haven't realised
> when updating a tree.
> (I can't say it's any one fault anywhere)

That's usually the awkward problem of "newer version of the
build rules doesn't work when it starts against older
version of the built objects" or similar. Does Meson do
a better job about handling that situation?

thanks
-- PMM

Re: [Qemu-devel] converting build system to Meson?

[Dr. David Alan Gilbert]

* Peter Maydell (peter.mayd...@linaro.org) wrote:
> On Thu, 7 Mar 2019 at 06:39, Thomas Huth  wrote:
> > On 06/03/2019 19.12, Paolo Bonzini wrote:
> > > lately I have been thinking of converting the QEMU build system to
> > > Meson.  Meson is a relatively new build system that can replace
> > > Autotools or hand-written Makefiles such as QEMU; as a die-hard
> > > Autotools fan, I must say that Meson is by far better than anything else
> > > that has ever tried to replace Autotools, and actually has the potential
> > > to do so.
> > >
> > > Advantages of Meson that directly matter for QEMU include:[...]
> >
> > I'm not objecting a new build system per se, but could you elaborate on
> >  problems of the current QEMU build system that will be fixed by this
> > change? Since apart from some minor glitches (with the *.mak file
> > dependencies for example), the current build system seems to work quite
> > well for me ... so at least I currently don't feel enough pain yet to do
> > such a big step, just because there is another new cool build system
> > around...
> 
> Yes, that tends to be my view. Our current build system:
>  * has no dependencies that are problematic for older hosts
>(contrast Meson, which needs Python 3.5, even if we take
>the drastic step of shipping an entire build tool along
>with QEMU; OSX python is 2.7 still)
>  * is not particularly hard to deal with for the common cases
>("add new source file" is straightforward)
>  * covers all our requirements as far as I'm aware
>(whereas you've listed a couple of places where Meson
>would need changes/extensions to support things we do already)
>  * is generally flexible enough to be hackable to deal with odd
>cases (it has escape mechanisms to generic-programmability,
>even if they're ugly and awkward)

Pretty regularly I just give up on build directories and recreate
them because of changes that our Makefiles haven't realised
when updating a tree.
(I can't say it's any one fault anywhere)

Having said that, the counterpoint in Meson is that it's build
directories seem tobe exact-meson version dependent; doing a minor
host update and you find you have to nuke your build directories.

Dave

> So I think we'd need a more compelling reason to move right now.
> (This might change in the future, eg if Meson catches on to the
> extent that everybody is using it and competitors like CMake are
> more obviously eclipsed by it, in the way that git took over
> from svn and relegated mercurial and bzr to obscurity.)
> 
> thanks
> -- PMM
> 
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [Qemu-devel] converting build system to Meson?

[Paolo Bonzini]

On 07/03/19 11:13, Peter Maydell wrote:
> Yes, that tends to be my view. Our current build system:
>  * has no dependencies that are problematic for older hosts
>(contrast Meson, which needs Python 3.5, even if we take
>the drastic step of shipping an entire build tool along
>with QEMU; OSX python is 2.7 still)

Regarding OS X, don't we require Homebrew anyway?  (We will have to deal
with the Python 2 issue sometime in 2020, and I don't expect much
happening wrt Meson in QEMU before then).

>  * is not particularly hard to deal with for the common cases
>("add new source file" is straightforward)
>  * covers all our requirements as far as I'm aware
>(whereas you've listed a couple of places where Meson
>would need changes/extensions to support things we do already)

Indeed---I mentioned that rejection of these changes/extensions would be
a blocker.  I think those extensions are useful anyway, so I am
contributing them to Meson even if we don't end up using them.  My
experience so far with the Meson community makes me positive that _some_
kind of solution will be found even if it's not what I'm proposing.

> (This might change in the future, eg if Meson catches on to the
> extent that everybody is using it and competitors like CMake are
> more obviously eclipsed by it, in the way that git took over
> from svn and relegated mercurial and bzr to obscurity.)

It's very hard to displace established contenders in this area because
unlike, switching to git or svn, a switch involves a major rewrite
effort.  Nevertheless, big projects _are_ switching.

Meson has a very different design than Autotools or even CMake, despite
the apparent similarity with the latter.  The lack of an escape
mechanism sounds annoying, but it is much less of a problem than you'd
think.  In fact it has the interesting side effect, of discouraging
forks very much: new features are contributed upstream for everyone's
use and vetted by the Meson developers.

This makes Meson way more polished than CMake or Autotools, or
hand-crafted Makefile such as ours; so it takes more effort to switch,
but the result is good.  I have started following Meson about a year
ago, and a lot of missing features have been added since then, slowly
but surely, which is why I am only reaching out now to you and others.

Paolo

Re: [Qemu-devel] converting build system to Meson?

[Paolo Bonzini]

On 07/03/19 11:48, Peter Maydell wrote:
> On Thu, 7 Mar 2019 at 10:40, Paolo Bonzini  wrote:
>> ---
>> # This would be ui/meson.build:
>>
>> vnc_sources = source_set.source_set()
>> vnc_sources.add([], files(
>> 'vnc.c',
>> 'vnc-auth-vencrypt.c',
>> 'vnc-enc-hextile.c',
>> 'vnc-enc-tight.c',
>> 'vnc-enc-zlib.c',
>> 'vnc-enc-zrle.c',
>> 'vnc-jobs.c',
>> 'vnc-palette.c',
>> 'vnc-ws.c'))
>> vnc_sources.add('CONFIG_VNC_SASL', files(
>> 'vnc-auth-sasl.c'))
> 
> All this foo_sources.add() seems pretty clunky syntax,
> which doesn't seem like a good sign for what is
> pretty much the simplest possible case for the build system.

ui/Makefile.objs is actually one of the most complex cases.  The quotes
and commas are indeed somewhat clunky, but I don't think the above
compares particularly negatively to

vnc-obj-y = vnc.o
vnc-obj-y += vnc-auth-vencrypt.o
vnc-obj-y += vnc-enc-hextile.o
vnc-obj-y += vnc-enc-tight.o
vnc-obj-y += vnc-enc-zlib.o
vnc-obj-y += vnc-enc-zrle.o
vnc-obj-y += vnc-jobs.o
vnc-obj-y += vnc-palette.o
vnc-obj-y += vnc-ws.o
vnc-obj-$(CONFIG_VNC_SASL) += vnc-auth-sasl.o

The simplest possible case is

common-obj-$(CONFIG_IPACK) += ipack.o tpci200.o

=>  common_obj.add('CONFIG_IPACK', files('ipack.c', 'tpci200.c'))

or

qemu-keymap$(EXESUF): qemu-keymap.o ui/input-keymap.o $(COMMON_LDADDS)
qemu-keymap$(EXESUF): LIBS += $(XKBCOMMON_LIBS)
qemu-keymap$(EXESUF): QEMU_CFLAGS += $(XKBCOMMON_CFLAGS)

=>  executable('qemu-keymap',
   ['qemu-keymap.c', 'ui/input-keymap.c'],
   dependencies: [common_deps, xkbcommon])

(where the latter provides "make install" functionality without having
to touch configure).

Paolo

Re: [Qemu-devel] converting build system to Meson?

[Daniel P . Berrangé]

On Thu, Mar 07, 2019 at 11:40:05AM +0100, Paolo Bonzini wrote:
> On 07/03/19 07:39, Thomas Huth wrote:
> > On 06/03/2019 19.12, Paolo Bonzini wrote:
> >> Hi all,
> >>
> >> lately I have been thinking of converting the QEMU build system to
> >> Meson.  Meson is a relatively new build system that can replace
> >> Autotools or hand-written Makefiles such as QEMU; as a die-hard
> >> Autotools fan, I must say that Meson is by far better than anything else
> >> that has ever tried to replace Autotools, and actually has the potential
> >> to do so.
> >>
> >> Advantages of Meson that directly matter for QEMU include:[...]
> > 
> > I'm not objecting a new build system per se, but could you elaborate on
> >  problems of the current QEMU build system that will be fixed by this
> > change?
> 
> In order of importance:
> 
> - Make lacks introspection abilities; people want to be able to quickly
> say "I don't have this security issue because I didn't compile this
> file".  Make only gives you the build logs, which may not be easily
> accessible to the end user.
> 
> - Makefile.objs unnesting does not apply throughout the tree, for
> example it does not apply to tests.  We don't apply it there because it
> it is hard to extend Makefile.objs to dozens of targets and it would
> also be very very slow.  The result is lack of homogeneity between
> places that use Makefile.objs and places that use Makefile.include.
> 
> - Makefile.objs unnesting is hard to understand---granted, you rarely
> have to understand it because it mostly works, but we should consider
> whether it is a local optimum.  For example, how many people know how to
> add a dynamically-loadd module to Makefile.objs?

We have a long standing bug in unnesting which breaks if two dirs
have a .c file with the same name and a per-o file library link:

  https://lists.gnu.org/archive/html/qemu-devel/2018-06/msg02654.html

The nesting stuff is insanely clever, but it is also fragile and full
of dragons like this one.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Qemu-devel] converting build system to Meson?

[Peter Maydell]

On Thu, 7 Mar 2019 at 10:40, Paolo Bonzini  wrote:
> ---
> # This would be ui/meson.build:
>
> vnc_sources = source_set.source_set()
> vnc_sources.add([], files(
> 'vnc.c',
> 'vnc-auth-vencrypt.c',
> 'vnc-enc-hextile.c',
> 'vnc-enc-tight.c',
> 'vnc-enc-zlib.c',
> 'vnc-enc-zrle.c',
> 'vnc-jobs.c',
> 'vnc-palette.c',
> 'vnc-ws.c'))
> vnc_sources.add('CONFIG_VNC_SASL', files(
> 'vnc-auth-sasl.c'))

All this foo_sources.add() seems pretty clunky syntax,
which doesn't seem like a good sign for what is
pretty much the simplest possible case for the build system.

thanks
-- PMM

Re: [Qemu-devel] converting build system to Meson?

[Paolo Bonzini]

On 07/03/19 07:39, Thomas Huth wrote:
> On 06/03/2019 19.12, Paolo Bonzini wrote:
>> Hi all,
>>
>> lately I have been thinking of converting the QEMU build system to
>> Meson.  Meson is a relatively new build system that can replace
>> Autotools or hand-written Makefiles such as QEMU; as a die-hard
>> Autotools fan, I must say that Meson is by far better than anything else
>> that has ever tried to replace Autotools, and actually has the potential
>> to do so.
>>
>> Advantages of Meson that directly matter for QEMU include:[...]
> 
> I'm not objecting a new build system per se, but could you elaborate on
>  problems of the current QEMU build system that will be fixed by this
> change?

In order of importance:

- Make lacks introspection abilities; people want to be able to quickly
say "I don't have this security issue because I didn't compile this
file".  Make only gives you the build logs, which may not be easily
accessible to the end user.

- Makefile.objs unnesting does not apply throughout the tree, for
example it does not apply to tests.  We don't apply it there because it
it is hard to extend Makefile.objs to dozens of targets and it would
also be very very slow.  The result is lack of homogeneity between
places that use Makefile.objs and places that use Makefile.include.

- Makefile.objs unnesting is hard to understand---granted, you rarely
have to understand it because it mostly works, but we should consider
whether it is a local optimum.  For example, how many people know how to
add a dynamically-loadd module to Makefile.objs?

- The mix of recursive and non-recursive builds is hard to understand
and slow, because we do the the Makefile.objs unnesting 20-30 times
(once for every target).  We do spend time cleaning up warts left when
contributors just hacked something together and reviewers lacked the
expertise to notice that.

- Makefile errors (e.g. pointing to a nonexistent file) only show when
trying to build the file, so that some errors may go unnoticed if they
don't reproduce on your machine.  Meson always detects missing source
files, independent of whether your build uses them.

FWIW and just to give an idea, I placed a quick (and of course untested)
conversion of ui/Makefile.objs at the end of this message.

Paolo

[1] https://clang.llvm.org/docs/JSONCompilationDatabase.html

---
# This would be ui/meson.build:

vnc_sources = source_set.source_set()
vnc_sources.add([], files(
'vnc.c',
'vnc-auth-vencrypt.c',
'vnc-enc-hextile.c',
'vnc-enc-tight.c',
'vnc-enc-zlib.c',
'vnc-enc-zrle.c',
'vnc-jobs.c',
'vnc-palette.c',
'vnc-ws.c'))
vnc_sources.add('CONFIG_VNC_SASL', files(
'vnc-auth-sasl.c'))

common_obj.add([], files(
'console.c',
'cursor.c',
'kbd-state.c',
'keymaps.c',
'input.c',
'input-keymap.c',
'input-legacy.c',
'qemu-pixman.c'))
common_obj.add('CONFIG_LINUX', files(
'input-linux.c'))
common_obj.add('CONFIG_SPICE', files(
'spice-core.c',
'spice-input.c',
'spice-display.c'))
common_obj.add('CONFIG_COCOA', files(
'cocoa.c'))
common_obj.add_all('CONFIG_VNC', vnc_sources)
common_obj.add('CONFIG_VNC',
if_false: files('vnc-stubs.c'))

if opengl.found()
common_obj.add([], files(
'console-gl.c',
'egl-helpers.c',
'egl-context.c',
'shader.c'), dependencies: opengl)
common_obj.add('CONFIG_OPENGL_DMABUF', files(
'egl-headless.c'))
endif

if sdl.found()
sdl_sources = source_set.source_set()
sdl_sources.add([], files(
'sdl2.c',
'sdl2-2d.c',
'sdl2-input.c'), dependencies: sdl)
sdl_sources.add('CONFIG_X11', files(
'x_keymap.c'), dependencies: x11)
sdl_sources.add('CONFIG_OPENGL', files(
'sdl2-gl.c'), dependencies: opengl)
modules += { 'ui-sdl': sdl_sources }
endif

if curses.found()
curses_sources = source_set.source_set()
curses_sources.add([], files(
'curses.c'), dependencies: curses)
modules += { 'ui-curses': curses_sources }
endif

if gtk.found()
gtk_sources = source_set.source_set()
gtk_sources.add([], files(
'gtk.c'), dependencies: gtk)
gtk_sources.add('CONFIG_X11', files(
'x_keymap.c'), dependencies: x11)
gtk_sources.add('CONFIG_OPENGL', files(
'gtk-egl.c'), dependencies: opengl)
gtk_sources.add('CONFIG_GTK_GL', files(
'gtk-gl-area.c'))
modules += { 'ui-gtk': gtk_sources }
fi

Re: [Qemu-devel] converting build system to Meson?

[Stefan Hajnoczi]

On Wed, Mar 06, 2019 at 07:12:33PM +0100, Paolo Bonzini wrote:
> lately I have been thinking of converting the QEMU build system to
> Meson.  Meson is a relatively new build system that can replace
> Autotools or hand-written Makefiles such as QEMU; as a die-hard
> Autotools fan, I must say that Meson is by far better than anything else
> that has ever tried to replace Autotools, and actually has the potential
> to do so.

Meson is gaining in popularity and I recently got some experience with
it.  It doesn't feel much worse or better than what QEMU or Linux have
for basic tasks.  For advanced tasks, I can't say.

One frustration is that any time the meson package is upgraded (when
your distro pushes an update) it forces a rebuild from scratch :(.

I think for 95% of people Meson vs Make will make little difference (the
syntax is easy enough) and 5% of people might be able to do more
advanced things cleanly with Meson.  Sounds good to me.

Stefan


signature.asc
Description: PGP signature

Re: [Qemu-devel] converting build system to Meson?

[Daniel P . Berrangé]

On Wed, Mar 06, 2019 at 07:50:49PM +0100, Marc-André Lureau wrote:
> Hi Paolo
> 
> On Wed, Mar 6, 2019 at 7:12 PM Paolo Bonzini  wrote:
> >
> > Hi all,
> >
> > lately I have been thinking of converting the QEMU build system to
> > Meson.  Meson is a relatively new build system that can replace
> > Autotools or hand-written Makefiles such as QEMU; as a die-hard
> > Autotools fan, I must say that Meson is by far better than anything else
> > that has ever tried to replace Autotools, and actually has the potential
> > to do so.
> >
> > Advantages of Meson that directly matter for QEMU include:
> >
> > - build definitions in a very readable and user friendly DSL, which
> > supports looping and conditions.
> >
> > - ability to introspect the build definitions so that you can find out
> > what is built without building it (the DSL is not Turing complete and
> > most objects in it are immutable, so it cannot be abused that much :))
> >
> > - support for a non-recursive build from per-subdirectory input (similar
> > to Makefile.objs)
> >
> > - ease of distributing a full copy of Meson to support distros that ship
> > an older version (no dependencies apart from Python 3.5).  At 4
> > lines of Python, Meson is relatively small.
> >
> >
> > Unlike Autoconf, meson.build files include both feature detection and
> > build rules.  Also unlike Autoconf, Meson must be present on the system
> > of whoever builds QEMU, which is why the last bullet above matters.
> > However, it is possible to keep a configure script, if only to provide
> > backwards command line compatibility and support a bundled copy of Meson.
> 
> Yes, we will probably have a transition period where both build-sys
> have to co-exist (same story with libslirp, which I proposed as
> meson-only first)

I'd very much *not* want to have them co-existing. Doing that significantly
increases the maint burden for any developer needing to change build rules.
It also introduces significant risks that each build system produces
different results. GTK took this approach and created a number of serious
problems where meson results differed from autoconf results.

> > Of course a full conversion of QEMU's build system, including the 7k
> > lines configure script is not easy.  In addition, even though Meson
> 
> I don't think a complete transition in one go is feasible. Could we
> start with some basic things?
> 
> For example, we could quite easily start converting tools (qemu-img
> etc), some tests...

The difficulty of doing this is that alot of our code is shared between
the tools and the system emulators.  It is particularly problematic for
code that is auto-generated upfront. For example trace files and qapi
files. The new build system would have to be able to consume .o & .c
files already built by the other build system, with correct dependancy
ordering to ensure parallel build succeeds.  I'm not saying it is
impossible, but it is certainly very hard, to the extent that it may
well be easier to do a big-bang.


> > even just "include" the Meson-generated build system into Make and keep
> > a (progressively more) minimal Makefile veneer around it.  I benchmarked
> > Make a bit and its slowness of QEMU on a do-nothing build is simply due
> > to the complexity of the unnest-vars macro machinery; Make could parse a
> > synthetic 9-target Makefile in 2 seconds (QEMU has 9000 object
> > files), and the ninja->Make converter is careful to avoid traps where
> > Make has quadratic complexity.
> >
> >
> > To be clear, this is not something I am going to work anytime soon.
> > Still in my opinion it's a worthwhile exercise to think about it.  As I
> > will see in the next few weeks what the Meson maintainers' reaction will
> > be to some of the proposed extensions, I wanted to gauge the reactions
> > of you all as well. :)
> 
> I like meson better than anything else. It is still young, one can
> find quickly limitations & documentation could be better (if you
> compare it with make/autoconf/automake reference manual for ex).

I actually though the meson documentation was really pretty decent.
Where I've personally had problems was when trying to find docs for
features that don't exist yet in meson :-P

I like it better than any of the other programs that has been put
forwards as an autotools killer, to the extent I've been converting
all my personal projects to it. It might not be mature enough to
cover all the hard bits, but what it does do, it does very clearly
making the resulting build files nice to maintain.

Converting QEMU though would certainly be a large challenge in scope
pushing the boundaries of what's possible.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Qemu-devel] converting build system to Meson?

[Daniel P . Berrangé]

On Thu, Mar 07, 2019 at 07:39:46AM +0100, Thomas Huth wrote:
> On 06/03/2019 19.12, Paolo Bonzini wrote:
> > Hi all,
> > 
> > lately I have been thinking of converting the QEMU build system to
> > Meson.  Meson is a relatively new build system that can replace
> > Autotools or hand-written Makefiles such as QEMU; as a die-hard
> > Autotools fan, I must say that Meson is by far better than anything else
> > that has ever tried to replace Autotools, and actually has the potential
> > to do so.
> > 
> > Advantages of Meson that directly matter for QEMU include:[...]
> 
> I'm not objecting a new build system per se, but could you elaborate on
>  problems of the current QEMU build system that will be fixed by this
> change? Since apart from some minor glitches (with the *.mak file
> dependencies for example), the current build system seems to work quite
> well for me ... so at least I currently don't feel enough pain yet to do
> such a big step, just because there is another new cool build system
> around...

Despite my effort to document how QEMU's makefiles are structured
(docs/devel/build-system.txt) I still find them to be pretty unintelligable
and painful to deal with. The way we have recursive make with some objects
built once & others built twice is particularly painful to deal with when
making non-trivial changes to the makefiles.

The configure script I think is very compelling to replace on the general
principle that any shell script longer than one line is better off being
written in another programming language. Portable shell programming (to
/bin/sh standards as opposed to /bn/bash) is difficult & we constantly
get it wrong introducing bash-isms. Error handling in shell is a complete
disaster zone and string quoting is not much better. 

The domain specific language used by meson is more attractive avoiding
these pitfalls of shell and reducing the need for write boilerplate
code to check features, letting you just write declarative statements
to a large extent.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Qemu-devel] converting build system to Meson?

[Peter Maydell]

On Thu, 7 Mar 2019 at 06:39, Thomas Huth  wrote:
> On 06/03/2019 19.12, Paolo Bonzini wrote:
> > lately I have been thinking of converting the QEMU build system to
> > Meson.  Meson is a relatively new build system that can replace
> > Autotools or hand-written Makefiles such as QEMU; as a die-hard
> > Autotools fan, I must say that Meson is by far better than anything else
> > that has ever tried to replace Autotools, and actually has the potential
> > to do so.
> >
> > Advantages of Meson that directly matter for QEMU include:[...]
>
> I'm not objecting a new build system per se, but could you elaborate on
>  problems of the current QEMU build system that will be fixed by this
> change? Since apart from some minor glitches (with the *.mak file
> dependencies for example), the current build system seems to work quite
> well for me ... so at least I currently don't feel enough pain yet to do
> such a big step, just because there is another new cool build system
> around...

Yes, that tends to be my view. Our current build system:
 * has no dependencies that are problematic for older hosts
   (contrast Meson, which needs Python 3.5, even if we take
   the drastic step of shipping an entire build tool along
   with QEMU; OSX python is 2.7 still)
 * is not particularly hard to deal with for the common cases
   ("add new source file" is straightforward)
 * covers all our requirements as far as I'm aware
   (whereas you've listed a couple of places where Meson
   would need changes/extensions to support things we do already)
 * is generally flexible enough to be hackable to deal with odd
   cases (it has escape mechanisms to generic-programmability,
   even if they're ugly and awkward)

So I think we'd need a more compelling reason to move right now.
(This might change in the future, eg if Meson catches on to the
extent that everybody is using it and competitors like CMake are
more obviously eclipsed by it, in the way that git took over
from svn and relegated mercurial and bzr to obscurity.)

thanks
-- PMM

Re: [Qemu-devel] converting build system to Meson?

[Thomas Huth]

On 06/03/2019 19.12, Paolo Bonzini wrote:
> Hi all,
> 
> lately I have been thinking of converting the QEMU build system to
> Meson.  Meson is a relatively new build system that can replace
> Autotools or hand-written Makefiles such as QEMU; as a die-hard
> Autotools fan, I must say that Meson is by far better than anything else
> that has ever tried to replace Autotools, and actually has the potential
> to do so.
> 
> Advantages of Meson that directly matter for QEMU include:[...]

I'm not objecting a new build system per se, but could you elaborate on
 problems of the current QEMU build system that will be fixed by this
change? Since apart from some minor glitches (with the *.mak file
dependencies for example), the current build system seems to work quite
well for me ... so at least I currently don't feel enough pain yet to do
such a big step, just because there is another new cool build system
around...

Just my 0.02 €, of course.

 Thomas


PS: I just saw https://mesonbuild.com/Dependencies.html#cmake ... that's
a funny idea :-)

Re: [Qemu-devel] converting build system to Meson?

[Paolo Bonzini]

On 06/03/19 19:50, Marc-André Lureau wrote:
>> Unlike Autoconf, meson.build files include both feature detection and
>> build rules.  Also unlike Autoconf, Meson must be present on the system
>> of whoever builds QEMU, which is why the last bullet above matters.
>> However, it is possible to keep a configure script, if only to provide
>> backwards command line compatibility and support a bundled copy of Meson.
> 
> Yes, we will probably have a transition period where both build-sys
> have to co-exist (same story with libslirp, which I proposed as
> meson-only first)
>
>> Of course a full conversion of QEMU's build system, including the 7k
>> lines configure script is not easy.  In addition, even though Meson
> 
> I don't think a complete transition in one go is feasible. Could we
> start with some basic things?
> 
> For example, we could quite easily start converting tools (qemu-img
> etc), some tests...

Tools use Makefile.objs just like everything else, on the other hand
tests and libqemu*.a may be a good start.  Also the generated files.

Co-existence as in being able to build with either one or the other is
complicated, and this is also why I did the ninja-to-make converter.
The idea is that configure would run "meson setup" just before exiting,
run ninja2make, and then the ninja rules would be included transparently
in the Makefile.

Even after a full transition to Meson, you'd do "../configure && ninja",
not "meson setup build && ninja -C build".  (Another possible thing to
do in preparation could be to deprecate in-tree builds).

>> supports extension modules that are written in Python, those are really
>> used only for namespacing purposes as the extensions need to live in the
> 
> I didn't know, I suppose meson doesn't want to support an internal API?

Yes.  Extensions need to know about all sorts of Meson innards,
especially if they do things such as defining custom "classes" for use
in meson.build.  They are not particularly easy to write if you haven't
previously hacked Meson, and given Meson's overall design philosophy,
module hackability is absolutely not part of the goals.

But I see no reason why the two extensions QEMU needs couldn't be
upstreamed; in fact the kconfig module was started two years ago (and
never completed, which is good because it wasn't a great API).  It's
ugly that until an extension is declared "stable" it prints a warning
when you run meson, but hopefully the Meson developers might declare
them stable as soon as QEMU is ready to switch.

>> - ability to use the currently in progress Kconfig declarations for
>> dependencies.  Meson developers have expressed that they would accept an
>> extension module that loads Kconfig-like declarations from a file; this
>> would also be useful to start using Meson only as a Make replacement
>> while keeping the current configure script[4], since we could slurp
> 
> (you meant [3])

No, it was supposed to be a "hidden gem" link to ninja2make, but then I
expanded more on the topic below and ended up not adding the link at all
as you noticed.

>>if config.get('BAR') == 'y' and not config.has('FOO') then
>>  config.set('FOO', 'y')
>>endif
>>if config.get('FOO') == 'y' then
>>  if config.has('BAZ') and config.get('BAZ') != 'y' then
>>error('FOO selects BAZ, but BAZ is disabled')
>>  endif
>>  config.set('BAZ', 'y')
>>endif
>>
>> I have not tried doing this, but it seems feasible.
> 
> I am confused, I thought minikconf was generating a .mak (I am not
> very familiar with it, I am waiting for it to land!)

Yes.  But with Meson it would have to be done differently, you would
have a Kconfig->meson.build translator and then the kconfig logic is
executed at "meson setup" time.

>> - Meson generates a build.ninja file rather than a Makefile, which also
>> complicates a bit interoperability with the current system.  This is
>> perhaps minor, since in most cases the change is just s/make/ninja/ but
>> it may matter for day-to-day development.  For this I have prototyped a
>> converter (written in Python) from ninja back to Make, so that we could
> 
> wow, interesting, and no link? :)

https://gist.githubusercontent.com/bonzini/8ed6659a14213818675e1039fa27c221/raw/1c68b1ecdd94b2270ee1a779e0edfafb42e35896/ninja2make.py

The kind of thing where you're not sure whether to be proud or ashamed
of what you've just done. :)

>> To be clear, this is not something I am going to work anytime soon.
>> Still in my opinion it's a worthwhile exercise to think about it.  As I
>> will see in the next few weeks what the Meson maintainers' reaction will
>> be to some of the proposed extensions, I wanted to gauge the reactions
>> of you all as well. :)
> 
> I like meson better than anything else. It is still young, one can
> find quickly limitations & documentation could be better (if you
> compare it with make/autoconf/automake reference manual for ex). But I
> also believe it has the potential to improve the qemu build-sys an

Re: [Qemu-devel] converting build system to Meson?

[Marc-André Lureau]

Hi Paolo

On Wed, Mar 6, 2019 at 7:12 PM Paolo Bonzini  wrote:
>
> Hi all,
>
> lately I have been thinking of converting the QEMU build system to
> Meson.  Meson is a relatively new build system that can replace
> Autotools or hand-written Makefiles such as QEMU; as a die-hard
> Autotools fan, I must say that Meson is by far better than anything else
> that has ever tried to replace Autotools, and actually has the potential
> to do so.
>
> Advantages of Meson that directly matter for QEMU include:
>
> - build definitions in a very readable and user friendly DSL, which
> supports looping and conditions.
>
> - ability to introspect the build definitions so that you can find out
> what is built without building it (the DSL is not Turing complete and
> most objects in it are immutable, so it cannot be abused that much :))
>
> - support for a non-recursive build from per-subdirectory input (similar
> to Makefile.objs)
>
> - ease of distributing a full copy of Meson to support distros that ship
> an older version (no dependencies apart from Python 3.5).  At 4
> lines of Python, Meson is relatively small.
>
>
> Unlike Autoconf, meson.build files include both feature detection and
> build rules.  Also unlike Autoconf, Meson must be present on the system
> of whoever builds QEMU, which is why the last bullet above matters.
> However, it is possible to keep a configure script, if only to provide
> backwards command line compatibility and support a bundled copy of Meson.

Yes, we will probably have a transition period where both build-sys
have to co-exist (same story with libslirp, which I proposed as
meson-only first)

>
> Of course a full conversion of QEMU's build system, including the 7k
> lines configure script is not easy.  In addition, even though Meson

I don't think a complete transition in one go is feasible. Could we
start with some basic things?

For example, we could quite easily start converting tools (qemu-img
etc), some tests...

> supports extension modules that are written in Python, those are really
> used only for namespacing purposes as the extensions need to live in the

I didn't know, I suppose meson doesn't want to support an internal API?

> Meson tree and not in the QEMU tree.  Therefore I have tried to identify
> the parts of QEMU's Makefiles that could complicate the transition, and
> see in advance if they can be addressed:
>
> - --enable-modules's usage of special linker options -u SYMBOL.  For
> this I initially wrote patches to Meson that would make everything
> automatic, however they were rejected.  Instead, the Meson maintainer
> prodded me to find a way to implement the behavior in QEMU's build
> system without having to extend Meson, and we managed to find one.  This
> was a good exercise because it showed that, despite being very
> "opinionated", Meson manages to be pretty flexible too.
>
> - ease of use for test logs and the ability to cut and paste test
> invocations from the logs to the command line.  For this I have started
> "probing" how the Meson developers feel about this kind of change[1],
> and intend to follow up until the meson test driver is comparable in
> usability to QEMU's "make check",
>
> - ease of converting Makefile.objs files.  The Makefile.objs files are
> very nice to change for simple modifications, and any replacement should
> have the same feature.  This will require a Meson extension which I have
> proposed already[2], where adding a source file will look like
>
> obj.add(['CONFIG_VIRTIO'], files('virtio.c'),
> if_false: files('virtio-stub.c'))
> common_obj.add(['CONFIG_SDL'], files('sdl.c'],
>dependencies: sdl)
>
> This is a little more verbose than Makefile.objs, but should be okay
> assuming the extension is accepted.  Rejection of my extension would be
> a blocker though, because QEMU is quite special in how it builds dozens
> of large binaries, all *from the same sources* (unlike e.g. gstreamer
> which has >100 build targets but they are all independent plugins).
>
> Rules for generators like trace-tool or QAPI would becomes quite a bit
> more verbose.  Hopefully this is offset by increased clarity if we don't
> rely anymore on stuff like pattern rules and instead have foreach loops.
>
> - unintended performance issues.  I wouldn't be surprised if QEMU
> stressed Meson more than many other projects did.  However, this should
> be mitigated by the fact that source selection is done in Python rather
> than in Meson's DSL.  Everything else is O(#binaries) or O(#sources),
> but not O(#sources*#binaries).
>
> - ability to use the currently in progress Kconfig declarations for
> dependencies.  Meson developers have expressed that they would accept an
> extension module that loads Kconfig-like declarations from a file; this
> would also be useful to start using Meson only as a Make replacement
> while keeping the current configure script[4], since we could slurp

(you meant [3])

> config-host.mak and config-ta

[Qemu-devel] converting build system to Meson?

[Paolo Bonzini]

Hi all,

lately I have been thinking of converting the QEMU build system to
Meson.  Meson is a relatively new build system that can replace
Autotools or hand-written Makefiles such as QEMU; as a die-hard
Autotools fan, I must say that Meson is by far better than anything else
that has ever tried to replace Autotools, and actually has the potential
to do so.

Advantages of Meson that directly matter for QEMU include:

- build definitions in a very readable and user friendly DSL, which
supports looping and conditions.

- ability to introspect the build definitions so that you can find out
what is built without building it (the DSL is not Turing complete and
most objects in it are immutable, so it cannot be abused that much :))

- support for a non-recursive build from per-subdirectory input (similar
to Makefile.objs)

- ease of distributing a full copy of Meson to support distros that ship
an older version (no dependencies apart from Python 3.5).  At 4
lines of Python, Meson is relatively small.


Unlike Autoconf, meson.build files include both feature detection and
build rules.  Also unlike Autoconf, Meson must be present on the system
of whoever builds QEMU, which is why the last bullet above matters.
However, it is possible to keep a configure script, if only to provide
backwards command line compatibility and support a bundled copy of Meson.

Of course a full conversion of QEMU's build system, including the 7k
lines configure script is not easy.  In addition, even though Meson
supports extension modules that are written in Python, those are really
used only for namespacing purposes as the extensions need to live in the
Meson tree and not in the QEMU tree.  Therefore I have tried to identify
the parts of QEMU's Makefiles that could complicate the transition, and
see in advance if they can be addressed:

- --enable-modules's usage of special linker options -u SYMBOL.  For
this I initially wrote patches to Meson that would make everything
automatic, however they were rejected.  Instead, the Meson maintainer
prodded me to find a way to implement the behavior in QEMU's build
system without having to extend Meson, and we managed to find one.  This
was a good exercise because it showed that, despite being very
"opinionated", Meson manages to be pretty flexible too.

- ease of use for test logs and the ability to cut and paste test
invocations from the logs to the command line.  For this I have started
"probing" how the Meson developers feel about this kind of change[1],
and intend to follow up until the meson test driver is comparable in
usability to QEMU's "make check",

- ease of converting Makefile.objs files.  The Makefile.objs files are
very nice to change for simple modifications, and any replacement should
have the same feature.  This will require a Meson extension which I have
proposed already[2], where adding a source file will look like

obj.add(['CONFIG_VIRTIO'], files('virtio.c'),
if_false: files('virtio-stub.c'))
common_obj.add(['CONFIG_SDL'], files('sdl.c'],
   dependencies: sdl)

This is a little more verbose than Makefile.objs, but should be okay
assuming the extension is accepted.  Rejection of my extension would be
a blocker though, because QEMU is quite special in how it builds dozens
of large binaries, all *from the same sources* (unlike e.g. gstreamer
which has >100 build targets but they are all independent plugins).

Rules for generators like trace-tool or QAPI would becomes quite a bit
more verbose.  Hopefully this is offset by increased clarity if we don't
rely anymore on stuff like pattern rules and instead have foreach loops.

- unintended performance issues.  I wouldn't be surprised if QEMU
stressed Meson more than many other projects did.  However, this should
be mitigated by the fact that source selection is done in Python rather
than in Meson's DSL.  Everything else is O(#binaries) or O(#sources),
but not O(#sources*#binaries).

- ability to use the currently in progress Kconfig declarations for
dependencies.  Meson developers have expressed that they would accept an
extension module that loads Kconfig-like declarations from a file; this
would also be useful to start using Meson only as a Make replacement
while keeping the current configure script[4], since we could slurp
config-host.mak and config-target.mak from Meson.  However, this is only
half of the story, since we also have to compute the Kconfig
dependencies.  For this it should be possible to take the existing
minikconf and change it to produce Meson's DSL.  For example

   config FOO
 default y if BAR
 select BAZ

would become

   if config.get('BAR') == 'y' and not config.has('FOO') then
 config.set('FOO', 'y')
   endif
   if config.get('FOO') == 'y' then
 if config.has('BAZ') and config.get('BAZ') != 'y' then
   error('FOO selects BAZ, but BAZ is disabled')
 endif
 config.set('BAZ', 'y')
   endif

I have not tried doing this, but it seems feasible.

-