Re: D support for the Meson build system
Btw, to make Meson and other build systems work really well, we would need this bug fixed in DMDFE: https://issues.dlang.org/show_bug.cgi?id=16746 At the moment, one needs to ninja clean way too often to get a good build.
Re: D support for the Meson build system
This looks very nice, but I'm having trouble getting it to work on Windows with DMD. What static linker are you using? DmdDCompiler is defined to need one, but the code in detect_static_linker expects the linker to support the "--version" command-line parameter (or "/?" if the linker is named "lib" or "lib.exe"). DMD's "lib.exe" doesn't support "--version" or "/?". I suspect this is what I should be pointing it to, but the term "static linker" confuses me - maybe I want to be pointing it to "link.exe". DMD's "link.exe" doesn't support "--version" but does support "/?". I can hack around this, but I'm wondering why I'm alone in encountering this. Thanks!
Re: D support for the Meson build system
On 04/11/2017 09:44 AM, Atila Neves wrote: > That's correct - __traits(getUnitTests) is broken unless compiling all > at once which is extremely unfortunate. I've filed a dmd bug already. > It's been on my TODO list for a while to fix it myself. [Issue 16995 – __traits(getUnittests) doesn't work with separate compilation](https://issues.dlang.org/show_bug.cgi?id=16995) Somewhat similar to [Issue 14894 – mangling of mixins and lambdas is not unique and depends on compilation flags](https://issues.dlang.org/show_bug.cgi?id=14894) Backlog Card: https://trello.com/c/YuW4JycJ/36-mangling-of-mixins-and-lambdas-can-change-w-unittest-or-other-versions
Re: D support for the Meson build system
On Tuesday, 11 April 2017 at 07:34:58 UTC, Russel Winder wrote: On Mon, 2017-04-10 at 12:41 +, Matthias Klumpp via Digitalmars-d- announce wrote: […] I am not buying the necessity of not-splitbuilding for optimizations yet. If that would be the case, how do optimizations work with projects using GCC/Clang where splitbuilding is the default and often only option (like Mesa, Linux, lots of scientific stuff). […] It seems that unit-threaded cannot be built module at a time due to various reflection bits, the only solution is all source at once. That's correct - __traits(getUnitTests) is broken unless compiling all at once which is extremely unfortunate. I've filed a dmd bug already. It's been on my TODO list for a while to fix it myself. Atila
Re: D support for the Meson build system
On Mon, 2017-04-10 at 12:41 +, Matthias Klumpp via Digitalmars-d- announce wrote: > […] > I am not buying the necessity of not-splitbuilding for > optimizations yet. If that would be the case, how do > optimizations work with projects using GCC/Clang where > splitbuilding is the default and often only option (like Mesa, > Linux, lots of scientific stuff). […] It seems that unit-threaded cannot be built module at a time due to various reflection bits, the only solution is all source at once. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: D support for the Meson build system
On 04/10/2017 02:41 PM, Matthias Klumpp wrote: > On Monday, 10 April 2017 at 12:10:41 UTC, Russel Winder wrote: >> On Mon, 2017-04-10 at 08:39 +, Atila Neves via Digitalmars-d- >> announce wrote: >>> […] >>> >>> As far as I know the only build system that does this by default for >>> D is reggae. >> >> I will be adding a new builder to the SCons D tools to do whole source >> and per package compiling – to add to the module at a time compiling. >> It would be good to add this for CMake-D and the D support in Meson. > > I am not buying the necessity of not-splitbuilding for optimizations > yet. If that would be the case, how do optimizations work with projects > using GCC/Clang where splitbuilding is the default and often only option > (like Mesa, Linux, lots of scientific stuff). The argument is not so much about optimization (which can be obtained via LTO as well), but primarily about build speed. Turns out that recompiling 10 modules at once (a whole package) is often faster than doing 3 single-module compilations when each of the 3 modules imports the same 4 modules. 10 < 3 + 3 * 4 Sure the latter is parallelizable, but a lot of computation is wasted repeating the same work. dmd has another build mode `-c` with multiple source files producing multiple object files, but that mode is fundamentally broken, b/c template instances are only emitted to one of the object files, leading to funny linker errors when rebuilding only some modules. We kinda know how to fix this (without resorting to `-allinst`), but it's a lot of work, and splitting into subpackages simply works, is fast, and can be parallelized as well. -Martin
Re: D support for the Meson build system
On Mon, Apr 10, 2017 at 05:56:38PM +, Matthias Klumpp via Digitalmars-d-announce wrote: > On Monday, 10 April 2017 at 15:27:25 UTC, Russel Winder wrote: [...] > > My thought for SCons was to delegate the package fetching to Dub as > > a subprocess or write some Python to use the Dub API. I'm not clear > > on that as yet, the issue is whether the Dub local source repo is > > the right way forward – using Dub for preparing the compiled > > artefact is likely not the right way forward for SCons. This would > > then make it easy to do something for Rust/Cargo – except that SCons > > doesn't really support Rust yet, and with Cargo are there any Rust > > users not using Cargo. > > > > Having said all this SCons stuff, if there was Meson support for > > this "get the source from the Dub repository, compile it and make it > > a dependency" I'd likely stay with Meson for my codes. > > SCons is considered evil, last time I checked ^^ => > https://wiki.debian.org/UpstreamGuide#line867 > (unless it's used right, which seems to be hard) - I have no idea > though on whether the issues with it were fixed, the entry on SCons > hasn't been updated in a while. I use SCons exclusively for my D projects, and haven't faced any major problems. I understand, however, that from a distro's POV, it can be annoying to work with if you're not familiar with how to patch SConstruct scripts to do what is needed. It's true that unless upstream explicitly supports installation targets, SCons won't do it for you. I do actually maintain a Debian package that uses SCons (well, written by yours truly, so perhaps that's cheating), and I did find that I have to write rules explicitly for supporting installation targets just so it will work properly with Debian's packaging scripts. As for SONAME support (mentioned by the wiki), I'm not sure what the big deal is... isn't it just a matter of passing the right linker flags to the compile command? So either adding something to CFLAGS or LDFLAGS in the construction environment should do the trick. SCons refusing to pick up settings from the user's environment can be annoying to distros, but in return for that, it gives you 100% reproducible (incremental!) builds by default, whereas most Makefiles require you to `make clean` just to be sure, to the point that it has become accepted practice to always "build from clean" because otherwise you just never know. Even Debian's packaging scripts have a `clean` target because of this. But in an SCons-based package, the `clean` target is a one-liner `scons -c`, since SCons knows what targets it has produced and practically guarantees you're back to a clean slate. Even though that's actually unnecessary to produce a good build! (Ironically, leaving the `clean` rule blank causes the packaging scripts to complain because they notice stray files lying around, obviously a feature written with a Makefile mindset because stray files are problematic in Makefile-based projects, but hardly worth attention in an SCons-based project!) T -- Curiosity kills the cat. Moral: don't be the cat.
Re: D support for the Meson build system
On Monday, 10 April 2017 at 18:11:44 UTC, Russel Winder wrote: [...] I'll look to ensuring my facts are correct, and then find out where to put an issue about this – I am assuming a GitHub repository with issues . Just file one at https://github.com/mesonbuild/meson/issues - it might even be that Meson supports this already, the project is moving so fast it's really hard to keep up with all the changes (Mesa and X investigating using it boosted it's development quite a bit, and it was really fast even before).
Re: D support for the Meson build system
On Mon, 2017-04-10 at 17:56 +, Matthias Klumpp via Digitalmars-d- announce wrote: > […] > > That's pretty cool! One way to do this with Meson is to spawn a > shell script as custom target, but that obviously sucks. It might > be worth reporting this as issue upstream, with a concrete > usecase like this, the Meson maintainers will highly likely add > support for it. > One could also always write a plugin as a last resort. > > > […] I'll look to ensuring my facts are correct, and then find out where to put an issue about this – I am assuming a GitHub repository with issues . -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: D support for the Meson build system
On Mon, 2017-04-10 at 17:56 +, Matthias Klumpp via Digitalmars-d- announce wrote: > […] > SCons is considered evil, last time I checked ^^ => > https://wiki.debian.org/UpstreamGuide#line867 > (unless it's used right, which seems to be hard) - I have no idea > though on whether the issues with it were fixed, the entry on > SCons hasn't been updated in a while. Given that some of the "facts" there are actually wrong, I think we can infer Debian people assume Autotools is all that is needed and are uninterested in new things. SCons has supported sonames for a while, for example. Prejudice in these things, as in all things, tends to get reified and become unalterable fact. This no longer worries me. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: D support for the Meson build system
On Monday, 10 April 2017 at 15:27:25 UTC, Russel Winder wrote: On Mon, 2017-04-10 at 12:41 +, Matthias Klumpp via Digitalmars-d- announce wrote: […]. I am not buying the necessity of not-splitbuilding for optimizations yet. If that would be the case, how do optimizations work with projects using GCC/Clang where splitbuilding is the default and often only option (like Mesa, Linux, lots of scientific stuff). I am investigating this architecture because Chapel code cannot really be compiled separately as far as I can see. cf. http://chapel.cray.com/ Chapel is a PGAS language (like X10) for use with Big Computation™ on serious computers and also any computer. I'm interested as a way of connecting Python visualisation to computation that NumPy can't really handle. That's pretty cool! One way to do this with Meson is to spawn a shell script as custom target, but that obviously sucks. It might be worth reporting this as issue upstream, with a concrete usecase like this, the Meson maintainers will highly likely add support for it. One could also always write a plugin as a last resort. It seems sensible therefore to offer this way of working for D since whether there is actually any optimisation benefit or not some people think there is and use it as a stick to beat you with if it isn't there. Having some level of dub integration is Meson would be neat indeed - maybe one could make a small helper binary Meson can call to fetch things from the dub registry. I wonder though how that would jive with Meson's own subprojects/wrap system. Probably worth investigating. My thought for SCons was to delegate the package fetching to Dub as a subprocess or write some Python to use the Dub API. I'm not clear on that as yet, the issue is whether the Dub local source repo is the right way forward – using Dub for preparing the compiled artefact is likely not the right way forward for SCons. This would then make it easy to do something for Rust/Cargo – except that SCons doesn't really support Rust yet, and with Cargo are there any Rust users not using Cargo. Having said all this SCons stuff, if there was Meson support for this "get the source from the Dub repository, compile it and make it a dependency" I'd likely stay with Meson for my codes. SCons is considered evil, last time I checked ^^ => https://wiki.debian.org/UpstreamGuide#line867 (unless it's used right, which seems to be hard) - I have no idea though on whether the issues with it were fixed, the entry on SCons hasn't been updated in a while.
Re: D support for the Meson build system
On Mon, 2017-04-10 at 12:41 +, Matthias Klumpp via Digitalmars-d- announce wrote: > […]. > > I am not buying the necessity of not-splitbuilding for > optimizations yet. If that would be the case, how do > optimizations work with projects using GCC/Clang where > splitbuilding is the default and often only option (like Mesa, > Linux, lots of scientific stuff). I am investigating this architecture because Chapel code cannot really be compiled separately as far as I can see. cf. http://chapel.cray.com/ Chapel is a PGAS language (like X10) for use with Big Computation™ on serious computers and also any computer. I'm interested as a way of connecting Python visualisation to computation that NumPy can't really handle. It seems sensible therefore to offer this way of working for D since whether there is actually any optimisation benefit or not some people think there is and use it as a stick to beat you with if it isn't there. > Having some level of dub integration is Meson would be neat > indeed - maybe one could make a small helper binary Meson can > call to fetch things from the dub registry. > I wonder though how that would jive with Meson's own > subprojects/wrap system. Probably worth investigating. My thought for SCons was to delegate the package fetching to Dub as a subprocess or write some Python to use the Dub API. I'm not clear on that as yet, the issue is whether the Dub local source repo is the right way forward – using Dub for preparing the compiled artefact is likely not the right way forward for SCons. This would then make it easy to do something for Rust/Cargo – except that SCons doesn't really support Rust yet, and with Cargo are there any Rust users not using Cargo. Having said all this SCons stuff, if there was Meson support for this "get the source from the Dub repository, compile it and make it a dependency" I'd likely stay with Meson for my codes. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: D support for the Meson build system
On Monday, 10 April 2017 at 12:10:41 UTC, Russel Winder wrote: On Mon, 2017-04-10 at 08:39 +, Atila Neves via Digitalmars-d- announce wrote: […] As far as I know the only build system that does this by default for D is reggae. I will be adding a new builder to the SCons D tools to do whole source and per package compiling – to add to the module at a time compiling. It would be good to add this for CMake-D and the D support in Meson. I am not buying the necessity of not-splitbuilding for optimizations yet. If that would be the case, how do optimizations work with projects using GCC/Clang where splitbuilding is the default and often only option (like Mesa, Linux, lots of scientific stuff). Having some level of dub integration is Meson would be neat indeed - maybe one could make a small helper binary Meson can call to fetch things from the dub registry. I wonder though how that would jive with Meson's own subprojects/wrap system. Probably worth investigating.
Re: D support for the Meson build system
On Mon, 2017-04-10 at 08:39 +, Atila Neves via Digitalmars-d- announce wrote: > […] > > As far as I know the only build system that does this by default > for D is reggae. I will be adding a new builder to the SCons D tools to do whole source and per package compiling – to add to the module at a time compiling. It would be good to add this for CMake-D and the D support in Meson. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: D support for the Meson build system
On Saturday, 8 April 2017 at 19:11:35 UTC, Martin Nowak wrote: On Friday, 7 April 2017 at 07:57:02 UTC, kinke wrote: So while compiling each file separately in parallel is potentially much much faster, the produced release binary may be slower due to less/no cross-module inlining (e.g., LDC's option is still experimental and known to have issues). In fact single-module compilation is slower than package compilation most of the time due to the redundant parsing of common imports. As an extreme example, last time I tried, single-module compilation for gtkd was ~10x slower than compiling the library at once. Savings on recompilation hardly make up for this huge overhead. Overall I'd recommend organizing and building subpackages when a project becomes too big. Per package is significantly faster. As far as I know the only build system that does this by default for D is reggae. Atila
Re: D support for the Meson build system
On Sunday, 21 August 2016 at 19:08:59 UTC, Matthias Klumpp wrote: I hope you find this as useful as I do :-) Yes, mature build systems for bigger projects are great. Any opinion about/experience with [Bazel](https://bazel.build/)? https://github.com/mesonbuild/meson/wiki/Comparisons#bazel
Re: D support for the Meson build system
On Friday, 7 April 2017 at 07:57:02 UTC, kinke wrote: So while compiling each file separately in parallel is potentially much much faster, the produced release binary may be slower due to less/no cross-module inlining (e.g., LDC's option is still experimental and known to have issues). In fact single-module compilation is slower than package compilation most of the time due to the redundant parsing of common imports. As an extreme example, last time I tried, single-module compilation for gtkd was ~10x slower than compiling the library at once. Savings on recompilation hardly make up for this huge overhead. Overall I'd recommend organizing and building subpackages when a project becomes too big.
Re: D support for the Meson build system
On Sunday, 21 August 2016 at 19:08:59 UTC, Matthias Klumpp wrote: Hi! Last week I was at this year's GUADEC conference and listened to a very interesting talk on the Meson build system[2] which is designed for very fast builds and as a much more modern replacement for Automake with a simple syntax. In the past few days I added support for D (all three major compilers) to the Meson build system, with some great results- [...] Seems like good news for us developing Linux apps.
Re: D support for the Meson build system
On Monday, 22 August 2016 at 11:45:37 UTC, Matthias Klumpp wrote: One important thing about Ninja is that it will perform split-builds by default, so if you change something, only the changed piece needs to be recompiled. I *think* dub can do that too, but for some reason it never does it (even when using --parallel). That behavior drastically reduces compile times when working on a project. So if I understand correctly, Meson/Ninja compile each module separately, while dub builds them all in a single command line. This makes quite a big difference. Compiling all at once requires a full rebuild every time a single file is modified and implies full single-threadedness. The advantages are that each file is only loaded and parsed once, but much more importantly, everything is emitted into a single object file, allowing for implicit full cross-module inlining (across the compiled modules). So while compiling each file separately in parallel is potentially much much faster, the produced release binary may be slower due to less/no cross-module inlining (e.g., LDC's option is still experimental and known to have issues).
Re: D support for the Meson build system
On Sunday, 21 August 2016 at 19:08:59 UTC, Matthias Klumpp wrote: […] In the past few days I added support for D (all three major compilers) to the Meson build system, with some great results- […] I hope you find this as useful as I do :-) […] Yes, I do , it's great. Currently though there is apparently no integration between Meson and the Dub repository, i.e. it is not possible to get dependencies via Dub as purely a dependency manager and nothing to do with actual build.
Re: D support for the Meson build system
On Monday, 22 August 2016 at 11:45:37 UTC, Matthias Klumpp wrote: On Monday, 22 August 2016 at 01:34:36 UTC, jkpl wrote: On Sunday, 21 August 2016 at 19:08:59 UTC, Matthias Klumpp wrote: for a project like Terminix, dub with LDC builds in 8.6s, while Meson and ninja take only 6s here. Did you try to build with DUB but with WIFI or ethernet interface toggled off ? I ran these a few times, and the results were comparable, although I think if I would perform proper statistics the variance would be relatively high. Compiler was LDC 1.0.0 (on a Xeon E3-1231 v3). Sorry I was thinking to something, that could have slowed down the DUB build, but this was obviouslt wrong.
Re: D support for the Meson build system
On Monday, 22 August 2016 at 01:34:36 UTC, jkpl wrote: On Sunday, 21 August 2016 at 19:08:59 UTC, Matthias Klumpp wrote: for a project like Terminix, dub with LDC builds in 8.6s, while Meson and ninja take only 6s here. Did you try to build with DUB but with WIFI or ethernet interface toggled off ? No, but I obviously excluded any download times, and also dependency build-times to be fair to dub. But, with network access toggled off: Terminix: debug build: dub: 0m8.60s ninja: 0m6.36s release build (standard args): dub: 0m12.923s ninja: 0m11.740s asgen: debug build: dub: 0m13.00s ninja: 0m13.04s release build: dub: 0m25.72s ninja: 0m19.19s I ran these a few times, and the results were comparable, although I think if I would perform proper statistics the variance would be relatively high. Compiler was LDC 1.0.0 (on a Xeon E3-1231 v3). One important thing about Ninja is that it will perform split-builds by default, so if you change something, only the changed piece needs to be recompiled. I *think* dub can do that too, but for some reason it never does it (even when using --parallel). That behavior drastically reduces compile times when working on a project. Also, Ninja knows about all the source files and generated files in advance, while dub needs to search for them on every run. That of course also has the drawback that one needs to specify the source files prior to building in the meson.build file.
Re: D support for the Meson build system
On Sunday, 21 August 2016 at 19:08:59 UTC, Matthias Klumpp wrote: for a project like Terminix, dub with LDC builds in 8.6s, while Meson and ninja take only 6s here. Did you try to build with DUB but with WIFI or ethernet interface toggled off ?
