Re: reggae v0.5.0: new features in the D meta-build system
On Saturday, 26 September 2015 at 03:51:11 UTC, Jason White wrote: I rarely visit the D forums and even more rarely make a post, but this thread caught my eye. I've been writing a build system in D too: https://github.com/jasonwhite/brilliant-build (I'm not very fond of the name. Naming is hard!) It is a general build system with an emphasis on correctness. It is a work in progress at this point, but I'm very happy with how it is turning out. I'm interested in your guys' thoughts on it. Looks good as a work-in-progress. A lot of the same ideas I used, including scripting languages as a way to describe builds. I'm probably going to use reggae though ;) Atila
Re: reggae v0.5.0: new features in the D meta-build system
On Saturday, 26 September 2015 at 08:23:46 UTC, Andy Smith wrote: HI Atilla, Dub's looking interesting! Some of the links are broken when browsing from code.dlang.org though. From http://code.dlang.org/packages/reggae click on, say, 'detailed documentation'. ( Where I wanted to go!). Get .. 404 - Not Found Not Found Internal error information: No routes match path '/packages/doc/index.md' Not sure if that's problem with reggae's layout of the site Cheers, A. ^dub^reggae^ :-) Doh! - morning coffee hasn't kicked in yet! Cheers, A.
Re: reggae v0.5.0: new features in the D meta-build system
On 2015-09-26 01:53, anonymous wrote: Compiling one file at a time is yet another thing. 1) Compile everything into one object file: dmd -c -ofresult.o foo.d bar.d 2) Compile to multiple object files in one run: dmd -c foo.d bar.d 3) Compile to multiple object files in multiple runs: dmd -c foo.d; dmd -c bar.d Since you didn't know about 2, I take it that you measured 1 vs 3. 3 is bound to be slower when the modules share dependencies, because they have to be parsed again and again. 2 may be faster or slower than 1, I don't know, a quick check didn't show a difference. When the template instantiation issue ever gets fixed, I wouldn't be surprised if 2 got slower than 1. Of course, any slowness of 2 must then be weighed against 1 compiling more than necessary. In theory 2 should be faster when it comes to recompiling because fewer files need to be recompiled. -- /Jacob Carlborg
Re: reggae v0.5.0: new features in the D meta-build system
HI Atilla, Dub's looking interesting! Some of the links are broken when browsing from code.dlang.org though. From http://code.dlang.org/packages/reggae click on, say, 'detailed documentation'. ( Where I wanted to go!). Get .. 404 - Not Found Not Found Internal error information: No routes match path '/packages/doc/index.md' Not sure if that's problem with reggae's layout of the site Cheers, A.
Re: reggae v0.5.0: new features in the D meta-build system
Am 26.09.2015 um 10:23 schrieb Andy Smith: HI Atilla, Dub's looking interesting! Some of the links are broken when browsing from code.dlang.org though. From http://code.dlang.org/packages/reggae click on, say, 'detailed documentation'. ( Where I wanted to go!). Get .. 404 - Not Found Not Found Internal error information: No routes match path '/packages/doc/index.md' Not sure if that's problem with reggae's layout of the site Cheers, A. That was due to a missing feature (special handling for relative links) of code.dlang.org. Fixed now.
Re: reggae v0.5.0: new features in the D meta-build system
On Friday, 25 September 2015 at 12:09:01 UTC, Jacob Carlborg wrote: On 2015-09-24 16:46, Atila Neves wrote: That's not been my experience at all using reggae. I only do incremental builds now and have never run into a problem. Can you give an example? Here's one old post [1] that describes the problem. I'm not sure how much of it still applies today. [1] http://forum.dlang.org/thread/h8ddc5$1h67$1...@digitalmars.com I wasn't around in 2009 so I don't how dmd was then. How does one compile 3 files "at the same time" and generate 3 object files? There was a reference to a -multiobj option in that post but that's not even in the man page. Even if it exists and works as I assume it does, it seems to me to be a silly way to compile files. As Andrei mentioned at DConf this year, the D idiom is to group source files per package when compiling. That's what reggae does by default for D files. It just works. Atila
Re: reggae v0.5.0: new features in the D meta-build system
On Thursday, 24 September 2015 at 15:04:49 UTC, Atila Neves wrote: I you want any advice on this matter please contact me. I'd be glad to be of service. Send me an email, I'm more than happy to waffle away about build systems. BTW. I'm planning on visiting Berlin for DConf 2016. We could have a chat there.
Re: reggae v0.5.0: new features in the D meta-build system
On Friday 25 September 2015 23:27, Atila Neves wrote: > How does one compile 3 files "at the same time" and generate 3 > object files? There was a reference to a -multiobj option in that > post but that's not even in the man page. dmd -c foo.d bar.d baz.d rdmd would probably do this by now, if it weren't for the template instantiation issue. There's a nice, simple example of the issue in the PR discussion: https://github.com/D-Programming-Language/tools/pull/170#issuecomment-112526734 > Even if it exists and works as I assume it does, it seems to me > to be a silly way to compile files. As Andrei mentioned at DConf > this year, the D idiom is to group source files per package when > compiling. That's what reggae does by default for D files. It > just works. Aside from the template instantiation issue, I don't see how doing a package at a time is better than compiling exactly those files that need to be compiled.
Re: reggae v0.5.0: new features in the D meta-build system
On Friday, 25 September 2015 at 22:12:49 UTC, anonymous wrote: On Friday 25 September 2015 23:27, Atila Neves wrote: How does one compile 3 files "at the same time" and generate 3 object files? There was a reference to a -multiobj option in that post but that's not even in the man page. dmd -c foo.d bar.d baz.d Huh, I didn't know that. I guess I've always used the `-of` option. Unsurprisingly, so does reggae. Which means that incremental compilation is no longer an issue. Aside from the template instantiation issue, I don't see how doing a package at a time is better than compiling exactly those files that need to be compiled. There have been threads about this before. It turns out that compiling per file is usually slower than compiling the whole package/app at once. It's not intuitive, but it's true (I measured it myself). reggae has an option to compile per-file but I haven't used it since switching to per-package once. Atila
Re: reggae v0.5.0: new features in the D meta-build system
On Saturday 26 September 2015 01:24, Atila Neves wrote: > There have been threads about this before. It turns out that > compiling per file is usually slower than compiling the whole > package/app at once. It's not intuitive, but it's true (I > measured it myself). reggae has an option to compile per-file but > I haven't used it since switching to per-package once. Compiling one file at a time is yet another thing. 1) Compile everything into one object file: dmd -c -ofresult.o foo.d bar.d 2) Compile to multiple object files in one run: dmd -c foo.d bar.d 3) Compile to multiple object files in multiple runs: dmd -c foo.d; dmd -c bar.d Since you didn't know about 2, I take it that you measured 1 vs 3. 3 is bound to be slower when the modules share dependencies, because they have to be parsed again and again. 2 may be faster or slower than 1, I don't know, a quick check didn't show a difference. When the template instantiation issue ever gets fixed, I wouldn't be surprised if 2 got slower than 1. Of course, any slowness of 2 must then be weighed against 1 compiling more than necessary.
Re: reggae v0.5.0: new features in the D meta-build system
I rarely visit the D forums and even more rarely make a post, but this thread caught my eye. I've been writing a build system in D too: https://github.com/jasonwhite/brilliant-build (I'm not very fond of the name. Naming is hard!) It is a general build system with an emphasis on correctness. It is a work in progress at this point, but I'm very happy with how it is turning out. I'm interested in your guys' thoughts on it.
Re: reggae v0.5.0: new features in the D meta-build system
On 2015-09-24 16:46, Atila Neves wrote: That's not been my experience at all using reggae. I only do incremental builds now and have never run into a problem. Can you give an example? Here's one old post [1] that describes the problem. I'm not sure how much of it still applies today. [1] http://forum.dlang.org/thread/h8ddc5$1h67$1...@digitalmars.com -- /Jacob Carlborg
Re: reggae v0.5.0: new features in the D meta-build system
On Wednesday, 23 September 2015 at 06:16:08 UTC, Jacob Carlborg wrote: On 2015-09-22 14:39, Per Nordlöw wrote: SCons has a very hidden feature called interactive mode via `--interactive` that supports instantaenous incremental builds via a very primitive CLI that basically supports to commands: Incremental builds in D are currently not reliable. Something about templates not being outputted to all object files it need to. That's not been my experience at all using reggae. I only do incremental builds now and have never run into a problem. Can you give an example? Atila
Re: reggae v0.5.0: new features in the D meta-build system
On Tuesday, 22 September 2015 at 12:39:48 UTC, Per Nordlöw wrote: On Wednesday, 16 September 2015 at 14:07:17 UTC, Atila Neves wrote: http://code.dlang.org/my_packages/reggae What's new: Atila If you want to build a really revolutionary *new* build system you should turn reggae into a client-server-architecture that listens to file moves/copies/notifications via inotify on Linux. I'm not sure how the develop/builder interface would look like then, however. A web-client would be one way. That idea is in the TODO.txt of the project. There's a 99.9% chance I won't bother. I just benchmarked a toy auto-generated project of 30'000 D files, because I wanted to profile and optimise the binary backend against ninja and tup. I'm doing this by measuring the overhead of checking all of these files for modification timestamps, since actually compiling and linking take much longer. The overhead for ninja is 0.18s. Tup does it even faster. My current binary backend spends all of 0.3s, and that's what I was trying to optimize. Make, however... 22s I think. It would seem obvious that using something like inotify would be incredibly fast, but in practice it's irrelevant. Well, it's irrelevant for pretty much every project out there. I'm incredibly sensible to how long my feedback loops are, but if a 30k file project takes under a second to check, I really don't care. Now, you may say that my toy project doesn't accurately represent real life. But I've used ninja on a real C codebase of 40M lines spread over ~14k files. This is much larger than most people will ever deal with. A no-op build took 1s. Make is slow. Just don't use make. The ninja, tup and binary backends of reggae are fast enough. Incidentally, this is why ninja didn't try to get faster. It doesn't need to. I might consider a system where you don't even have to tell the computer to build anything. I'm not even sure that's that useful. Maybe it's just the way I work, I don't know. This however does not rebuild the dependency tree between invokations which is a serious limitation. I believe reggae could be even smarter here. It already is ;) I've spent the last two years designing and implementing a SCons-based build system at work. Involving, among others, adding support for Ada and auto-detection of include-paths as you have done in reggae. Feel free to issue a PR for Ada auto-detection for reggae. :) I you want any advice on this matter please contact me. I'd be glad to be of service. Send me an email, I'm more than happy to waffle away about build systems.
Re: reggae v0.5.0: new features in the D meta-build system
On 2015-09-23 08:41, NVolcz wrote: I heard that SBT does something similar (http://www.se-radio.net/2015/07/se-radio-episode-231-joshua-suereth-and-matthew-farwell-on-sbt-and-software-builds/). From what I understand it is faster due to that you can skip the overhead of startup. I also presume it is a big win for user experience. You can do both with SBT. But I though the main reason was because of the JVM startup time. -- /Jacob Carlborg
Re: reggae v0.5.0: new features in the D meta-build system
On Tuesday, 22 September 2015 at 13:22:25 UTC, Joakim wrote: Like ekam? https://github.com/sandstorm-io/ekam Sounds very promising! We talked about it when I interviewed Atila: http://arsdnet.net/this-week-in-d/sep-06.html Thanks!
Re: reggae v0.5.0: new features in the D meta-build system
On 2015-09-22 14:39, Per Nordlöw wrote: SCons has a very hidden feature called interactive mode via `--interactive` that supports instantaenous incremental builds via a very primitive CLI that basically supports to commands: Incremental builds in D are currently not reliable. Something about templates not being outputted to all object files it need to. -- /Jacob Carlborg
Re: reggae v0.5.0: new features in the D meta-build system
On Tuesday, 22 September 2015 at 12:39:48 UTC, Per Nordlöw wrote: On Wednesday, 16 September 2015 at 14:07:17 UTC, Atila Neves wrote: http://code.dlang.org/my_packages/reggae What's new: Atila If you want to build a really revolutionary *new* build system you should turn reggae into a client-server-architecture that listens to file moves/copies/notifications via inotify on Linux. I'm not sure how the develop/builder interface would look like then, however. A web-client would be one way. Like ekam? https://github.com/sandstorm-io/ekam We talked about it when I interviewed Atila: http://arsdnet.net/this-week-in-d/sep-06.html
Re: reggae v0.5.0: new features in the D meta-build system
On Wednesday, 16 September 2015 at 14:07:17 UTC, Atila Neves wrote: http://code.dlang.org/my_packages/reggae What's new: Atila If you want to build a really revolutionary *new* build system you should turn reggae into a client-server-architecture that listens to file moves/copies/notifications via inotify on Linux. I'm not sure how the develop/builder interface would look like then, however. A web-client would be one way. SCons has a very hidden feature called interactive mode via `--interactive` that supports instantaenous incremental builds via a very primitive CLI that basically supports to commands: - build/b TARGET - clean/c TARGET This however does not rebuild the dependency tree between invokations which is a serious limitation. I believe reggae could be even smarter here. I've spent the last two years designing and implementing a SCons-based build system at work. Involving, among others, adding support for Ada and auto-detection of include-paths as you have done in reggae. I you want any advice on this matter please contact me. I'd be glad to be of service. Good luck!
reggae v0.5.0: new features in the D meta-build system
http://code.dlang.org/my_packages/reggae What's new: . API changes: main high-level rules are now called objectFiles, link, and scriptlike . Optional top-level targets: aren't built by default but can be built on request . Phony targets . staticLibrary rule that does what it says . unityBuild rule for C/C++ projects that builds a binary as one translation unit to speed up compilation . Support for writing build descriptions in Python, Ruby, Lua and JavaScript That last one might cause scratchy head syndrome. Why would I want to enable writing builds in a language other than D? Here's why: 1. Interpreting a build description is faster than compiling, linking and running it. Not a big deal since the build description isn't read often, but it's true 2. No confusion about when to use `enum` and when to use `alias`. Again, probably wasn't catching anybody unawares but also true 3. Not having to install a D compiler in order to generate builds - I ran into this issue at work the other day trying to use reggae on a weird VM with no root access 4. More importantly... to not limit reggae to only D developers There are quite a few build systems / tools out there in which different languages are used. Almost all of them are only used by developers of their language niche: Ruby/Rake, Python/SCons/Waf, Groovy/Java/Gradle, Haskell/Shake, etc. This is my attempt to make reggae appeal to a wider audience. I wanted to add a Lisp too but then I'd have to pick a dialect and an implemetation, only to annoy fans of other dialects. I might just do Emacs Lisp to annoy everyone equally! Also, for the lulz. Atila
Re: reggae v0.5.0: new features in the D meta-build system
This link should work for everyone: http://code.dlang.org/packages/reggae (I never tried reggae. Maybe I should, it looks good.) LMB On Wed, Sep 16, 2015 at 11:07 AM, Atila Neves via Digitalmars-d-announce < digitalmars-d-announce@puremagic.com> wrote: > http://code.dlang.org/my_packages/reggae > > What's new: > . API changes: main high-level rules are now called objectFiles, link, and > scriptlike > . Optional top-level targets: aren't built by default but can be built on > request > . Phony targets > . staticLibrary rule that does what it says > . unityBuild rule for C/C++ projects that builds a binary as one > translation unit to speed up compilation > . Support for writing build descriptions in Python, Ruby, Lua and > JavaScript > > That last one might cause scratchy head syndrome. Why would I want to > enable writing builds in a language other than D? Here's why: > > 1. Interpreting a build description is faster than compiling, linking and > running it. Not a big deal since the build description isn't read often, > but it's true > 2. No confusion about when to use `enum` and when to use `alias`. Again, > probably wasn't catching anybody unawares but also true > 3. Not having to install a D compiler in order to generate builds - I ran > into this issue at work the other day trying to use reggae on a weird VM > with no root access > 4. More importantly... to not limit reggae to only D developers > > There are quite a few build systems / tools out there in which different > languages are used. Almost all of them are only used by developers of their > language niche: Ruby/Rake, Python/SCons/Waf, Groovy/Java/Gradle, > Haskell/Shake, etc. This is my attempt to make reggae appeal to a wider > audience. > > I wanted to add a Lisp too but then I'd have to pick a dialect and an > implemetation, only to annoy fans of other dialects. I might just do Emacs > Lisp to annoy everyone equally! Also, for the lulz. > > Atila >
Re: reggae v0.5.0: new features in the D meta-build system
Ruby link points to reggae-python.
