Re: Moving away from make
Hi, On Sat, Oct 01, 2011 at 01:35:02PM +0100, Graham Percival wrote: I found some info on creating loops in gnu make, but it didn't seem possible to have loops in pure bsd make. Well, why would it need to work with BSD Make? Gmake is available as an optional tool pretty much everywhere -- and as you considered cmake, a dependency an extra build tool is apparently not a problem... -antrik- ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: website.make (was: Moving away from make)
On Sun, Oct 02, 2011 at 02:50:12PM +0200, Julien Rioux wrote: I had a look at website.make, and it strikes me as a shell script written in make. That's quite a fair assesement. You of course have loops in make, attached you find a short rewrite of that particular snippet. I can probably reduce it to even fewer targets and make it simpler. If you want me to, I can look at the whole file to makeify it. Looks nice, and it seems to work, but I'm not certain that I follow all the steps. More comments? :) Specifics: - line 107 has an extra trailing ) - comments like lines 105 and 109 should probably read get xrefs for english te?? manuals - I'm not certain if web is a little bit special (line 119) or not. My initial instinct is that it's not special in this context. Cheers, - Graham ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
Hi, On Sun, Sep 25, 2011 at 07:01:04PM +0200, Karl Hammar wrote: Instead I've made two scripts depend_ly and depen_tex [1] which finds out what depends on what (think gcc -M), and make [2] takes care of the rest. I think that's precisely the right thing to do :-) Would it be good to make lilypond print out a files dependancies like gcc? Sounds like a good idea as well. No need to maintain external parsers, when lilypond itself knows best what the dependencies are... -antrik- ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
Hi, On Sun, Sep 25, 2011 at 03:12:14AM +0100, Graham Percival wrote: I will admit there is one aspect in which I *am* spoiled, though: I am totally spoiled by python's readable code. I am so accustomed to writing stuff like cmd = compiler + ' -o ' + exe_name + src_files or cmd = %(compiler)s -o %(exe_name)s %(src_files) % locals() that I find stuff like $(CC) -o $@ $ silly. The readability for casual contributors -- which is what most people looking at build system stuff are -- is ridiculously better in python than anything else. It might be true that Python is more readable for newcomers than make (though reading your examples, I'm not at all convinced of that...) -- but how much does that really matter? The reason casual contributors have trouble making build system modifications, is not the syntax (which is really easy to learn IMHO), but the fact that build systems are inherently a very complex matter. (As you observed yourself...) The best syntax imaginable won't do anything to change this. -antrik- ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
On Sat, Oct 01, 2011 at 02:12:44AM +0200, olafbuddenha...@gmx.net wrote:
It might be true that Python is more readable for newcomers than make
(though reading your examples, I'm not at all convinced of that...) --
but how much does that really matter?
Many people already have some familiarity with python --
particularly in lilypond, where a good chunk of our code is in
python. This lessens the burden of understanding build-system
stuff.
The reason casual contributors
have trouble making build system modifications, is not the syntax (which
is really easy to learn IMHO), but the fact that build systems are
inherently a very complex matter. (As you observed yourself...) The best
syntax imaginable won't do anything to change this.
The benefit of python is that you can leverage existing python
knowledge, as well as using niceties like loops and functions.
For example, take a look at make/website.make. I spent somewhere
on the order of 50 hours working on the new website build system
(including other scripts, not just that file). I found some info
on creating loops in gnu make, but it didn't seem possible to have
loops in pure bsd make. So I resorted to bash, i.e.:
website-xrefs: website-version
for l in '' $(WEB_LANGS); do \
len=$${#l} ; \
r=$$l; \
if [ $$len -gt 0 ] ; then \
r=$$r/; \
fi ; \
$(EXTRACT_TEXI_FILENAMES) \
-I $(top-src-dir)/Documentation \
-I $(top-src-dir)/Documentation/$$l \
-I $(OUT) -o $(OUT) --split=node \
--known-missing-files=$(top-src-dir)/scripts/build/website-known-missing-files.txt
\
$(quiet-flag) \
$(top-src-dir)/Documentation/$$l/web.texi
;\
for m in $(MANUALS); do \
n=`echo $$m | sed
's/Documentation/Documentation\/'$$l'/'` ; \
b=`basename $$n .texi`; \
d=`basename $$b .tely`; \
if [ -e $$n ] ; then \
$(EXTRACT_TEXI_FILENAMES) \
-I $(top-src-dir)/Documentation \
-I
$(top-src-dir)/Documentation/$$l \
-I
$(top-src-dir)/Documentation/$$r$$d \
--known-missing-files=$(top-src-dir)/scripts/build/website-known-missing-files.txt
\
$(quiet-flag) \
-I $(OUT) -o $(OUT) $$n ; \
fi ; \
done; \
done;
this is, of course, completely disgusting. line continuations on
top of a bash for loop on top of make results in horrible
substition rules like
len=$${#l}
I really, really wish that I could have written this in python.
Rewriting that as 2 or 3 python functions would make it so much
easier to read and understand!
(can you have functions in makefiles? that would simplify stuff)
I may have missed some simple way to do the above in make. If so,
please let me know about it!
Cheers,
- Graham
___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
Graham Percival gra...@percival-music.ca writes: On Sat, Sep 24, 2011 at 10:41:05PM -0300, Han-Wen Nienhuys wrote: On Sat, Sep 24, 2011 at 10:05 PM, Graham Percival gra...@percival-music.ca wrote: This didn't happen for fun and giggles. Over the years, we've built up hack upon hack, and ended up with this unholy mess. You sound spoiled. Why, because I think it's not perfect? How many discussions, how many developer-hours, have we spend talking about build problems on this list? How much frustration and wasted time/goodwill have we spent on those problems? The main problem I see is that dependencies don't work out, and that presumably is mostly because the temporary/work files of lilypond-book are not in the rules and get stomped over by parallel make. We need to get this fixed in order to be able to use big bad multicore machines for automated building, testing and bisecting. Cross fingers and retry is nothing you want for automated work. I am totally spoiled by python's readable code. I am so accustomed to writing stuff like cmd = compiler + ' -o ' + exe_name + src_files or cmd = %(compiler)s -o %(exe_name)s %(src_files) % locals() that I find stuff like $(CC) -o $@ $ silly. The readability for casual contributors -- which is what most people looking at build system stuff are -- is ridiculously better in python than anything else. I don't do Python or many newfangled languages. I have worked with Make for over 20 years. The casual contributor will be one used to the technology and thinking underlying Lilypond. More likely than not someone with more than a trace of free software project experience in them. I guess that at least 98% of all free software projects use Make for its build system. Make is not the problem. The problem is managing the Makefiles once you start writing them automatically. Then you have another layer. How much of this layer simplifies things, how much complicated them? Do we need it at all? Probably the main thing that needs to be outsourced is the intelligent handling of lilypond-book/snippets. -- David Kastrup ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
On Sun, Sep 25, 2011 at 08:33:45AM +0200, David Kastrup wrote: I don't do Python or many newfangled languages. I have worked with Make for over 20 years. The casual contributor will be one used to the technology and thinking underlying Lilypond. More likely than not someone with more than a trace of free software project experience in them. You are incorrect on this point. In early summer, I tried to start an effort to figure out how the build system worked. The main volunteers were Phil and Colin: two people with zero previous free software project experience. And both working on windows, actually! Public record of this effort: https://lists.gnu.org/archive/html/lilypond-devel/2011-03/msg00059.html You can see the results so far in git. Undoubtedly you will laugh at how primitive and possibly incorrect these descriptions are, but this what we've managed to do in six months: gitk Documentation/contributor/build-notes.itexi Yes, various expert FLOSS members (such as Reinhold, Carl, and IIRC yourself) have stepped forward to fix a few things in the builds -- but the only people who are working on the build system full time are windows users. (that said, Phil recently bought a fast computer so that he can make builds faster and thus experiment more easily) Don't tell me that experienced hackers will take care of the build system. They aren't. - Graham ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
Graham Percival gra...@percival-music.ca writes: On Sun, Sep 25, 2011 at 08:33:45AM +0200, David Kastrup wrote: I don't do Python or many newfangled languages. I have worked with Make for over 20 years. The casual contributor will be one used to the technology and thinking underlying Lilypond. More likely than not someone with more than a trace of free software project experience in them. You are incorrect on this point. [...] Yes, various expert FLOSS members (such as Reinhold, Carl, and IIRC yourself) have stepped forward to fix a few things in the builds -- but the only people who are working on the build system full time are windows users. (that said, Phil recently bought a fast computer so that he can make builds faster and thus experiment more easily) Uh, you changed topic. The topic was the casual contributor, not working fulltime. I was talking about those people who _don't_ focus on the build system because they prefer contributing elsewhere. And I was saying that turning the build system into something that only experts in the build system can handle is a bad idea. A contorted system using autogenerated Makefiles is that. But a system written and maintained in Python is that as well. In my opinion, jumping to yet-another-system instead of cleaning up the current system is a step backwards, unless one can prove that the current system can't provide the required functionality without exceeding reasonable complexity, and a new system can. Don't tell me that experienced hackers will take care of the build system. They aren't. Changing the build system to something where they have a plausible reason not to take care is not going to improve matters. Even full-time contributors come and go eventually. A build system that is only accessible to those who have left already is not going to help. -- David Kastrup ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
On Sun, Sep 25, 2011 at 09:50:52AM +0200, David Kastrup wrote: Graham Percival gra...@percival-music.ca writes: Yes, various expert FLOSS members (such as Reinhold, Carl, and IIRC yourself) have stepped forward to fix a few things in the builds -- but the only people who are working on the build system full time are windows users. Uh, you changed topic. The topic was the casual contributor, not working fulltime. My apologies. You are correct. I need to stop talking. Especially these days, when I'm disagreeing with everybody for no good reason. In my opinion, jumping to yet-another-system instead of cleaning up the current system is a step backwards, unless one can prove that the current system can't provide the required functionality without exceeding reasonable complexity, and a new system can. Agreed. - Graham ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
Graham Percival gra...@percival-music.ca writes: On Sun, Sep 25, 2011 at 09:50:52AM +0200, David Kastrup wrote: Graham Percival gra...@percival-music.ca writes: Yes, various expert FLOSS members (such as Reinhold, Carl, and IIRC yourself) have stepped forward to fix a few things in the builds -- but the only people who are working on the build system full time are windows users. Uh, you changed topic. The topic was the casual contributor, not working fulltime. My apologies. You are correct. I need to stop talking. Especially these days, when I'm disagreeing with everybody for no good reason. When things go wrong too much, not least of all because people don't listen to policies and good reason, disagreement is a natural impulse, more natural than rereading carefully. It happens occasionally that I find myself in the situation where in the progression of a heated disagreement I want to dig up the original quotes starting the fight, and they are not quite what I remembered, much more ambiguous. Probably not quite what the other side remembered either. Anyway, I know the scenario enthusiastic contributor redoes a subsystem according to his own ideas and tools, and sometime later leaves too well. It is rarely a well-scaling idea even in a corporate environment when the chances for continuity are usually better than in free projects: shifting workloads almost always mean that the original author at some point of time no longer can reasonably be bothered with exclusive maintenance. Historically grown systems often are garbage heaps, so redo from scratch may really offer reasonable improvement with permanent advantages. And it is a psychological crutch to do it with a different tool/toolset because then the temptation to reuse parts that work so-so is not there. I've seen that kind of thing. But then it ends in the to make this really maintainable for the bulk of people who will end up working on it, we ought to redo it again, this time reimplementing it for the old tools again phase, and guess how much motivation to do that is there among the few people who actually bothere to learn the new system. Project management sucks. I don't envy you. -- David Kastrup ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
On Sun, Sep 25, 2011 at 4:39 AM, Graham Percival gra...@percival-music.ca wrote: Yes, various expert FLOSS members (such as Reinhold, Carl, and IIRC yourself) have stepped forward to fix a few things in the builds -- but the only people who are working on the build system full time are windows users. (that said, Phil recently bought a fast computer so that he can make builds faster and thus experiment more easily) Don't tell me that experienced hackers will take care of the build system. They aren't. The real problem is that there are no build systems that do not suck: * you have to discouple build rule organization from directory hierarchy. To keep our sanity, we organize around per-directory recursive makes. This means that lily/out/lilypond can not directly depend on flower/path.cc; instead we have to run a recursive make just to figure it hasn't changed. * it has to run as a server, and keep the tree state in memory. Otherwise, incremental builds will be dog-slow. Scons got this wong. * it has to be programmable in a sane language (make fails this badly) Make does suck, but at least, it is simple, universal and well-understood. This is why we should stick to it until something significantly better comes along. If we're sticking to make, I doubt that we will end up with something better than what we currently have. There is some argument to use makefile generators instead of our own system, eg. CMake has some promise, but most of the lilypond build involves custom tools, so we'd need to write a lot of cmake templates from scratch. It's not clear to me this will be much better than the make templates we use right now. The only system I know that does not suck is not open source, unfortunately: http://google-engtools.blogspot.com/2011/08/build-in-cloud-how-build-system-works.html. Sometimes my hands itch to rewrite and open source a functional subset of it, but there is only have so much time in a day. Regarding parallelism and lilypond-book: AFAICS - running parallel builds should work correctly right now, although the combination of make -jX and lilypond -djob-count=X is not pretty. It would be an interesting exercise to see if lilypond-book coud use a recursive make invocation to schedule document sub-jobs. -- Han-Wen Nienhuys - han...@xs4all.nl - http://www.xs4all.nl/~hanwen ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
David Kastrup: ... The main problem I see is that dependencies don't work out, and that presumably is mostly because the temporary/work files of lilypond-book are not in the rules and get stomped over by parallel make. ... I have given up on lilypond-book and make. Instead I've made two scripts depend_ly and depen_tex [1] which finds out what depends on what (think gcc -M), and make [2] takes care of the rest. Would it be good to make lilypond print out a files dependancies like gcc? Regards, /Karl Hammar [1] http://turkos.aspodata.se/git/musik/bin/ [2] http://turkos.aspodata.se/git/musik/include/Makefile --- Aspö Data Lilla Aspö 148 S-742 94 Östhammar Sweden +46 173 140 57 ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
Hi, On Fri, Aug 12, 2011 at 02:53:56PM +0100, Phil Holmes wrote: I understand it's been discussed before, but I am wondering whether it's worth thinking the unthinkable and considering moving away from make. I know it's been used in loads of projects and is much loved, but actually, from a design perspective, it's appalling. I don't understand why so many people think it is... The Make language as it stands does have some quirks, but I like the fundamental concept. But of course that's mostly irrelevant anyways when using a Makefile generator such as Autotools... If I was writing a make system from scratch, I would describe dependencies in data structures that are viewable and editable, and have a separate program that uses those structures to determine which files need making. I'm not sure why you need extra structures for that? For C/C++, gcc can figure out the dependencies as a side effect of compiling the normal source code; and this can be done for most other languages as well. Very few non-trivial programs actually maintain their dependencies by hand nowadays... I've done 5 minutes research and have found SCons. I've not gone into any more depth with that yet. Does it seem worth looking into this, or something else, in more detail? I don't know about SCons, but at least the often-proposed Cmake is *not* a Make replacement. It's a Makefile generator, just like Autotools. Also please consider that the build system is not used only by developers, but also by distribution packagers, and anyone else building the software. Projects using Autotools are still by far the most convenient; anything else means extra effort. -antrik- ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
On Fri, Sep 23, 2011 at 07:08:26PM +0200, olafbuddenha...@gmx.net wrote: Hi, On Fri, Aug 12, 2011 at 02:53:56PM +0100, Phil Holmes wrote: I understand it's been discussed before, but I am wondering whether it's worth thinking the unthinkable and considering moving away from make. I know it's been used in loads of projects and is much loved, but actually, from a design perspective, it's appalling. I don't understand why so many people think it is... The Make language as it stands does have some quirks, but I like the fundamental concept. That's like saying java has some quirks, but I like the fundamental concept of architecture-independent interpreted code and just-in-time compilers. There's an order of magnitude between ease of programming in python and java. But of course that's mostly irrelevant anyways when using a Makefile generator such as Autotools... Phil was using the term make to refer to our stepmake system, which is on top of autotools, which (of course) is on top of actual make(1). Yes, this is technically incorrect, but I think that's a relatively small nit to pick. If I was writing a make system from scratch, I would describe dependencies in data structures that are viewable and editable, and have a separate program that uses those structures to determine which files need making. I'm not sure why you need extra structures for that? For C/C++, gcc can figure out the dependencies as a side effect of compiling the normal source code; and this can be done for most other languages as well. Very few non-trivial programs actually maintain their dependencies by hand nowadays... That's a great comfort for our lilypond texinfo documentation, which *does* require explicit dependencies. As far as I know, every single build system has trivially easy support for C/C++ compilation. That's not an issue. The issue is building documentation, building swig python modules (for me personally, not lilypond), generating translations, generating test cases and automatically showing potential problems, etc. According to find . -name *.make -or -name GNUmakefile* | xargs cat | wc -l we have 5059 lines of build-system stuff. Actually, that's not counting python helper scripts: wc scripts/build/*.py -l 4113 total So... 9171 lines of build-system junk. I think this meme is appropriate: It's over NINE THOUSAND!!! This didn't happen for fun and giggles. Over the years, we've built up hack upon hack, and ended up with this unholy mess. Is that purely the fault of autotools? No. Could it be *worse* if we used a different build system? Sure. But it could also be *better* if we used a different build system. I've done 5 minutes research and have found SCons. I've not gone into any more depth with that yet. Does it seem worth looking into this, or something else, in more detail? I don't know about SCons, but at least the often-proposed Cmake is *not* a Make replacement. It's a Makefile generator, just like Autotools. Yes, but it's still a different *build system*. Our interaction with the build system would take the form of CMakeLists.txt files. Would that interaction result in a more maintainable system, i.e. one which does not have a powerlevel of OVER 9000? Probably yes. What's uncertain is how much work would be involved in switching to that (or any other) build system, vs. how much better that would make matters, Cheers, - Graham ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
On Sat, Sep 24, 2011 at 10:05 PM, Graham Percival gra...@percival-music.ca wrote: If I was writing a make system from scratch, I would describe dependencies in data structures that are viewable and editable, and have a separate program that uses those structures to determine which files need making. I'm not sure why you need extra structures for that? For C/C++, gcc can figure out the dependencies as a side effect of compiling the normal source code; and this can be done for most other languages as well. Very few non-trivial programs actually maintain their dependencies by hand nowadays... That's a great comfort for our lilypond texinfo documentation, which *does* require explicit dependencies. As far as I know, every single build system has trivially easy support for C/C++ compilation. That's not an issue. The issue is building documentation, building swig python modules (for me personally, not lilypond), generating translations, generating test cases and automatically showing potential problems, etc. According to find . -name *.make -or -name GNUmakefile* | xargs cat | wc -l we have 5059 lines of build-system stuff. Actually, that's not counting python helper scripts: wc scripts/build/*.py -l 4113 total So... 9171 lines of build-system junk. I think this meme is appropriate: It's over NINE THOUSAND!!! This didn't happen for fun and giggles. Over the years, we've built up hack upon hack, and ended up with this unholy mess. Is that purely the fault of autotools? No. Could it be *worse* if we used a different build system? Sure. But it could also be *better* if we used a different build system. You sound spoiled. * The build scripts are not really part of the complexity of the build system. They are normal programs, they are just not intended to be run by end users. I assume you are not considering a rewrite of output-distance.py in CMake's scripting language. * You make 5000 lines of makefile sound as if it is a lot. I think you should look at other packages' build systems more closely. For example, coreutils, which is 70k lines of C, and a trivial documtentation and test process has, count'dem: 2000 lines of Makefile.am. This is for just linking together a bunch of bog-standard C object files, generating an info page, and running some shell scripts as tests. I think 5k lines is pretty impressive for a package as complex as LilyPond, considering that it has a bizarre documentation process, translated docs, python bindings, fonts that are run through metapost and fontforge, a complex regression test, and needs weird acrobatics to find its gazilion different init files. -- Han-Wen Nienhuys - han...@xs4all.nl - http://www.xs4all.nl/~hanwen ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
On Sat, Sep 24, 2011 at 10:41:05PM -0300, Han-Wen Nienhuys wrote: On Sat, Sep 24, 2011 at 10:05 PM, Graham Percival gra...@percival-music.ca wrote: This didn't happen for fun and giggles. Over the years, we've built up hack upon hack, and ended up with this unholy mess. You sound spoiled. Why, because I think it's not perfect? How many discussions, how many developer-hours, have we spend talking about build problems on this list? How much frustration and wasted time/goodwill have we spent on those problems? Despite that, I'm not advocating a move away from make; it would take us about 200 hours to reach merely the current level of build system-ness. There are far more important things to work on -- but I'm not going to pretend that the status quo is great, nor do I think it's silly to ask about switching to a different system. I will admit there is one aspect in which I *am* spoiled, though: I am totally spoiled by python's readable code. I am so accustomed to writing stuff like cmd = compiler + ' -o ' + exe_name + src_files or cmd = %(compiler)s -o %(exe_name)s %(src_files) % locals() that I find stuff like $(CC) -o $@ $ silly. The readability for casual contributors -- which is what most people looking at build system stuff are -- is ridiculously better in python than anything else. * The build scripts are not really part of the complexity of the build system. They are normal programs, they are just not intended to be run by end users. I assume you are not considering a rewrite of output-distance.py in CMake's scripting language. True. OTOH, there is certainly complexity in there that we don't need -- AFAIK we don't use bib2html.pl any more; we instead ose bib2texi.py. So we could remove the perl file from the GNUmakefiles, and even remove the script entirely from our source tree. It's stuff like that which bugs me. Some time down the road, somebody will be trying to fix some bug in the bibliography, and they'll spend hours looking at bib2html and the files it produces, before finally discovering that it's not actually used. * You make 5000 lines of makefile sound as if it is a lot. I think you should look at other packages' build systems more closely. For example, coreutils, which is 70k lines of C, and a trivial documtentation and test process has, count'dem: 2000 lines of Makefile.am. This is for just linking together a bunch of bog-standard C object files, generating an info page, and running some shell scripts as tests. Wow. That's insane. I think 5k lines is pretty impressive for a package as complex as LilyPond, considering that it has a bizarre documentation process, translated docs, python bindings, fonts that are run through metapost and fontforge, a complex regression test, and needs weird acrobatics to find its gazilion different init files. It's certainly impressive, and it could definitely be worse. But it's not easy to maintain. At least, it's not easy to maintain for casual developers like me. I'd be overjoyed if somebody non-spoiled and non-stupid (i.e. not me) offered to step forward and fix the known problems with the build system. Cheers, - Graham ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
On Sun, Sep 25, 2011 at 03:12:14AM +0100, Graham Percival wrote: On Sat, Sep 24, 2011 at 10:41:05PM -0300, Han-Wen Nienhuys wrote: You sound spoiled. On second thought, I really *am* spoiled: I don't want to even notice the build system. I view it in the same way I view food: a waste of time that's unfortunately necessary for survival. I mean, food is a pain. You need to leave the house/university to buy it, you need to carry it home, you need to cook (or otherwise prepare) it, then eat it, then wash the dishes, and occasionally empty the garbage depending on what you were preparing. Now, with frozen lazagna the cooking+washing isn't so bad (since you can eat it out the dish; you only need to wash one fork). But if you eat nothing but microwaved food, you get sick. Trust me, I've tried. Life would be so much easier if/when they have some kind of grey mush that provides all nutrients you need! That's what I want from a build system. I want a grey mush that just works. No problems, no confusion, no work required to maintain it. I want to spend time improving documentation or fixing bugs in graphical output or adding new features! Unfortunately that's not possible, so I'm hoping for the next best thing: to have a build system which requires the least amount of work, while still providing the most robust environment with the least amount of confusion for developers. Now, I suppose that since I have such hostility for build systems, I really shouldn't be involved in discussing them -- I should leave it up to people who enjoy dealing with their crap. Unfortunately, just like the kitchen garbage in my shared apartment, if I don't clean up the garbage from time to time, nobody else does. Cheers, - Graham ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
On Sat, Aug 20, 2011 at 6:06 AM, Jan Nieuwenhuizen jann...@gnu.org wrote: Han-Wen Nienhuys writes: Given that Cmake has a large following (examples include KDE and LLVM), I'd be comfortable with switching to that. Interesting; have you ever used Cmake? Lately I've been doing tweaks to parallel compilations of LLVM (cmake) and GCC (autoconf + GNU make + some bizarre templating), and the Cmake one is the one which gave less headaches. From a general perspective, choosing something that has a larger following is a smarter choice, since it is likely that others have already dealt with most of the obvious problems. If I lived in an ideal world, I'd write a build-system from scratch (and maybe some day, I will). -- Han-Wen Nienhuys - han...@xs4all.nl - http://www.xs4all.nl/~hanwen ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
Han-Wen Nienhuys writes: Given that Cmake has a large following (examples include KDE and LLVM), I'd be comfortable with switching to that. Interesting; have you ever used Cmake? Last time I looked (migrated a cmake project to autotools), Cmake did only have proprietary documentation (I hear most documentation is in a wiki now), introduced a rather crappy home brew language, cached make information in generated makefiles that do not use variables for $(CC) or $(CFLAGS), has a IMHO nasty way of overriding such variables on the cmake command line. Also, it did not have an easy way to create help from the command line or a facility to have virtual targets or a nice way to say: `make -C lily out/parser.o' because all file names were absolute, fully expanded names. Jan -- Jan Nieuwenhuizen jann...@gnu.org | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.nl ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
On Sat, Aug 20, 2011 at 11:06:29AM +0200, Jan Nieuwenhuizen wrote: Han-Wen Nienhuys writes: Given that Cmake has a large following (examples include KDE and LLVM), I'd be comfortable with switching to that. Interesting; have you ever used Cmake? I migrated Marsyas (a moderately-sized project; 200,000 lines of code) from a combination of autotools and qmake (i.e. two separate build systems, maintained in tandem!) to cmake. Last time I looked (migrated a cmake project to autotools), Cmake did only have proprietary documentation Yes. Most of my cmake knowledge from reading slides of conference presentations and bug reports on their mailing list (or on blogs). introduced a rather crappy home brew language, Change that to extremely crappy home-brew language. Their crappy home-brew language is ridiculously far away from the readability of modern scripting languages like python and lua. That said, it still avoids the confusing $ $@ syntax from make. cached make information in generated makefiles that do not use variables for $(CC) or $(CFLAGS), I know it's extremely easy for users to change CC/CFLAGS/CXXFLAGS variables during the cmake configure stage. IMO, the generated makefiles aren't particularly intended to be human-readable. has a IMHO nasty way of overriding such variables on the cmake command line. Never use the cmake command line; always use ccmake (or even the graphical one, although I've never done that myself). Also, it did not have an easy way to create help from the command line There's good help strings in ccmake; I'm not bothered by this one. or a facility to have virtual targets or a nice way to say: `make -C lily out/parser.o' because all file names were absolute, fully expanded names. Hmm, I can't comment either way on this one. I'd be surprised if there was no way to do this, but OTOH I've been surprised about some brain-dead cmake stuff in the past. Most recently: in cmake 2.6, you can specify a working directory for a compile target, but you can't specify a working directory for a unit test. Recommended solution: write a python wrapper that changes to the relevant directory and runs your command, then exports the exit code. Overall, cmake is not an unambiguous win... but if I were starting a new moderate-sized project, I'd probably use cmake rather than any of the other build systems. Cheers, - Graham ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Moving away from make
I understand it's been discussed before, but I am wondering whether it's worth thinking the unthinkable and considering moving away from make. I know it's been used in loads of projects and is much loved, but actually, from a design perspective, it's appalling. If I was writing a make system from scratch, I would describe dependencies in data structures that are viewable and editable, and have a separate program that uses those structures to determine which files need making. Instead we have a fairly impenetrable system of makefiles that are created by (to me) a completely impenetrable autoconf system, and the only way of checking dependencies is to open all the makefiles (sourcefiles in effect) and read and understand each. It's rather as if one had to read the LilyPond .cpp files to understand how to change a piece of music. I've done 5 minutes research and have found SCons. I've not gone into any more depth with that yet. Does it seem worth looking into this, or something else, in more detail? -- Phil Holmes ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
Am Freitag, 12. August 2011, 15:53:56 schrieb Phil Holmes: I understand it's been discussed before, but I am wondering whether it's worth thinking the unthinkable and considering moving away from make. I suppose that everyone here would be glad if we could get away from make. It's just that our build system is really, really complicated and very complex, including dozens of different functionalities, different handling of the same file type depending on the directory, etc. I've done 5 minutes research and have found SCons. I've not gone into any more depth with that yet. Does it seem worth looking into this, or something else, in more detail? Jan did some work ok building lilypond with scons from 2004-2007. All traces of that have been removed 2009 by John in commit 24cd9ffc8b5a4ea03a29414eb7ae038a2d568d45. Another candidate would be cmake, which is used by the KDE project, so it is also able to handle large projects. I don't know, however, whether any of those is really able to provide the functionality that we really want/need (bin/doc/web/check/test-baseline builds, cross-compiling for GUB, etc.) Cheers, Reinhold PS: There was also some discussion on -devel to use waf, but IIRC that's lacking some vital features. PS2: When KDE switched to cmake, they initially favored SCons, but that was not up to the task of building KDE, so they finally ended up with cmake. See http://lwn.net/Articles/188693/ -- -- Reinhold Kainhofer, reinh...@kainhofer.com, http://reinhold.kainhofer.com/ * Financial Actuarial Math., Vienna Univ. of Technology, Austria * http://www.fam.tuwien.ac.at/, DVR: 0005886 * LilyPond, Music typesetting, http://www.lilypond.org ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
I understand it's been discussed before, but I am wondering whether it's worth thinking the unthinkable and considering moving away from make. Any simplification is welcomed, I think. I've done 5 minutes research and have found SCons. I've not gone into any more depth with that yet. Does it seem worth looking into this, or something else, in more detail? LilyPond had support for SCons some time ago, IIRC, but this has been dropped meanwhile. Other thinkable make alternatives are cmake and makepp. Werner ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
On Fri, Aug 12, 2011 at 02:53:56PM +0100, Phil Holmes wrote: I understand it's been discussed before, but I am wondering whether it's worth thinking the unthinkable and considering moving away from make. Budget 2000 hours. That's not a typo. I don't think it provides a good cost/benefit ratio. Another problem is that every build system sucks. I've tried scons, waf, and cmake. Out of all of those, cmake is the least icky, but I hate that they invented Yet Another Scripting Language and don't let me do stuff with the simplicity and elegance of python. waf looks the nicest at first glance, but they don't support having files with the same name in the source tree and build tree, and I was appalled at their attitude (you shouldn't want to do that). The job of the build system is to do whatever we need; being told that we shouldn't want to view html documents, or that we should change our directory structure, does not impress me. Look for previous discussion about this in the archives, and if you're interested, you could try to talk some sense into the waf people. I know it's been used in loads of projects and is much loved, but actually, from a design perspective, it's appalling. Oh, make is a disaster. But the sad thing is that it's less of a disaster than those other major contenders. I've heard rumors of a google build system; if that's open-source, it might be a possible contender. But failing that, I think we're stuck. Cheers, - Graham ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
On 8/12/11 9:32 AM, Graham Percival gra...@percival-music.ca wrote: On Fri, Aug 12, 2011 at 02:53:56PM +0100, Phil Holmes wrote: I understand it's been discussed before, but I am wondering whether it's worth thinking the unthinkable and considering moving away from make. Budget 2000 hours. That's not a typo. I don't think it provides a good cost/benefit ratio. Another problem is that every build system sucks. I've tried scons, waf, and cmake. Out of all of those, cmake is the least icky, but I hate that they invented Yet Another Scripting Language and don't let me do stuff with the simplicity and elegance of python. waf looks the nicest at first glance, but they don't support having files with the same name in the source tree and build tree, and I was appalled at their attitude (you shouldn't want to do that). The job of the build system is to do whatever we need; being told that we shouldn't want to view html documents, or that we should change our directory structure, does not impress me. Look for previous discussion about this in the archives, and if you're interested, you could try to talk some sense into the waf people. I've been loosely following waf, and I think the restriction about the same name was in waf 1.5 but has been eliminated in 1.6, which is now out. I think it would be worth taking another look. Thanks, Carl ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
On Fri, Aug 12, 2011 at 09:51:46AM -0600, Carl Sorensen wrote: On 8/12/11 9:32 AM, Graham Percival gra...@percival-music.ca wrote: waf looks the nicest at first glance, but they don't support having files with the same name in the source tree and build tree, I've been loosely following waf, and I think the restriction about the same name was in waf 1.5 but has been eliminated in 1.6, which is now out. I think it would be worth taking another look. Ok. Here you go: http://git.savannah.gnu.org/gitweb/?p=lilypond.git;a=shortlog;h=refs/heads/dev/waf If you, or Phil, or anybody, can get some basic html documents happening in dev/waf, we can discuss it further. If it's actually possible, then ETA: 2 hours (not counting time learning how to switch branches, and not counting time spent reading the previous serious discussion in fall 2009) Cheers, - Graham ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Moving away from make
On Fri, Aug 12, 2011 at 10:53 AM, Phil Holmes em...@philholmes.net wrote: I understand it's been discussed before, but I am wondering whether it's worth thinking the unthinkable and considering moving away from make. I know it's been used in loads of projects and is much loved, but actually, from a design perspective, it's appalling. If I was writing a make system from scratch, Careful: many people have tried writing something better, and most attempts failed. It is not a trivial problem. I would describe dependencies in data structures that are viewable and editable, and have a separate program that uses those structures to determine which files need making. Instead we have a fairly impenetrable system of makefiles that are created by (to me) a completely impenetrable autoconf system, and the only way of checking dependencies is to open all the makefiles (sourcefiles in effect) and read and understand each. It's rather as if one had to read the LilyPond .cpp files to understand how to change a piece of music. We tried scons for a while, but it is extremely slow for incremental builds. Given that Cmake has a large following (examples include KDE and LLVM), I'd be comfortable with switching to that. -- Han-Wen Nienhuys - han...@xs4all.nl - http://www.xs4all.nl/~hanwen ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
