Re: Doc build hanging (with memory leak?)
On 12-05-14 07:13 AM, Joseph Rushton Wakeling wrote: Might be a nice moment to see what I can do about that long-dormant doc work. :-) ___ Let me know if I can help, Joseph, and welcome back! Colin "the older one" Campbell -- I've learned that you shouldn't go through life with a catcher's mitt on both hands. You need to be able to throw something back. -Maya Angelou, poet (1928- ) ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Doc build hanging (with memory leak?)
Graham Percival writes: > Tomorrow I'll go see the "science and technology" museum (maybe 4 > hours?), 4 weeks is more realistic. -- David Kastrup ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Doc build hanging (with memory leak?)
On Mon, May 14, 2012 at 01:48:54PM +0200, Joseph Rushton Wakeling wrote: > On 14/05/12 11:41, David Kastrup wrote: > > > > Before saying anything more, I'm sorry if my earlier email was > offensive or intemperate; it wasn't meant to be. Ditto. > I was writing out of concern for the ease of contributing to > LilyPond (more on that in a moment). I agree that it's harder to contribute than it should be. However, I am confident in stating that the summary for experienced developers is the best we can offer new (experienced) contributors given our current system and the amount of time+effort+interest the current developers have in mentoring new people. It's not ideal, but it's the best compromise I can find. > If I understand correctly, Rietveld is a server-side app. My > objection isn't to Rietveld per se, but to the requirement to use a > custom app on my system to get my code _into_ Rietveld. This seems > to be Rietveld's fault, so I'm sorry if it seemed like I was > apportioning blame to the LP team. You can upload to rietveld without our custom app (but using a different one); I think you can even upload without any app at all. The reason we push our own git-cl (custom app) is that this keeps track of patches for us. Before we started using it, we lost approximately 10-20% of patches. "lost" as in "hey guys, I sent a patch three months ago, has anybody looked at it?". Or even sadder, not having any follow-up at at all and completely losing that work. Of course, even in the "three-month inquiry" case, it sometimes still led to completely losing that work because git master had changed sufficiently that there were many conflicts (or else the patch just didn't make sense any more given the changed architecture). If you think that's a horrific record, then I quite agree. If you think that somebody should take responsibility for new contributors / new patches... well, that would be nice, but that's historically been lacking in lilypond. Best compromise? put the onus on patch submitters to submit with our special tool, which at least ensures that we won't lost their patch. Assuming that our current amount of interest in new patches stays constant, I am certain that turning away contributors unwilling or unable to run git-cl is doing them a favor in the long run. > I was just astonished to be asked to run a tiny doc > patch through a code-testing service. If you're lucky, somebody might offer to handle the "red tape" for you. But that would be a matter of individual luck and somebody individually offering. > I'll add an extra story which may give some context to my reaction. ... > That experience may have coloured my reaction when a small and > easy-to-include patch, knocked off as a friendly gesture to try and > make sure someone else didn't have the same scary experience I did > with the new doc build system, got in response a terse instruction > to submit it via a complicated and unfamiliar set of custom tools > whose whole raison d'etre is to test code, not docs. It was terse because I'm on vacation but I still need to deal with lilypond crap because it's likely that nobody else will. I've spent 6 hours sightseeing in Munich, but I'm spending the rest of the day in the hotel room doing lilypond, reviewing academic conference papers, and if I'm lucky working on my thesis. Tomorrow I'll go see the "science and technology" museum (maybe 4 hours?), then spend the rest of the day in my hotel room again. > I'm not trying to suggest that anyone is evil, bad or stupid, but it > really didn't seem to me to be the best way to handle things for a > case like this. Given the numerous problems that new contributors face, I believe that the most honest response is to discourage them. No, it's not easy. No, you won't get a lot of help. If that's not for you, then please wait a few months and try again -- hopefully things will be better then. This is not a pleasant policy, but I'm trying to save you guys (there have been other new contributors as well) a lot of heartache. It would be easy for me to write a "feel-good" email that encouraged you to keep on trying, but that would be dishonest since I know how hard it will be. - Graham ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Doc build hanging (with memory leak?)
Joseph Rushton Wakeling writes: > One of the frustrations on that occasion a year ago was that I would > have been quite likely on my own initiative to see if I could do > anything, _if_ I hadn't been treated to the lecture. It is a reasonably safe guess that the "lecture" had been by Graham. Graham has been the de facto project leader for a number of years now and has been frequently willing to quit. He has some renown for being "grumpy" (from a time where I did not start topping him in that regard), and part of the reason is that he has been bogged down with a _lot_ of administrative work for LilyPond that nobody else was really eager to step up for. As a result, his reactions to "wouldn't it be fantastic if you took on all of the following additional work" kind of suggestions tends to be less than enthusiastic, and there is rarely a large cool-off period between two of them. This may at times constitute sort of an entry threshold. When people are forced to take sides, they will tend to favor opinions more compatible with Graham staying around rather than a newcomer. Now it would be unfair to put all the blame and all the credit on Graham: the fact is that musicians are somewhat special, and programmers are somewhat special, and the combination does not get all that less special. Try reading through the recent thread on the developer list with "s1*0" in its title. You'll find that long-term contributors with a high respect of each other's contributions and personality can't help going for the others' jugular. And that's because they are passionate about what they do with and to LilyPond. For better or worse, it is not necessarily an environment abounding with pleasantry. But there _is_ work being done, decisions made, and LilyPond brought forward with dedication and compassion. >> Have you considered the possibility that he was simply telling you >> the truth? > > At the time it didn't seem that way. But I'm prepared to believe that > was in fact the case and I misinterpreted it. I am pretty sure it was the truth, though likely not told with a surplus amount of delicacy. > Might be a nice moment to see what I can do about that long-dormant > doc work. :-) Well, I can't promise you a Rosegarden. -- David Kastrup ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Doc build hanging (with memory leak?)
On 14/05/12 14:15, David Kastrup wrote: They are treated by the bug squad picking up the suggestion and filing it in the issue problem. If someone suggested to you that they will refuse doing that, that someone was not giving you correct information. OK. :-) To be fair, reconsidering things, I wonder if the suggestion to go via Rietveld was simply because it was assumed that if I know git and TeXinfo, I'd have no problem with Rietveld either. Yes, it is demotivating. But you are shooting the messenger in this case. If nobody goes through all the make documentation and build scripts and rewrites them, the dependencies are not there for making the process faster. It turns out that, unless I am mistaken, somebody more or less did that. But he did not, as far as I am aware, start out from a better informed state than you did. Yes, I noticed that as I was preparing my patch. It was very nice to see. :-) One of the frustrations on that occasion a year ago was that I would have been quite likely on my own initiative to see if I could do anything, _if_ I hadn't been treated to the lecture. If you consider the problems in the LilyPond code base as a personal insult, you are not likely to have much fun. It's never the problems in the _code base_ -- they're a challenge, not an insult! Have you considered the possibility that he was simply telling you the truth? At the time it didn't seem that way. But I'm prepared to believe that was in fact the case and I misinterpreted it. Might be a nice moment to see what I can do about that long-dormant doc work. :-) ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Doc build hanging (with memory leak?)
Joseph Rushton Wakeling writes: > It's not trivial if I have to install custom code in order to submit a > tiny doc patch! > > If I were hacking on LP itself then it would surely be a small issue > by comparison, but I wasn't, and I don't think small tweak-y > contributions should be treated this way. They are treated by the bug squad picking up the suggestion and filing it in the issue problem. If someone suggested to you that they will refuse doing that, that someone was not giving you correct information. > The workaround which several helpful list members suggested was to > touch' one of the files for the manual concerned. The problem with > this workaround was that it meant that everything in the manual had to > be rebuilt -- it was much less than a full doc build, but still a > large build process. I wanted to make sure I hadn't missed anything, > so I asked if there was really no way to get the rebuild to just cover > my changes and not all the extra untouched material. > > ... and at this point someone chipped in with a long and fairly > unpleasant lecture along the lines of, "Sure, you can read all the > make documentation and go through the build scripts and rewrite them > all ...". It was a big and nasty shock especially as I'd been > perfectly polite and was eager to contribute. It was extremely > demotivating Yes, it is demotivating. But you are shooting the messenger in this case. If nobody goes through all the make documentation and build scripts and rewrites them, the dependencies are not there for making the process faster. It turns out that, unless I am mistaken, somebody more or less did that. But he did not, as far as I am aware, start out from a better informed state than you did. > and I was quite angry about it, not only on my own behalf but because > I thought it was an awful way to treat anyone who was actually wanting > to make a contribution. If you consider the problems in the LilyPond code base as a personal insult, you are not likely to have much fun. > Now, there could be a plenty of reasons why that developer had that > reaction -- just a bad day, feeling undervalued, too often getting > complaints about the system without offers to help -- but it really > made me wonder about the project's attitude to potential contributors. Have you considered the possibility that he was simply telling you the truth? -- David Kastrup ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Doc build hanging (with memory leak?)
On 14/05/12 11:41, David Kastrup wrote: Before saying anything more, I'm sorry if my earlier email was offensive or intemperate; it wasn't meant to be. I was writing out of concern for the ease of contributing to LilyPond (more on that in a moment). Have you actually used Rietveld for reviewing code? It does a job that the mentioned tools don't. If I understand correctly, Rietveld is a server-side app. My objection isn't to Rietveld per se, but to the requirement to use a custom app on my system to get my code _into_ Rietveld. This seems to be Rietveld's fault, so I'm sorry if it seemed like I was apportioning blame to the LP team. I _strongly_ dislike any requirement to install project-specific custom tools and system tweaks in order to contribute. It's not just about the personal hassle, but about wanting to lower the barrier for all contributions and in particular for off-the-cuff contributions. That's not saying that I don't understand you have very good reasons for wanting to use Rietveld, or that I have a better alternative up my sleeve. I was just astonished to be asked to run a tiny doc patch through a code-testing service. You can use any branch name for "git cl upload". Using stuff other than master has the disadvantage that the diff might not apply for the centralized regression tests. To be sure that I've understood correctly, are the "master branch/separat branch" sections of http://lilypond.org/doc/v2.15/Documentation/contributor/commits-and-patches#uploading-a-patch-for-review ... referring to the name of the local branch on my machine, or the LP repo branch that it's based on? It just seems utterly weird that the name of the branch on my machine should matter at _all_. If it's referring to the server-side branch, then to my eyes the docs aren't clear enough in meaning ... :-) I think you are not clear about the number of merge conflicts such a style would cause when using LilyPond. It is bad enough as it is. Isn't normal practice to expect the submitter of a patch to merge or rebase from master and resolve any conflicts before it gets merged in? I'm not proposing "commit first, question later". I'm saying that the process of _submitting_ a patch or merge request should be trivial. It is. Have you actually tried it? It's not trivial if I have to install custom code in order to submit a tiny doc patch! If I were hacking on LP itself then it would surely be a small issue by comparison, but I wasn't, and I don't think small tweak-y contributions should be treated this way. There are reasons the show is run in the manner it is, and the reasons are not just because everybody except yourself is an idiot and just waited for you to tell them how to make everything simple and smooth. I certainly didn't intend to suggest that and I apologize unreservedly if it seemed that way. I could be malicious and offer you the responsibility to make everything better, but the fact is that I am not in the position to even make such an offer, and a lot of hard work and discussions have gone into working with the current system, and not all of the reasons are strictly technical but are also done in order to utilize the distribution of talent available for LilyPond best. I think if anything is clear it's that LP is the result of an extremely large amount of work by a small and very dedicated group of people. I _do_ appreciate that, very much indeed. There is no sensible way around learning the ropes before attempting to replace them. Certainly people would welcome a transition to systems better suited to git. But your hand-waving proposals not even include a substitute for the Rietveld vetting system (like, say, Gerrit) including organizing the hosting for it, let alone adapting the existing workflow and toolchains. My remarks were made in haste as I was not expecting such an extensive response to my original, short remark on the complications of the process. I'll add an extra story which may give some context to my reaction. The last time I attempted to make a contribution to LP was probably about a year ago -- there was some documentation work I was interested in doing. At this time there were some issues with the doc build system where make did not automatically pick up on changes to input files. The workaround which several helpful list members suggested was to 'touch' one of the files for the manual concerned. The problem with this workaround was that it meant that everything in the manual had to be rebuilt -- it was much less than a full doc build, but still a large build process. I wanted to make sure I hadn't missed anything, so I asked if there was really no way to get the rebuild to just cover my changes and not all the extra untouched material. ... and at this point someone chipped in with a long and fairly unpleasant lecture along the lines of, "Sure, you can read all the make documentation and go
Re: Doc build hanging (with memory leak?)
Joseph Rushton Wakeling writes: > On 14/05/12 09:46, David Kastrup wrote: >> We don't have a canonical developer, one whose personal >> branch/repository would be official for the project. > > GitHub and Launchpad both permit branches to be owned by groups as > well as individuals. I'm sure other DVCS-based code hosts do as well, > but those are the two I'm most familiar with. Does not change the problem of coordination between individuals. >> We have automated regression testing taking a _lot_ of computation >> time. We have a history of our canonical repository _breaking_. We >> have a total dearth of high-level developers, and a number of >> volunteers at lower level. > > What's the particular problem here -- is it that the regression tests > are too heavy for users to run on their own machines prior to > submitting a merge request, or is it that they can't be trusted to run > them? Both. >> If you compare the complexity of the code base and infrastructure to >> the available manpower and skill sets, the logical verdict is death >> by bitrot. The red tape is doing a surprisingly good job at keeping >> the project from rusting in its tracks in spite of the large mismatch >> between project complexity and active workers. Partly because >> sufficiently hardened red tape can be replaced by automatic >> procedures. >> >> We have a fine-grained mesh of regression tests where _lots_ of stuff >> gets caught before making it into the code base. > > Don't get me wrong. I'm not arguing against regression tests or > careful review -- I strongly support them. My objection is that the > way it's been implemented is unnecessary complicated: > > -- you ask me to install a custom program, unique to LP, just in > order to upload my code to your testing site. Why not let me upload > to a repo of my choice (GitHub, Gitorious, Bitbucket, Google Code > ...) and just submit to your testing system a repo location and > branch name from which it can pull? Come to that, why can't I just > manually upload a patch file like the one I emailed to the list? Or > create and push to a personal repo on the testing system? Have you actually used Rietveld for reviewing code? It does a job that the mentioned tools don't. > -- this custom-built program can't readily handle git branches other > than master. Yes, I can use the SHA1, but that's finnicky on the > user side, an unnecessary complication. You can use any branch name for "git cl upload". Using stuff other than master has the disadvantage that the diff might not apply for the centralized regression tests. > -- the custom-built program asks for my Google ID and password. Can > I be sure that it handles them securely? Read the source. Or don't use it. You can upload without using a custom tool IIRC. It will just cause a lot of unnecessary work, and you are complaining about unnecessary work. >> Yes, I have bursts of creativity where I bypass proper procedures for >> well-chosen subsets of patches in order to avoid getting into a >> tangle of diverging commits. But that is an exception to the rule. >> The "norm for DVCS-hosted software projects" is commit first, >> question later. > > That's not my experience of DVCS-based projects. It's perfectly > possible to run a contributor's merge request through a test suite > before accepting it, or to request them to ensure the test suite > passes before submitting such a request. What I don't see is that it's > necessary to entangle the technical means for submitting (and > tracking) the merge request with the technical means for applying the > test procedure -- or to require project-specific programs to do > either. I think you are not clear about the number of merge conflicts such a style would cause when using LilyPond. It is bad enough as it is. >> We don't have the amount of qualified cleanup personnel to deal with >> this style. > > I'm not proposing "commit first, question later". I'm saying that the > process of _submitting_ a patch or merge request should be trivial. It is. Have you actually tried it? > There also has to be some common sense involved. My _real_ objection > here is that a tiny, spur-of-the-moment contribution done as a casual > favour to the project was greeted with a request for me to go through > a complicated submission procedure. Small contributions tend to get registered by the bug squad and work their way through the queue from there. If you are interested in better ability to oversee your contribution's progress, it makes sense looking at the tools yourself. > It's a trivial documentation patch and it's hardly onerous for a core > developer to check that it's OK and just add it in. That's what happens when something gets picked up by the bug squad and entered as an issue. But there are very few "core developers" to go around, and their status of "core developer" is not because they tend to idle around and wait for the chance of coaching spur-of
Re: Doc build hanging (with memory leak?)
On 14/05/12 09:46, David Kastrup wrote: We don't have a canonical developer, one whose personal branch/repository would be official for the project. GitHub and Launchpad both permit branches to be owned by groups as well as individuals. I'm sure other DVCS-based code hosts do as well, but those are the two I'm most familiar with. We have automated regression testing taking a _lot_ of computation time. We have a history of our canonical repository _breaking_. We have a total dearth of high-level developers, and a number of volunteers at lower level. What's the particular problem here -- is it that the regression tests are too heavy for users to run on their own machines prior to submitting a merge request, or is it that they can't be trusted to run them? If you compare the complexity of the code base and infrastructure to the available manpower and skill sets, the logical verdict is death by bitrot. The red tape is doing a surprisingly good job at keeping the project from rusting in its tracks in spite of the large mismatch between project complexity and active workers. Partly because sufficiently hardened red tape can be replaced by automatic procedures. We have a fine-grained mesh of regression tests where _lots_ of stuff gets caught before making it into the code base. Don't get me wrong. I'm not arguing against regression tests or careful review -- I strongly support them. My objection is that the way it's been implemented is unnecessary complicated: -- you ask me to install a custom program, unique to LP, just in order to upload my code to your testing site. Why not let me upload to a repo of my choice (GitHub, Gitorious, Bitbucket, Google Code ...) and just submit to your testing system a repo location and branch name from which it can pull? Come to that, why can't I just manually upload a patch file like the one I emailed to the list? Or create and push to a personal repo on the testing system? -- this custom-built program can't readily handle git branches other than master. Yes, I can use the SHA1, but that's finnicky on the user side, an unnecessary complication. -- the custom-built program asks for my Google ID and password. Can I be sure that it handles them securely? Yes, I have bursts of creativity where I bypass proper procedures for well-chosen subsets of patches in order to avoid getting into a tangle of diverging commits. But that is an exception to the rule. The "norm for DVCS-hosted software projects" is commit first, question later. That's not my experience of DVCS-based projects. It's perfectly possible to run a contributor's merge request through a test suite before accepting it, or to request them to ensure the test suite passes before submitting such a request. What I don't see is that it's necessary to entangle the technical means for submitting (and tracking) the merge request with the technical means for applying the test procedure -- or to require project-specific programs to do either. We don't have the amount of qualified cleanup personnel to deal with this style. I'm not proposing "commit first, question later". I'm saying that the process of _submitting_ a patch or merge request should be trivial. That merge request can still be required to pass rigorous tests before it is accepted. There also has to be some common sense involved. My _real_ objection here is that a tiny, spur-of-the-moment contribution done as a casual favour to the project was greeted with a request for me to go through a complicated submission procedure. It's a trivial documentation patch and it's hardly onerous for a core developer to check that it's OK and just add it in. ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Doc build hanging (with memory leak?)
Joseph Rushton Wakeling writes: > On 14/05/12 07:35, Graham Percival wrote: >> On Mon, May 14, 2012 at 03:38:54AM +0200, Joseph Rushton Wakeling wrote: >>> Here you go. :-) Let me know if it needs tweaking or might be >>> better in another section of the guide. >> >> Please see the "summary for experienced developers" in the CG. > > That really seems an astonishingly complicated procedure, especially > when you compare to what is now the norm for DVCS-hosted software > projects (i.e. upload to personal branch on code-hosting service; > submit merge request). We don't have a canonical developer, one whose personal branch/repository would be official for the project. We have automated regression testing taking a _lot_ of computation time. We have a history of our canonical repository _breaking_. We have a total dearth of high-level developers, and a number of volunteers at lower level. If you compare the complexity of the code base and infrastructure to the available manpower and skill sets, the logical verdict is death by bitrot. The red tape is doing a surprisingly good job at keeping the project from rusting in its tracks in spite of the large mismatch between project complexity and active workers. Partly because sufficiently hardened red tape can be replaced by automatic procedures. We have a fine-grained mesh of regression tests where _lots_ of stuff gets caught before making it into the code base. Yes, I have bursts of creativity where I bypass proper procedures for well-chosen subsets of patches in order to avoid getting into a tangle of diverging commits. But that is an exception to the rule. The "norm for DVCS-hosted software projects" is commit first, question later. We don't have the amount of qualified cleanup personnel to deal with this style. And part of the problem is that LilyPond is monolithic rather than modular, and opposed to the Linux kernel which is claimed to suffer from the same condition, this does not just mean the memory/protection model but means a _lot_ of centralized data structures without dedicated ways of extension. And that means merge conflicts as a regular consequence of extending LilyPond, even along standard lines, making a decentralized development style less workable. Of course, some choices are also dictated by our tool chains: the review system Rietveld is based on a centralized repository view, even though it uses git internally for its working. -- David Kastrup ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Doc build hanging (with memory leak?)
On 14/05/12 07:35, Graham Percival wrote: On Mon, May 14, 2012 at 03:38:54AM +0200, Joseph Rushton Wakeling wrote: Here you go. :-) Let me know if it needs tweaking or might be better in another section of the guide. Please see the "summary for experienced developers" in the CG. That really seems an astonishingly complicated procedure, especially when you compare to what is now the norm for DVCS-hosted software projects (i.e. upload to personal branch on code-hosting service; submit merge request). ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Doc build hanging (with memory leak?)
On Mon, May 14, 2012 at 03:38:54AM +0200, Joseph Rushton Wakeling wrote: > Here you go. :-) Let me know if it needs tweaking or might be > better in another section of the guide. Please see the "summary for experienced developers" in the CG. - Graham ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Doc build hanging (with memory leak?)
On 12/05/12 17:22, James wrote:
On 12 May 2012 14:24, Joseph Rushton Wakeling
A small word to this effect might be a nice addition to the contributor
guide (I'll make a patch if you like).
Sure go ahead.
Here you go. :-) Let me know if it needs tweaking or might be better in another
section of the guide.
>From b166358e76bfcdc982fb76d621be81751fc613db Mon Sep 17 00:00:00 2001
From: Joseph Rushton Wakeling
Date: Mon, 14 May 2012 03:21:03 +0200
Subject: [PATCH] CG tweak: some extra info on make doc and doc build times.
---
Documentation/included/compile.itexi | 17 +
1 file changed, 17 insertions(+)
diff --git a/Documentation/included/compile.itexi
b/Documentation/included/compile.itexi
index e95cf26..4b87ce1 100644
--- a/Documentation/included/compile.itexi
+++ b/Documentation/included/compile.itexi
@@ -456,6 +456,23 @@ targets, run:
make help
@end example
+For example, to build documentation, you can use
+@example
+make doc
+@end example
+
+or to build only the PDF documentation,
+
+@example
+make doc-stage-1
+@end example
+
+Documentation builds can take much longer than building LilyPond itself.
+Some individual commands issued during the make process may take as long
+as an hour to complete on older single-core machines, so don't panic if
+for a long time it looks like nothing is happening! Be patient, and the
+build will complete.
+
TODO: Describe what @command{make} actually does.
--
1.7.9.5
___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Doc build hanging (with memory leak?)
hello, On 12 May 2012 14:24, Joseph Rushton Wakeling wrote: > On 12/05/12 14:29, James wrote: >> >> Assuming you are building from current master then make doc does >> compile as all new checkins go to staging tree first and sit there >> while a script runs (as it happens on my computer) that compiles >> staging through all the tests and if it passes them all (and that >> includes a make doc) it merges the staging branch to master. > > > Yea, it built fine in the end once dblatex was installed. It was just scary > having an initial command issued by make which just sat there for 10+ > minutes doing nothing while the memory usage went up and up. Completely > different to what I'd experienced with earlier builds of the docs. > > A small word to this effect might be a nice addition to the contributor > guide (I'll make a patch if you like). Sure go ahead. > >> My only advice having built docs and gone through all the pain of a >> non-programmer building doc is to use an out of tree build > > > Yes, I followed that advice from the guide. Is it a recent addition? I > don't remember it from the last time I looked at the guide, which was > probably 6+ months back. Not sure, but the CG is constantly changing, or rather constantly being 'added' to and evolving. There have been *significant* under-the-hood changes to the whole build system not least Phil's work on reducing the sheer number of output lines - I recall that you can increase the logging if you want to (I don't and haven't bothered to look that up in he CG to see if it exists but I think it does) - and some issues with translations that caused extraneous/erroneous errors that would increase the time it takes to build doc. You can also now use the LANG option to just build the language you need (again see the CG). > >> I've found it makes this so much more easier especially when you have >> a slower machine where make doc can take an age. > > > On my quad-core i7 it took about 35 minutes in total to build doc-stage-1, > the first 10-12 minutes of which were the seeming "hang" I reported. I > remember it used to take much, much, much longer on the machine I had 6 > months back, but there was plenty flashing past in the terminal window to > make it clear something was happening! Yes and most of it was warnings and errors that were 'harmless'. Most of the heavy lifting with regard to fixing this was done by multi-core CPU machines (Phil and myself) which have helped take the tedium of checking building doc and the like allowing for much more 'quick' testing of fixes to the build system - we both manage build a brand new make doc from scratch in less than 15 minutes! So can forget sometimes what building doc looks like on a less powered machine. > > If I understand right, that first documentation build command is making all > the PNG, PS and PDF files for LP snippets -- right? -- which if my > estimation is correct are a bit over 10,000 in number. So no wonder it > takes so long. Something like that yes. As a non-programmer I just 'test' what is done and give the results. I don't pretend to understand what the changes are ;) Regards James ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Doc build hanging (with memory leak?)
On Sat, May 12, 2012 at 01:56:33PM +0200, Joseph Rushton Wakeling wrote: > On 12/05/12 13:37, David Kastrup wrote: > >and that does not quite work. It would appear that the error handling > >for a missing texi2html script is totally awful. I'd install texi2html > >and rerun configure. > > Texi2html was already installed, but re-running configure revealed > that dblatex was missing (for some reason aptitude build-dep > lilypond didn't include that). huh, thanks for the report. http://code.google.com/p/lilypond/issues/detail?id=2529 - Graham ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Doc build hanging (with memory leak?)
On 12/05/12 14:29, James wrote: Assuming you are building from current master then make doc does compile as all new checkins go to staging tree first and sit there while a script runs (as it happens on my computer) that compiles staging through all the tests and if it passes them all (and that includes a make doc) it merges the staging branch to master. Yea, it built fine in the end once dblatex was installed. It was just scary having an initial command issued by make which just sat there for 10+ minutes doing nothing while the memory usage went up and up. Completely different to what I'd experienced with earlier builds of the docs. A small word to this effect might be a nice addition to the contributor guide (I'll make a patch if you like). My only advice having built docs and gone through all the pain of a non-programmer building doc is to use an out of tree build Yes, I followed that advice from the guide. Is it a recent addition? I don't remember it from the last time I looked at the guide, which was probably 6+ months back. I've found it makes this so much more easier especially when you have a slower machine where make doc can take an age. On my quad-core i7 it took about 35 minutes in total to build doc-stage-1, the first 10-12 minutes of which were the seeming "hang" I reported. I remember it used to take much, much, much longer on the machine I had 6 months back, but there was plenty flashing past in the terminal window to make it clear something was happening! If I understand right, that first documentation build command is making all the PNG, PS and PDF files for LP snippets -- right? -- which if my estimation is correct are a bit over 10,000 in number. So no wonder it takes so long. Thanks to all for your advice and explanations. :-) -- Joe ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Doc build hanging (with memory leak?)
Joseph, On 12 May 2012 12:56, Joseph Rushton Wakeling wrote: > On 12/05/12 13:37, David Kastrup wrote: >> >> You are running a command >> >> - echo texi2html not found >> >> and that does not quite work. It would appear that the error handling >> for a missing texi2html script is totally awful. I'd install texi2html >> and rerun configure. > > > Texi2html was already installed, but re-running configure revealed that > dblatex was missing (for some reason aptitude build-dep lilypond didn't > include that). Seems to be working now (can't be certain, it's still > building:-). > Assuming you are building from current master then make doc does compile as all new checkins go to staging tree first and sit there while a script runs (as it happens on my computer) that compiles staging through all the tests and if it passes them all (and that includes a make doc) it merges the staging branch to master. So you can pretty much safely assume that master is compilable, and that any issues you have are probably down to the need to simple make clean, re-rerun autogen.sh and configure then make and so on. My only advice having built docs and gone through all the pain of a non-programmer building doc is to use an out of tree build (i.e. make a build dir in your lilypond repository, run ./autogen.sh --noconfigure outside of the build dir, then cd into it and run ../configure; make ; make doc that way. If you get any odditites you can simply rm your build and start again without worrying too much about if you have made a mess of your tree and make clean is not working. This is all mentioned (if you are not already doing this) here: http://lilypond.org/doc/v2.15/Documentation/contributor/compiling-with-lilydev and although this is for specifically using the Ubuntu based iso we have available, the same principles apply. I've found it makes this so much more easier especially when you have a slower machine where make doc can take an age. James ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Doc build hanging (with memory leak?)
On 12/05/12 13:37, David Kastrup wrote: You are running a command - echo texi2html not found and that does not quite work. It would appear that the error handling for a missing texi2html script is totally awful. I'd install texi2html and rerun configure. Texi2html was already installed, but re-running configure revealed that dblatex was missing (for some reason aptitude build-dep lilypond didn't include that). Seems to be working now (can't be certain, it's still building:-). ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Doc build hanging (with memory leak?)
Joseph Rushton Wakeling writes: > /home/joseph/code/lily/build/scripts/build/out/run-and-check > "DEPTH=../../.. AJAX_SEARCH= TOP_SRC_DIR=/home/joseph/code/lily > PERL_UNICODE=SD - echo texi2html not found --error-limit=0 [...] > The logfile reads: > >build/scripts/build/out/run-and-check: 1: eval: -: not found > > What's going wrong here? You are running a command - echo texi2html not found and that does not quite work. It would appear that the error handling for a missing texi2html script is totally awful. I'd install texi2html and rerun configure. -- David Kastrup ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Doc build hanging (with memory leak?)
On 12/05/12 11:59, Phil Holmes wrote: The doc build now issues far less "chatter" than it used to. With a single core machine, it could well go an hour without a single message. If it's using CPU and taking memory, be patient. If there's no response after a day, let us know. OK, cool. It's nice that the volume of make chatter has been reduced, but quite scary to have it reduced quite so much! The build on my system starts emitting further messages after about 10-12 minutes, fairly quickly once that first build command has passed. But it falls over later: -- make[4]: Entering directory `/home/joseph/code/lily/build/input/regression/lilypond-book' /home/joseph/code/lily/./input/regression/lilypond-book/GNUmakefile:33: warning: overriding commands for target `out-www/collated-files.list' /home/joseph/code/lily/./make/lysdoc-rules.make:6: warning: ignoring old commands for target `out-www/collated-files.list' make[4]: Circular out-www/collated-files.list <- suffix-tely.tely dependency dropped. make[4]: Circular out-www/collated-files.list <- texinfo-include-file.tely dependency dropped. make[4]: Circular out-www/collated-files.list <- texinfo-include-language-detection.tely dependency dropped. make[4]: Circular out-www/collated-files.list <- texinfo-language-detection.tely dependency dropped. make[4]: Circular out-www/collated-files.list <- texinfo-musicxml-file-options.tely dependency dropped. make[4]: Circular out-www/collated-files.list <- texinfo-musicxml-file.tely dependency dropped. make[4]: Circular out-www/collated-files.list <- texinfo-papersize-docs.tely dependency dropped. make[4]: Circular out-www/texinfo-include-file.texi <- texinfo-include-file.tely dependency dropped. make[4]: Circular out-www/texinfo-include-language-detection.texi <- texinfo-include-language-detection.tely dependency dropped. make[4]: Circular out-www/texinfo-language-detection.texi <- texinfo-language-detection.tely dependency dropped. make[4]: Circular out-www/texinfo-musicxml-file-options.texi <- texinfo-musicxml-file-options.tely dependency dropped. make[4]: Circular out-www/texinfo-musicxml-file.texi <- texinfo-musicxml-file.tely dependency dropped. make[4]: Circular out-www/texinfo-papersize-docs.texi <- texinfo-papersize-docs.tely dependency dropped. /home/joseph/code/lily/build/scripts/build/out/run-and-check "DEPTH=../../.. AJAX_SEARCH= TOP_SRC_DIR=/home/joseph/code/lily PERL_UNICODE=SD - echo texi2html not found --error-limit=0 --I=/home/joseph/code/lily/input/regression/lilypond-book --I=./out-www -I /home/joseph/code/lily/Documentation --I=/home/joseph/code/lily/build/./out-www/xref-maps --init-file=/home/joseph/code/lily/Documentation/lilypond-texi2html.init --output=out-www/suffix-tely.html out-www/suffix-tely.texi" "suffix-tely.texilog.log" Please check the logfile suffix-tely.texilog.log for errors make[4]: *** [out-www/suffix-tely.html] Error 127 make[4]: Leaving directory `/home/joseph/code/lily/build/input/regression/lilypond-book' make[3]: *** [WWW-1] Error 2 make[3]: Leaving directory `/home/joseph/code/lily/build/input/regression' make[2]: *** [WWW-1] Error 2 make[2]: Leaving directory `/home/joseph/code/lily/build/input' make[1]: *** [WWW-1] Error 2 make[1]: Leaving directory `/home/joseph/code/lily/build' make: *** [doc-stage-1] Error 2 -- The logfile reads: build/scripts/build/out/run-and-check: 1: eval: -: not found What's going wrong here? This is just a doc-stage-1 build attempt. I tried rerunning it with single-core make just to make sure it wasn't falling over from this, but no joy, same result. ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Doc build hanging (with memory leak?)
- Original Message - From: "Joseph Rushton Wakeling" To: Sent: Saturday, May 12, 2012 8:37 AM Subject: Doc build hanging (with memory leak?) Hello all, I've successfully build Lilypond itself from source, but when I try to make doc, the build hangs on the first file it attempts to compile: make[3]: Entering directory `/home/joseph/code/lily/build/input/regression' LILYPOND_VERSION=2.15.39 /usr/bin/python /home/joseph/code/lily/scripts/lilypond-book.py -I /home/joseph/code/lily/input/regression/ -I ./out-www -I /home/joseph/code/lily/input -I /home/joseph/code/lily/Documentation -I /home/joseph/code/lily/Documentation/snippets -I /home/joseph/code/lily/input/regression/ -I /home/joseph/code/lily/Documentation/included/ -I /home/joseph/code/lily/build/mf/out/ -I /home/joseph/code/lily/build/mf/out/ -I /home/joseph/code/lily/Documentation/pictures -I /home/joseph/code/lily/build/Documentation/pictures/./out-www --process='/home/joseph/code/lily/build/out/bin/lilypond -I /home/joseph/code/lily/input/regression/ -I ./out-www -I /home/joseph/code/lily/input -I /home/joseph/code/lily/Documentation -I /home/joseph/code/lily/Documentation/snippets -I /home/joseph/code/lily/input/regression/ -I /home/joseph/code/lily/Documentation/included/ -I /home/joseph/code/lily/build/mf/out/ -I /home/joseph/code/lily/build/mf/out/ -I /home/joseph/code/lily/Documentation/pictures -I /home/joseph/code/lily/build/Documentation/pictures/./out-www -dbackend=eps --formats=ps,png,pdf -dinclude-eps-fonts -dgs-load-fonts --header=doctitle --header=doctitlecs --header=doctitlede --header=doctitlees --header=doctitlefr --header=doctitlehu --header=doctitleit --header=doctitleja --header=doctitlenl --header=doctitlezh --header=texidoc --header=texidoccs --header=texidocde --header=texidoces --header=texidocfr --header=texidochu --header=texidocit --header=texidocja --header=texidocnl --header=texidoczh -dcheck-internal-types -ddump-signatures -danti-alias-factor=2' --output=./out-www --format=texi-html --loglevel=WARN --lily-output-dir /home/joseph/code/lily/build/out/lybook-db --redirect-lilypond-output out-www/collated-files.tely langdefs.py: warning: lilypond-doc gettext domain not found. Checking 'top' shows that the memory consumption of Lilypond is steadily increasing over time, but nothing appears to actually be happening. This is completely at odds with the last doc builds I attempted for Lilypond, probably late last year. Is this a bug, or is the doc build system just operating differently? Thanks and best wishes, The doc build now issues far less "chatter" than it used to. With a single core machine, it could well go an hour without a single message. If it's using CPU and taking memory, be patient. If there's no response after a day, let us know. -- Phil Holmes ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Doc build hanging (with memory leak?)
Hello all, I've successfully build Lilypond itself from source, but when I try to make doc, the build hangs on the first file it attempts to compile: make[3]: Entering directory `/home/joseph/code/lily/build/input/regression' LILYPOND_VERSION=2.15.39 /usr/bin/python /home/joseph/code/lily/scripts/lilypond-book.py -I /home/joseph/code/lily/input/regression/ -I ./out-www -I /home/joseph/code/lily/input -I /home/joseph/code/lily/Documentation -I /home/joseph/code/lily/Documentation/snippets -I /home/joseph/code/lily/input/regression/ -I /home/joseph/code/lily/Documentation/included/ -I /home/joseph/code/lily/build/mf/out/ -I /home/joseph/code/lily/build/mf/out/ -I /home/joseph/code/lily/Documentation/pictures -I /home/joseph/code/lily/build/Documentation/pictures/./out-www --process='/home/joseph/code/lily/build/out/bin/lilypond -I /home/joseph/code/lily/input/regression/ -I ./out-www -I /home/joseph/code/lily/input -I /home/joseph/code/lily/Documentation -I /home/joseph/code/lily/Documentation/snippets -I /home/joseph/code/lily/input/regression/ -I /home/joseph/code/lily/Documentation/included/ -I /home/joseph/code/lily/build/mf/out/ -I /home/joseph/code/lily/build/mf/out/ -I /home/joseph/code/lily/Documentation/pictures -I /home/joseph/code/lily/build/Documentation/pictures/./out-www -dbackend=eps --formats=ps,png,pdf -dinclude-eps-fonts -dgs-load-fonts --header=doctitle --header=doctitlecs --header=doctitlede --header=doctitlees --header=doctitlefr --header=doctitlehu --header=doctitleit --header=doctitleja --header=doctitlenl --header=doctitlezh --header=texidoc --header=texidoccs --header=texidocde --header=texidoces --header=texidocfr --header=texidochu --header=texidocit --header=texidocja --header=texidocnl --header=texidoczh -dcheck-internal-types -ddump-signatures -danti-alias-factor=2' --output=./out-www --format=texi-html --loglevel=WARN --lily-output-dir /home/joseph/code/lily/build/out/lybook-db --redirect-lilypond-output out-www/collated-files.tely langdefs.py: warning: lilypond-doc gettext domain not found. Checking 'top' shows that the memory consumption of Lilypond is steadily increasing over time, but nothing appears to actually be happening. This is completely at odds with the last doc builds I attempted for Lilypond, probably late last year. Is this a bug, or is the doc build system just operating differently? Thanks and best wishes, -- Joe ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
