debug spam (was: GOP-PROP 5: build system output (probable 2?))
On Fri, Jul 29, 2011 at 11:36:38PM +0100, Neil Puttock wrote:
> On 29 July 2011 17:20, Graham Percival wrote:
>
> > Could somebody get rid of these already? They're left-over from
> > Valentin's note name changes from Dec 2010 or so;
>
> They come from parsing string-tunings-init.ly.
> > they were debugging messages which were supposed to be
> > removed, but weren't completely removed.
> No, parsing a string via ly:parser-parse-string (which is ultimately
> what the hash-extend syntax for parsing .ly code inside scheme via #{
> ... #} uses) causes the lexer to set new input called `'.
Why do we output lexer input in this case, but not other cases?
I cannot see how 20 [] (whether or not they're on
newlines) could in any way help debugging. I think the underlying
architecture should be reworked so that we don't see that stuff.
Bottom line: C++ only prints stuff with printf() and cout/cerr;
guile on prints stuff with (display...) and related functions. We
have control over all that code. This isn't an urgent problem by
any means, but I think it *is* a problem, and if fixing this
requires some low-level changes, so be it.
Cheers,
- Graham
___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Fwd: Re: GOP-PROP 5: build system output (probable 2?)
Am Samstag, 30. Juli 2011, 00:42:58 schrieb Reinhold Kainhofer: > Am Freitag, 29. Juli 2011, 18:20:25 schrieben Sie: > > On Fri, Jul 29, 2011 at 12:30:24PM +0200, Reinhold Kainhofer wrote: > > > [/home/reinhold/lilypond/out/share/lilypond/current/ly/string-tunings > > > - > > > > > > init.ly > > > Using `nederlands' note names... > > > [] > > > [] > > > > Could somebody get rid of these already? They're left-over from > > Valentin's note name changes from Dec 2010 or so; they were > > debugging messages which were supposed to be removed, but weren't > > completely removed. > > I don't see an easy way to get rid of them. The only possibility I see is > to completely remove the debug output in (notice the second argument!) > Includable_lexer::new_input (string name, string data, Sources *sources) > which is the function to include the given data. In that case, we don't > have any filename to print, so we might simply remove the debug output > altogether. On second thought, we might still print [], but simply not add a linebreak there? My --loglevel patch allows this to be tweaked (in pariticular, call debug_output(..., false) instad of debug_output(...) ). Cheers, Reinhold -- -- Reinhold Kainhofer, reinh...@kainhofer.com, http://reinhold.kainhofer.com/ * Financial & Actuarial Math., Vienna Univ. of Technology, Austria * http://www.fam.tuwien.ac.at/, DVR: 0005886 * LilyPond, Music typesetting, http://www.lilypond.org ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Fwd: Re: GOP-PROP 5: build system output (probable 2?)
Sorry, this reply went only to Graham by accident. Here it is for lilypond- devel: -- Weitergeleitete Nachricht -- Betreff: Re: GOP-PROP 5: build system output (probable 2?) Datum: Freitag, 29. Juli 2011, 23:07:11 Von: Reinhold Kainhofer An: Graham Percival Am Freitag, 29. Juli 2011, 18:20:25 schrieben Sie: > On Fri, Jul 29, 2011 at 12:30:24PM +0200, Reinhold Kainhofer wrote: > > [/home/reinhold/lilypond/out/share/lilypond/current/ly/string-tunings- > > > > init.ly > > Using `nederlands' note names... > > [] > > [] > > Could somebody get rid of these already? They're left-over from > Valentin's note name changes from Dec 2010 or so; they were > debugging messages which were supposed to be removed, but weren't > completely removed. AFAICS, they are caused by Includable_lexer::new_input, where the note names are apparently passed as a string to be included into the parsed file rather than as a file... I don't see an easy way to get rid of them. The only possibility I see is to completely remove the debug output in (notice the second argument!) Includable_lexer::new_input (string name, string data, Sources *sources) which is the function to include the given data. In that case, we don't have any filename to print, so we might simply remove the debug output altogether. But then we have a problem with the closing "]" in Includable_lexer::close_input, since at that place we don't know whether the input was created from a string or from a file... Cheers, Reinhold -- -- Reinhold Kainhofer, reinh...@kainhofer.com, http://reinhold.kainhofer.com/ * Financial & Actuarial Math., Vienna Univ. of Technology, Austria * http://www.fam.tuwien.ac.at/, DVR: 0005886 * LilyPond, Music typesetting, http://www.lilypond.org - -- -- Reinhold Kainhofer, reinh...@kainhofer.com, http://reinhold.kainhofer.com/ * Financial & Actuarial Math., Vienna Univ. of Technology, Austria * http://www.fam.tuwien.ac.at/, DVR: 0005886 * LilyPond, Music typesetting, http://www.lilypond.org ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: GOP-PROP 5: build system output (probable 2?)
On 29 July 2011 17:20, Graham Percival wrote:
> Could somebody get rid of these already? They're left-over from
> Valentin's note name changes from Dec 2010 or so;
They come from parsing string-tunings-init.ly.
> they were
> debugging messages which were supposed to be removed, but weren't
> completely removed.
No, parsing a string via ly:parser-parse-string (which is ultimately
what the hash-extend syntax for parsing .ly code inside scheme via #{
... #} uses) causes the lexer to set new input called `'. To
remove this, you'd probably have to have a flag set when parsing
strings (or included strings) to silence the verbose output.
Cheers,
Neil
___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: GOP-PROP 5: build system output (probable 2?)
On Fri, Jul 29, 2011 at 06:38:53PM +0100, Phil Holmes wrote: > - Original Message - From: "Reinhold Kainhofer" > > >Yes, that would be *extremely* helpful (not only for the lilypond > >documentation, but also to other lilypond-book users). The only > >question is: > >who will implement it? ;-) > > This is the sort of thing I'm looking at, once the discussion about > what is needed is complete. I want to emphasize this point. We have somebody willing to work on this stuff. But I don't want him to spend 10 hours preparing a patch to do XYZ, only to have somebody say "no, we shouldn't XYZ at all!". Of course reviewers will discover technical flaws; that's expected. But I don't want reviewers to disagree about the general principles underlying this work. If I encouraged him to work on this stuff without some level of certainty that his work would be well received, I would fail horribly as a mentor. My proposed guidelines for our "ideal" build system (at least, "ideal" without a massive rewrite), is here: http://lilypond.org/~graham/gop/gop_5.html Leaving aside technical implementation details, are there any problems with those guidelines? If you disagree with a point (and if I have not convinced you that you should accept the proposal), please speak up. If you would like to add a point, please speak up. This policy discussion has been dragging on for almost a month, and while that's happening we have a skilled contributor sitting on his thumbs. Cheers, - Graham ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: GOP-PROP 5: build system output (probable 2?)
- Original Message - From: "Reinhold Kainhofer" To: "Werner LEMBERG" ; Sent: Friday, July 29, 2011 6:31 PM Subject: Re: GOP-PROP 5: build system output (probable 2?) Am Freitag, 29. Juli 2011, 18:56:36 schrieben Sie: > However, in the docs build, we are not interested in how lilypond > works internally, but rather where a doc build fails due to bad > input in a .ly or .tely file. I suggest a different route: Normally, after an error message has been emitted, a developer wants to debug the problem. In most cases, the problem is with the lilypond binary itself, That's where I would disagree. Typically a *doc* build fails due to a typo in the documentation (i.e. wrong lilypond code in an example included in the docs). All problems with the lilypond binary itself should have already failed the regtests (in theory, of course). What about reducing the verbosity as much as possible, but make lilypond-book emit something like: Error `foo' encountered while processing file `file-XXX'! lilypond's error message: blablabla The used command line was FOO=bar lilypond --pdf -d... -d... file-XXX.ly A command line to debug the problem can be found in file `file-XXX-debug.sh'. if an error is happening? Yes, that would be *extremely* helpful (not only for the lilypond documentation, but also to other lilypond-book users). The only question is: who will implement it? ;-) This is the sort of thing I'm looking at, once the discussion about what is needed is complete. -- Phil Holmes ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: GOP-PROP 5: build system output (probable 2?)
Am Freitag, 29. Juli 2011, 18:56:36 schrieben Sie: > > However, in the docs build, we are not interested in how lilypond > > works internally, but rather where a doc build fails due to bad > > input in a .ly or .tely file. > > I suggest a different route: Normally, after an error message has been > emitted, a developer wants to debug the problem. In most cases, the > problem is with the lilypond binary itself, That's where I would disagree. Typically a *doc* build fails due to a typo in the documentation (i.e. wrong lilypond code in an example included in the docs). All problems with the lilypond binary itself should have already failed the regtests (in theory, of course). > What about reducing the verbosity as much as possible, but make > lilypond-book emit something like: > >Error `foo' encountered while processing file `file-XXX'! >lilypond's error message: blablabla >The used command line was > > FOO=bar lilypond --pdf -d... -d... file-XXX.ly > >A command line to debug the problem can be found in file >`file-XXX-debug.sh'. > > if an error is happening? Yes, that would be *extremely* helpful (not only for the lilypond documentation, but also to other lilypond-book users). The only question is: who will implement it? ;-) Cheers, Reinhold -- -- Reinhold Kainhofer, reinh...@kainhofer.com, http://reinhold.kainhofer.com/ * Financial & Actuarial Math., Vienna Univ. of Technology, Austria * http://www.fam.tuwien.ac.at/, DVR: 0005886 * LilyPond, Music typesetting, http://www.lilypond.org ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: GOP-PROP 5: build system output (probable 2?)
> However, I have failed and still fail to see where the lilypond > internals printed with --verbose can be helpful in any way during > the docs build. Those verbose debug messages are useful for > debugging a lilypond bug. Yep. > However, in the docs build, we are not interested in how lilypond > works internally, but rather where a doc build fails due to bad > input in a .ly or .tely file. I suggest a different route: Normally, after an error message has been emitted, a developer wants to debug the problem. In most cases, the problem is with the lilypond binary itself, so it would be most convenient if there is an easy way to find out how to call lilypond exactly. Currently, lilypond-book hides this very effectively, or the calling command is mentioned 10 lines above in the log file... What about reducing the verbosity as much as possible, but make lilypond-book emit something like: Error `foo' encountered while processing file `file-XXX'! lilypond's error message: blablabla The used command line was FOO=bar lilypond --pdf -d... -d... file-XXX.ly A command line to debug the problem can be found in file `file-XXX-debug.sh'. if an error is happening? Werner ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: GOP-PROP 5: build system output (probable 2?)
Am Freitag, 29. Juli 2011, 18:05:40 schrieben Sie: > - Original Message - > From: "Reinhold Kainhofer" > > However, I have failed and still fail to see where the lilypond internals > > printed with --verbose can be helpful in any way during the docs build. > > Those > > verbose debug messages are useful for debugging a lilypond bug. However, > > in > > the docs build, we are not interested in how lilypond works internally, > > but > > rather where a doc build fails due to bad input in a .ly or .tely file. > > Well, I clearly agree. However, just pushing the output to log files > effectively does the same thing. I don't think so. You don't easily get the error / warning messages printed directly on the console without wading through the log files. Having the important output (and only the important output) directly at the console speeds up debugging immensely. > > The loglevel would then be set as 'lilypond --log=WARN file.ly', with > > PROGRESS > > (or INFO) being the default, i.e. no change to the current output. > > Think this risks over-gilding the lily, TBH. Personally, I've never used > verbose. We already have an option to suppress output/direct to > logfile -dgui. I should mention that I'm now no longer talking about the lilypond build system, but as a lilypond user, who writes large orchestral / choir scores (sometimes 12 movements with 23 instruments, resulting in 23 instrumental scores, choir score, full score, piano reduction etc.). There, I'm encountering the same problems as the lilypond build system, though. I also have not used --verbose at all. To the contrary, I often wished that I could suppress the progress messages and just get the warnings / errors. I have my own makefiles that call lilypond on all score files. Now, unfortunately, each first prints out >300 "Interpreting music..." lines. So, when I simply want to look at which warnings each of the instrumental scores generated, I have to wade through thousands of lines, which usually don't fit in the output buffer of the console window any more. If lilypond has an option to suppress all progress messages and print just the warnings and errors, that would make my life as a music publisher much easier. That's my real intention here. Redirecting all output to a log file only moves the problem to wading through a logfile, which is cluttered with progress messages. I should also mention that internally in the code we already distinguish error, warning, progress and debug messages (ly:error, ly:warning, ly:progress, etc.). The only thing that is missing is a command-line option to let the user choose which of those (s)he wants to see. Such a patch is pretty straightforward and a matter of hours. Of course, that no longer belongs here in the GOP-PROP 5 (build system output), but is a general feature request for lilypond. The build system could then use it, but that's a different matter. > > Since we print all messages to stderr, there is no way to filter the > > output, > > so we need to add a command line option to lilypond to prevent printing > > undesired output in the first place (e.g. I have makefiles that call > > lilypond > > on several different huge files; there I'm not interested in the progress > > messages, just in the warnings and error messages. The --log=... option > > would > > allow exactly this). > > I remain of the view that this is wrong, and at some point there's likely > to be a GOP discussion. FWIW make sends progress to stdout and error to > stderr... Hehe, but lilypond prints everything to stderr, so all progress will end up in stderr, even when lilypond is run through make... Cheers, Reinhold -- -- Reinhold Kainhofer, reinh...@kainhofer.com, http://reinhold.kainhofer.com/ * Financial & Actuarial Math., Vienna Univ. of Technology, Austria * http://www.fam.tuwien.ac.at/, DVR: 0005886 * LilyPond, Music typesetting, http://www.lilypond.org ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: GOP-PROP 5: build system output (probable 2?)
On Fri, Jul 29, 2011 at 12:30:24PM +0200, Reinhold Kainhofer wrote: > [/home/reinhold/lilypond/out/share/lilypond/current/ly/string-tunings- > init.ly > Using `nederlands' note names... > [] ... > [] Could somebody get rid of these already? They're left-over from Valentin's note name changes from Dec 2010 or so; they were debugging messages which were supposed to be removed, but weren't completely removed. Cheers, - Graham ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: GOP-PROP 5: build system output (probable 2?)
- Original Message -
From: "Reinhold Kainhofer"
To: "Phil Holmes"
Cc:
Sent: Friday, July 29, 2011 4:46 PM
Subject: Re: GOP-PROP 5: build system output (probable 2?)
Am Freitag, 29. Juli 2011, 12:55:09 schrieb Phil Holmes:
- Original Message -
> Currently, the doc build is calling lilypond in verbose mode, creating
> thousands of unnecessary lines like
Reinhold - I've been looking at the build system in some depth and am
very
well aware of this. I suggested turning off verbose mode and was told
that
it was sometimes useful to see whay a build had failed.
Yes, I was told the same a long while ago, when I was also complaining
about
the verbosity. See e.g. the discussion:
http://www.mail-archive.com/lilypond-devel@gnu.org/msg17212.html
http://www.mail-archive.com/lilypond-devel@gnu.org/msg17243.html
However, I have failed and still fail to see where the lilypond internals
printed with --verbose can be helpful in any way during the docs build.
Those
verbose debug messages are useful for debugging a lilypond bug. However,
in
the docs build, we are not interested in how lilypond works internally,
but
rather where a doc build fails due to bad input in a .ly or .tely file.
Well, I clearly agree. However, just pushing the output to log files
effectively does the same thing.
Anyway, I think that the real problem is a layer deeper and that lilypond
itself should get better error/warning/debug options.
In particular, I'd like to propose several loglevels (in increasing
verbosity,
each one including messages from the ones above, too):
-) NONE: No output at all, just a negative return code from the binary
-) ERROR: output from error/programming_error (ly:error)
-) WARN: output from current warning (ly:warning)
-) PROGRESS: output from current progress_indication/success (ly:progress)
-) INFO: output from current message (ly:message) function,
i.e. what lilypond prints out currently
-) DEBUG: Everything currently wrapped in "if be_verbose_global {...}",
i.e. what --verbose does currently
-) FULLDEBUG
The loglevel would then be set as 'lilypond --log=WARN file.ly', with
PROGRESS
(or INFO) being the default, i.e. no change to the current output.
Think this risks over-gilding the lily, TBH. Personally, I've never used
verbose. We already have an option to suppress output/direct to
logfile -dgui.
Since we print all messages to stderr, there is no way to filter the
output,
so we need to add a command line option to lilypond to prevent printing
undesired output in the first place (e.g. I have makefiles that call
lilypond
on several different huge files; there I'm not interested in the progress
messages, just in the warnings and error messages. The --log=... option
would
allow exactly this).
I remain of the view that this is wrong, and at some point there's likely to
be a GOP discussion. FWIW make sends progress to stdout and error to
stderr...
I think it would be really easy to implement those loglevels in LilyPond
(actually, I have already started on it), because warn.cc already has
basically the right structure. It just ignores whether a message is
generated
by the error, warning or progress_indication functions.
The next step could then be a prettification of the lilypond-book output:
We
can add similar loglevels there, and even set a different loglevel for
lilypond invocations (e.g. --lilypond-loglevel=...). When running
lilypond-
book, I'm not really interested in the progress messages for each included
snippet. I just want to know which snippet is current processed.
What do you think?
GOP
--
Phil Holmes
___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: GOP-PROP 5: build system output (probable 2?)
Am Freitag, 29. Juli 2011, 12:55:09 schrieb Phil Holmes:
> - Original Message -
> > Currently, the doc build is calling lilypond in verbose mode, creating
> > thousands of unnecessary lines like
>
> Reinhold - I've been looking at the build system in some depth and am very
> well aware of this. I suggested turning off verbose mode and was told that
> it was sometimes useful to see whay a build had failed.
Yes, I was told the same a long while ago, when I was also complaining about
the verbosity. See e.g. the discussion:
http://www.mail-archive.com/lilypond-devel@gnu.org/msg17212.html
http://www.mail-archive.com/lilypond-devel@gnu.org/msg17243.html
However, I have failed and still fail to see where the lilypond internals
printed with --verbose can be helpful in any way during the docs build. Those
verbose debug messages are useful for debugging a lilypond bug. However, in
the docs build, we are not interested in how lilypond works internally, but
rather where a doc build fails due to bad input in a .ly or .tely file.
Printing the lilypond internals in the docs build does not help you find the
typo in the docs. To the contrary, it might hide useful warning and error
messages with excessive debug output.
Problems in lilypond itself will/should be caught with make or 'make check',
not with 'make doc'.
Anyway, I think that the real problem is a layer deeper and that lilypond
itself should get better error/warning/debug options.
In particular, I'd like to propose several loglevels (in increasing verbosity,
each one including messages from the ones above, too):
-) NONE: No output at all, just a negative return code from the binary
-) ERROR: output from error/programming_error (ly:error)
-) WARN: output from current warning (ly:warning)
-) PROGRESS: output from current progress_indication/success (ly:progress)
-) INFO: output from current message (ly:message) function,
i.e. what lilypond prints out currently
-) DEBUG: Everything currently wrapped in "if be_verbose_global {...}",
i.e. what --verbose does currently
-) FULLDEBUG
The loglevel would then be set as 'lilypond --log=WARN file.ly', with PROGRESS
(or INFO) being the default, i.e. no change to the current output.
Since we print all messages to stderr, there is no way to filter the output,
so we need to add a command line option to lilypond to prevent printing
undesired output in the first place (e.g. I have makefiles that call lilypond
on several different huge files; there I'm not interested in the progress
messages, just in the warnings and error messages. The --log=... option would
allow exactly this).
I think it would be really easy to implement those loglevels in LilyPond
(actually, I have already started on it), because warn.cc already has
basically the right structure. It just ignores whether a message is generated
by the error, warning or progress_indication functions.
The next step could then be a prettification of the lilypond-book output: We
can add similar loglevels there, and even set a different loglevel for
lilypond invocations (e.g. --lilypond-loglevel=...). When running lilypond-
book, I'm not really interested in the progress messages for each included
snippet. I just want to know which snippet is current processed.
What do you think?
Cheers,
Reinhold
--
--
Reinhold Kainhofer, reinh...@kainhofer.com, http://reinhold.kainhofer.com/
* Financial & Actuarial Math., Vienna Univ. of Technology, Austria
* http://www.fam.tuwien.ac.at/, DVR: 0005886
* LilyPond, Music typesetting, http://www.lilypond.org
___
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: GOP-PROP 5: build system output (probable 2?)
- Original Message - From: "Francisco Vila" To: "Reinhold Kainhofer" Cc: Sent: Friday, July 29, 2011 11:45 AM Subject: Re: GOP-PROP 5: build system output (probable 2?) 2011/7/29 Reinhold Kainhofer : > The other thing is that all commands called by make are echoed on the > console, > always including several lines of include pathes. While this might sound > useful, in fact it isn't because the exact command does not help you. > make > seems to set some env variables, too, so exactly duplicating the command > called in make does not work. In particular, the Python includes cannot > be > found. +1 +1 I'm doing almost exactly this at present, solving the "missing pictures in web" bug. A general solution to replicate the python command you see as output from make is to use $HOME/lilypond-git/build/out/bin/ as the path to the python command - e.g $HOME/lilypond-git/build/out/bin/lilypond-book HTH -- Phil Holmes ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: GOP-PROP 5: build system output (probable 2?)
- Original Message - From: "Reinhold Kainhofer" To: Sent: Friday, July 29, 2011 11:30 AM Subject: Re: GOP-PROP 5: build system output (probable 2?) Am Donnerstag 28 Juli 2011, 08:25:25 schrieb Jan Nieuwenhuizen: Graham Percival writes: > You mean, like > > 23cdda9506931d5b9a1e75ee8be8b74f9084a7c0 > > ? Yes (I would have called the option --log). > I'd call it 20% rather than 90%, but yes, Phil's work on > lilypond-book will certainly be valuable! Assuming that --redirect-lilypond-output is used during build now, you mention 500,000 and 370,000 lines of output for make doc. Am I assuming correctly that currently make doc prints 130,000 lines? Which programs are responsible for that? Currently, the doc build is calling lilypond in verbose mode, creating thousands of unnecessary lines like [snip] Reinhold - I've been looking at the build system in some depth and am very well aware of this. I suggested turning off verbose mode and was told that it was sometimes useful to see whay a build had failed. That's part of the reason for resorting to the logfile suggestion. The other thing is that all commands called by make are echoed on the console, always including several lines of include pathes. While this might sound useful, in fact it isn't because the exact command does not help you. make seems to set some env variables, too, so exactly duplicating the command called in make does not work. In particular, the Python includes cannot be found. I encountered this when trying to debug musicxml support in lilypond book: calling lilypond-book exactly as printed by make does NOT find lilylib! Ditto. FWIW if you want to suppress make echoes, make -s (silent) will do this. -- Phil Holmes ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: GOP-PROP 5: build system output (probable 2?)
2011/7/29 Reinhold Kainhofer : > The other thing is that all commands called by make are echoed on the console, > always including several lines of include pathes. While this might sound > useful, in fact it isn't because the exact command does not help you. make > seems to set some env variables, too, so exactly duplicating the command > called in make does not work. In particular, the Python includes cannot be > found. +1 +1 -- Francisco Vila. Badajoz (Spain) www.paconet.org , www.csmbadajoz.com ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: GOP-PROP 5: build system output (probable 2?)
Am Donnerstag 28 Juli 2011, 08:25:25 schrieb Jan Nieuwenhuizen: > Graham Percival writes: > > You mean, like > > > > 23cdda9506931d5b9a1e75ee8be8b74f9084a7c0 > > > > ? > > Yes (I would have called the option --log). > > > I'd call it 20% rather than 90%, but yes, Phil's work on > > lilypond-book will certainly be valuable! > > Assuming that --redirect-lilypond-output is used during build now, you > mention 500,000 and 370,000 lines of output for make doc. Am I assuming > correctly that currently make doc prints 130,000 lines? Which programs > are responsible for that? Currently, the doc build is calling lilypond in verbose mode, creating thousands of unnecessary lines like [/home/reinhold/lilypond/out/share/lilypond/current/ly/init.ly [/home/reinhold/lilypond/out/share/lilypond/current/ly/declarations-init.ly [/home/reinhold/lilypond/out/share/lilypond/current/ly/music-functions- init.ly] [/home/reinhold/lilypond/out/share/lilypond/current/ly/toc-init.ly] Using `nederlands' note names... [/home/reinhold/lilypond/out/share/lilypond/current/ly/drumpitch-init.ly] [/home/reinhold/lilypond/out/share/lilypond/current/ly/chord-modifiers- init.ly] [/home/reinhold/lilypond/out/share/lilypond/current/ly/script-init.ly] [/home/reinhold/lilypond/out/share/lilypond/current/ly/chord-repetition- init.ly] [/home/reinhold/lilypond/out/share/lilypond/current/ly/scale-definitions- init.ly] [/home/reinhold/lilypond/out/share/lilypond/current/ly/dynamic-scripts- init.ly] [/home/reinhold/lilypond/out/share/lilypond/current/ly/spanners-init.ly] [/home/reinhold/lilypond/out/share/lilypond/current/ly/predefined-fretboards- init.ly] [/home/reinhold/lilypond/out/share/lilypond/current/ly/string-tunings- init.ly Using `nederlands' note names... [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] As a very simple solution, a first step would be not to run lilypond in verbose mode. You only need that debug output if a problem occurs. And then you normally want to see where the problem is, not so much what lilypond does internally. Once you know where the problem lies, you usually investigate just this one problematic file. So providing full debug output for the slim chance that there is some nasty error in the lilypond binary sounds like overkill. On the other hand, this way all warnings are buried inside thousands of debug output lines. The other thing is that all commands called by make are echoed on the console, always including several lines of include pathes. While this might sound useful, in fact it isn't because the exact command does not help you. make seems to set some env variables, too, so exactly duplicating the command called in make does not work. In particular, the Python includes cannot be found. I encountered this when trying to debug musicxml support in lilypond book: calling lilypond-book exactly as printed by make does NOT find lilylib! And most of the other output comes from lilypond-book. > > I don't agree. Log files are cheap; I think we should always > > write the logfiles They might be cheap, unless you are running backups on the lilypond tree... Cheers, Reinhold -- -- Reinhold Kainhofer, Vienna University of Technology, Austria email: reinh...@kainhofer.com, http://reinhold.kainhofer.com/ * Financial and Actuarial Mathematics, TU Wien, http://www.fam.tuwien.ac.at/ * Edition Kainhofer Music Publishing, http://www.edition-kainhofer.com/ * LilyPond music typesetting software, http://www.lilypond.org/ ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: GOP-PROP 5: build system output (probable 2?)
On Thu, Jul 28, 2011 at 08:25:25AM +0200, Jan Nieuwenhuizen wrote: > Graham Percival writes: > > > You mean, like > > 23cdda9506931d5b9a1e75ee8be8b74f9084a7c0 > > ? > > Yes (I would have called the option --log). IMO a long descriptive name is better than a short name that's open to interpretation. Note that --redirect-lilypond-output does *not* save any output from lilypond-book (the python script) to a file. I'd expect a --log to do that. We may well look at adding --log in the future, or else do that redirection in the build system. > > I'd call it 20% rather than 90%, but yes, Phil's work on > > lilypond-book will certainly be valuable! > > Assuming that --redirect-lilypond-output is used during build now, It is not. We are moving very slowly and cautiously to avoid creating more problems. --redirect-lilypond-output needs a lot more testing, particularly stuff like .tely files including .itely files including .ly files. The build system has not been touched, in large part because we have not yet decided on what the overall policy should be. > > I don't agree. Log files are cheap; I think we should always > > write the logfiles > > I don't get this. Any sort of complexity added is expensive. It must > be written, it must be documented, it must be remembered, it must be > maintained. One of the biggest responsibilities as a maintainer is to > deny most if not all `nice to have' features in favour of simplicity and > more important things. I submit to you that since we have 1 thread about broken builds every 2 months, the current build system needs to change. I think this extra complexity is worth it. BTW, do you consider the logging system in GUB to be unnecessary added complexity? I find it incredibly useful. > Moreover, I figure that c/c++ compilation amounts to only a maximum of > about 0% to the sea of output burden. What are you trying to fix? We're mostly trying to fix the doc output. In case there is ambiguity here, "build system output" refers to make doc in addition to plain old make We are trying to figure out general policies for the build system. > Also, you are [should be] probably running c/c++ compilation in -j4 > mode; how are you going to find/determine which compile failed and > what log file it is? Then, you have to open the log file and tell > your editor to go to the right location. If we cannot clearly distinguish which log file (from -j4) contains the error, then obviously we would simply redirect all threads to the same log file. I consider this a relatively minor implementation detail; the relevant policy point is: "Ideally, a failing build should provide hints about the reason why it failed, or at least hints about which log file(s) to examine." If a certain arrangement of log files (be that merging files, splitting files, whatever) does a better job of bringing the error to the user's attention, then we should use that arrangement. Cheers, - Graham ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: GOP-PROP 5: build system output (probable 2?)
Graham Percival writes: > You mean, like > 23cdda9506931d5b9a1e75ee8be8b74f9084a7c0 > ? Yes (I would have called the option --log). > I'd call it 20% rather than 90%, but yes, Phil's work on > lilypond-book will certainly be valuable! Assuming that --redirect-lilypond-output is used during build now, you mention 500,000 and 370,000 lines of output for make doc. Am I assuming correctly that currently make doc prints 130,000 lines? Which programs are responsible for that? >> I can imagine also adding stepmake rules to handle V=0 for >> c/c++ compilation. Possibly not logging that would be OK, >> because a new compile with V=1 would almost instantly show >> the problem? > > I don't agree. Log files are cheap; I think we should always > write the logfiles I don't get this. Any sort of complexity added is expensive. It must be written, it must be documented, it must be remembered, it must be maintained. One of the biggest responsibilities as a maintainer is to deny most if not all `nice to have' features in favour of simplicity and more important things. Moreover, I figure that c/c++ compilation amounts to only a maximum of about 0% to the sea of output burden. What are you trying to fix? Also, you are [should be] probably running c/c++ compilation in -j4 mode; how are you going to find/determine which compile failed and what log file it is? Then, you have to open the log file and tell your editor to go to the right location. Isn't it smarter to hit compile [bound to make V=[ [-j1]] in your editor and have it jump to the error location? Jan -- Jan Nieuwenhuizen | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.nl ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: GOP-PROP 5: build system output (probable 2?)
Graham Percival writes: > You mean, like > 23cdda9506931d5b9a1e75ee8be8b74f9084a7c0 > ? Yes (I would have called the option --log). > I'd call it 20% rather than 90%, but yes, Phil's work on > lilypond-book will certainly be valuable! Assuming that --redirect-lilypond-output is used during build now, you mention 500,000 and 370,000 lines of output for make doc. Am I assuming correctly that currently make doc prints 130,000 lines? Which programs are responsible for that? >> I can imagine also adding stepmake rules to handle V=0 for >> c/c++ compilation. Possibly not logging that would be OK, >> because a new compile with V=1 would almost instantly show >> the problem? > > I don't agree. Log files are cheap; I think we should always > write the logfiles I don't get this. Any sort of complexity added is expensive. It must be written, it must be documented, it must be remembered, it must be maintained. One of the biggest responsibilities as a maintainer is to deny most if not all `nice to have' features in favour of simplicity and more important things. Moreover, I figure that c/c++ compilation amounts to only a maximum of about 0% to the sea of output burden. What are you trying to fix? Also, you are [should be] probably running c/c++ compilation in -j4 mode; how are you going to find/determine which compile failed and what log file it is? Then, you have to open the log file and tell your editor to go to the right location. It's just much smarter to hit compile [bound to make V=[ [-j1]] in your editor and have it jump to the error location. Jan -- Jan Nieuwenhuizen | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.nl ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: GOP-PROP 5: build system output (probable 2?)
On Thu, Jul 28, 2011 at 07:33:04AM +0200, Jan Nieuwenhuizen wrote: > Graham Percival writes: > > > I still don't feel that we have any kind of consensus on this. > > Here's an updated proposal. > > So what if we add a --log option to lilypond-book (and probably > to lilypond), that [always in verbose mode?] writes individual > .log files alongside the output. Would this fix 90% of this > proposal? You mean, like 23cdda9506931d5b9a1e75ee8be8b74f9084a7c0 ? I'd call it 20% rather than 90%, but yes, Phil's work on lilypond-book will certainly be valuable! > I can imagine also adding stepmake rules to handle V=0 for > c/c++ compilation. Possibly not logging that would be OK, > because a new compile with V=1 would almost instantly show > the problem? I don't agree. Log files are cheap; I think we should always write the logfiles -- but I'd be ok if we had a special option to omit writing logfiles if the user really really wants that. Cheers, - Graham ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: GOP-PROP 5: build system output (probable 2?)
Graham Percival writes: > I still don't feel that we have any kind of consensus on this. > Here's an updated proposal. Ah, great. So what if we add a --log option to lilypond-book (and probably to lilypond), that [always in verbose mode?] writes individual .log files alongside the output. Would this fix 90% of this proposal? Sure, we also have metafont/post, fontforge. I can imagine also adding stepmake rules to handle V=0 for c/c++ compilation. Possibly not logging that would be OK, because a new compile with V=1 would almost instantly show the problem? Jan -- Jan Nieuwenhuizen | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.nl ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
GOP-PROP 5: build system output (probable 2?)
I still don't feel that we have any kind of consensus on this. Here's an updated proposal. http://lilypond.org/~graham/gop/gop_5.html ** Proposal summary When you run make or make doc, * All output will be saved to various log files, with the exception of output directly from make(1). * By default, no other output will be displayed on the console, with one exception: if a build fails, we might display some portion(s) of log file(s) which give useful clues about the reason for the failure. The user may optionally request additional output to be printed; this is controlled with the VERBOSE=x flag. In such cases, all output will still be written to log files; the console output is strictly additional to the log files. * Logfiles from calling lilypond (as part of lilypond-book) will go in the relevant ‘build/out/lybook-db/12/lily-123456.log’ file. All other logfiles will go in the ‘build/logfiles/’ directory. * Both stderr and stdout will be saved in *.log. The order of lines from these streams should be preserved. * There will be no additional “progress messages” during the build process. If you run make --silent, a non-failing build should print absolutely nothing to the screen. * Ideally, a failing build should provide hints about the reason why it failed, or at least hints about which log file(s) to examine. ** Rationale When the lilypond build breaks, it’s too hard to figure out why it broke. We see emails to lilypond-devel approximately once every two months about broken builds. On a subjective note, Graham has been the documentation editor since 2003, but even he cannot reliably pinpoint the cause of a failing doc build within 10 minutes. We waste a ridiculous amount of time, effort, and patience on build problems. ** Sea of output Before any of the current work on reducing output from make, the result of a "make doc" was over 500,000 lines of text. The prime reason for the output being so verbose is that all the processes that run as a result of the call to make echo their output to the screen, often in verbose mode. Lilypond itself produces around 370,000 lines of output as a result of lilypond-book building all the snippets. Much of this output can be redirected to logfiles and so the impossible-to-read clutter on the screen is cut out and could be referred to later. ** Don’t cause more build problems However, there is a danger in this approach, that vital error messages can also be lost, thus preventing the cause of the failure of a make being found. We therefore need to be exceptionally careful to move cautiously, include plenty of tests, and give time for people to experiment/find problems in each stage before proceeding to the next stage. ** Implementation notes There is an existing make variable QUIET_BUILD, which alter the amount of output being displayed (http://lilypond.org/doc/v2.15/Documentation/contributor/useful-make-variables ). We are not planning on keeping this make variable. The standard way for GNU packages to give more output is with a V=x option. Presumably this is done by increasing x? If we support this option, we should still write log files; we would simply print more of the info in those log files to screen. ** Bottom line If there are no build problems, there should be no change to people’s workflow. If there are build problems, then it should be easier to find out why it’s failing. Cheers, - Graham ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
