----- Original Message -----
From: "Graham Percival" <gra...@percival-music.ca>
To: "Phil Holmes" <m...@philholmes.net>
Cc: "Carl Sorensen" <c_soren...@byu.edu>; <lilypond-devel@gnu.org>
Sent: Saturday, May 28, 2011 6:22 PM
Subject: Re: make doc
On Sat, May 28, 2011 at 05:48:39PM +0100, Phil Holmes wrote:
So the command line would need to be something like
lilypond filename.ly > filename.log?
Something like that, probably. Dunno. What happens when you try
that command on the command-line?
Remember the first rule of build systems: a build system just sets
up the environment and then runs command-line commands. (no wait,
that's the second rule. The first rule is that all build systems
suck; most of them suck exponentially.)
Will have to look at how that could be done, since the times lily is
called that I've so far found are initiated by lilypond-book.
Well, figure out how to do it with lilypond by itself, then figure
out how to do it with lilypond-book, then figure out how to do it
inside the build system.
Cheers,
- Graham
OK - so I now know how to redirect the output from lilypond-book.py when
it's making, for example, collated-files.tely - I can redirect it to
collated-files.log. FWIW simply doing the redirect on
/input/regression/out-www/collated-files.tely creates a logfile of 30 Meg
and 68,000-odd lines! I think this is a good first step. For those who are
interested in the build system, I attach my notes to date of how make doc
works - slightly stream-of-consciousness, I'm afraid.
--
Phil Holmes
Preliminary question to be answered some time: where do all the
GNUmakefiles come from. They're in the build directory, but this
is not part of source. Must be the configure script. And it
looks like this comes from autogen.sh. Must at some point kill
the whole git directory, repull and see what is created when.
Anyway, here's how make doc progresses:
This is the build dependency tree from
stepmake/stepmake/generic-targets.make:
doc: doc-stage-1
doc-stage-1:
$(MAKE) -C $(depth)/scripts/build out=
$(MAKE) out=www WWW-1
WWW-1: local-WWW-1
$(LOOP)
$(MAKE) out=www WWW-2
WWW-2: local-WWW-2
$(LOOP)
$(MAKE) out=www WWW-post
MAKE = make --no-builtin-rules
-C = Change to directory before make
doc-stage-1 does lots of opening and looking in files, but no
processing.
Variable LOOP =
+ make PACKAGE=LILYPOND package=lilypond -C python && make PACKAGE=LILYPOND package=lilypond -C scripts && make PACKAGE=LILYPOND package=lilypond -C flower && make
PACKAGE=LILYPOND package=lilypond -C lily && make PACKAGE=LILYPOND package=lilypond -C mf && make PACKAGE=LILYPOND package=lilypond -C ly && make PACKAGE=LILYPOND
package=lilypond -C tex && make PACKAGE=LILYPOND package=lilypond -C ps && make PACKAGE=LILYPOND package=lilypond -C scm && make PACKAGE=LILYPOND package=lilypond -C po
&& make PACKAGE=LILYPOND package=lilypond -C make && make PACKAGE=LILYPOND package=lilypond -C elisp && make PACKAGE=LILYPOND package=lilypond -C vim && make
PACKAGE=LILYPOND package=lilypond -C input && make PACKAGE=LILYPOND package=lilypond -C stepmake && make PACKAGE=LILYPOND package=lilypond -C Documentation && true
From git grep:
stepmake/stepmake/generic-vars.make has this:
LOOP=+$(foreach i, $(SUBDIRS), $(MAKE) PACKAGE=$(PACKAGE) package=$(package) -C $(i) $@
&&) true
$@ is the name of the target - WWW-1 in this case.
In GNUmakefile.in we find:
SUBDIRS = python scripts \
flower lily \
mf ly \
tex ps scm \
po make \
elisp vim \
input \
stepmake $(documentation-dir)
So that's how we get the main make loop...
That loop expands like this:
make PACKAGE=LILYPOND package=lilypond -C python WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C scripts WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C flower WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C lily WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C mf WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C ly WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C tex WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C ps WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C scm WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C po WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C make WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C elisp WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C vim WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C input WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C stepmake WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C Documentation WWW-1 &&
true
The directories up to and including vim produce no effect with
make in non-debug mode, although debug does show lots of action.
git/build/input/GNUmakefile is:
depth=../
include $(depth)/config$(if $(conf),-$(conf),).make
include $(configure-srcdir)/./input/GNUmakefile
MODULE_INCLUDES += $(src-dir)/$(outbase)
The first include is:
..//config.make (note the // which is strictly wrong)
which has lots of variables to set, but no action occurs.
The second is:
lilypond-git/./input/GNUmakefile
which similarly doesn't create any actual action.
An error message at the end of build/input/GNUmakefile stops
make processing before it moves on to regression - so where does
that come from?
And the answer is - make processes all directories in the
directory it's entered (with some exceptions like out and out-www)
and so it changes to /regression.
It then seems to consider whether it needs to make/remake loads of
makefiles. Don't understand this yet. Possibly these are all the
makefiles it's processing, and it always checks they're up to date
before processing other files?
Could be correct - some of this output is:
Must remake target `../../make/ly-inclusions.make'.
Failed to remake target file `../../make/ly-inclusions.make'.
Having decided that, it then leaves the directory and re-executes:
make PACKAGE=LILYPOND package=lilypond -C regression WWW-1
The top of this make is:
This program built for i486-pc-linux-gnu
Reading makefiles...
Reading makefile `GNUmakefile'...
Reading makefile `../..//config.make' (search path) (no ~ expansion)...
which looks like it's re-reading all its known makefiles to check
they're up to date.
(From the make manual:
To this end, after reading in all makefiles, make will consider each as a goal
target and
attempt to update it. If a makefile has a rule which says how to update it
(found either
in that very makefile or in another one) or if an implicit rule applies to it
(see Chapter 10
[Using Implicit Rules], page 103), it will be updated if necessary. After all
makefiles have
been checked, if any have actually been changed, make starts with a clean slate
and reads
all the makefiles over again. (It will also attempt to update each of them over
again, but
normally this will not change them again, since they are already up to date.)
So my assumption seems correct)
There appear to be about 74 of them. After all the makefile
checking, we get this:
Updating goal targets....
Considering target file `WWW-1'.
File `WWW-1' does not exist.
Considering target file `local-WWW-1'.
File `local-WWW-1' does not exist.
Considering target file `out-www/collated-files.texi'.
File `out-www/collated-files.texi' does not exist.
Looking for an implicit rule for `out-www/collated-files.texi'.
Trying pattern rule with stem `collated-files.texi'.
Trying implicit prerequisite `collated-files.texi.in'.
Trying pattern rule with stem `collated-files.texi'.
Trying implicit prerequisite `collated-files.texi.in'.
Trying pattern rule with stem `collated-files'.
Trying implicit prerequisite `collated-files.tely'.
Trying pattern rule with stem `collated-files'.
Trying implicit prerequisite `out-www/collated-files.tely'.
Trying rule prerequisite `out-www/version.itexi'.
Found prerequisite `out-www/version.itexi' as VPATH
`/home/phil/lilypond-git/input/regression/out-www/version.itexi'
grep finds this if searching for local-WWW-1:
make/lysdoc-targets.make:
local-WWW-1: $(outdir)/collated-files.texi $(outdir)/collated-files.pdf
which means that local-WWW-1 depends on coll*.texi and coll*.pdf
and so these will need to be checked to see if they're up to date.
So make needs to find rules for both of those and (as it says) it
certainly needs to make coll*.texi, since it doesn't exist.
In ly-rules.make we have:
.SUFFIXES: .doc .tely .texi .ly
which I'll work out at some point, and also this rule:
$(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi
$(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK)
$(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS)
$(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir)
--format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<
Note that the recipe is a very long line - it could probably
benefit from splitting. The same makefile also has:
$(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi
$(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK)
$(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS)
$(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir)
--format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<
which seems to be an almost exact duplicate. Whatever, the first
one is executed first. Have not checked if the second executes.
The first recipe translates as this:
LILYPOND_VERSION=2.15.0 /usr/bin/python --process=' ' --output=./out-www --format= --lily-output-dir /home/phil/lilypond-git/build/out/lybook-db
if we stop the build with an $(error), but I think this is because
we need to allow it to process the dependencies first. It looks
like foo.texi is shown as being dependent on foo.tely, plus a load
of other files.
DOCUMENTATION_LOCALE_TARGET is blank
INIT_LY_SOURCES = /home/phil/lilypond-git/scm/auto-beam.scm
/home/phil/lilypond-git/scm/autochange.scm
plus 10s (100s?) of other .scm files.
SCHEME_SOURCES = /home/phil/lilypond-git/ly/Welcome-to-LilyPond-MacOS.ly /home/phil/lilypond-git/ly/Welcome_to_LilyPond.ly
ditto .ly files. This does seem a teency bit wrong - it looks like
the .ly and .scm files have been interchanged. ly-vars.make has
these 2 lines:
INIT_LY_SOURCES = $(wildcard $(top-src-dir)/scm/*.scm)
SCHEME_SOURCES = $(wildcard $(top-src-dir)/ly/*.ly)
Looks like a bug.....
So it now works its way through all these files, checking if they
need to be remade. This is 100s of lines of the debug listing,
although none in the normal list. Clearly none has to be made
since they're source files. It concludes:
Must remake target `out-www/collated-files.tely'
lysdoc-rules.make has this:
$(outdir)/collated-files.tely: $(COLLATED_FILES)
$(LYS_TO_TELY) --name=$(outdir)/collated-files.tely --title="$(TITLE)"
--author="$(AUTHOR)" $^
lysdoc-vars.make has:
COLLATED_FILES = $(sort $(TEXINFO_SOURCES) $(LY_FILES) $(OUT_LY_FILES) )
We find that:
TEXINFO_SOURCES = AAA-intro-regression.tely
OUT_LY_FILES is empty
so LY_FILES has the big long list of all the .ly files in the
regression directory.
This kicks off
/home/phil/lilypond-git/build/scripts/build/out/lys-to-tely
with a list of all the files in the regression test directory. This
should (I believe) create the file collated-files.tely.
So the next rule in make is for version.itexi, and make duly
checks this. There's a rule in doc-i18n-root-rules.make that this
depends on git/VERSION:
$(outdir)/version.%: $(top-src-dir)/VERSION
$(PYTHON) $(top-src-dir)/scripts/build/create-version-itexi.py > $@
This causes create-version-itexi.py to run and create
version.itexi.
Once that's done, all the other *.scm and *.ly files are checked
and since they have no rules associated, they aren't remade (just
as well for source files, really). Since version.itexi was remade
make concludes that collated-files.texi must be remade. To do
this, it runs lilypond-book.py on collated-files.tely, as below:
LILYPOND_VERSION=2.15.0 /usr/bin/python
/home/phil/lilypond-git/scripts/lilypond-book.py -I
/home/phil/lilypond-git/input/regression/ -I ./out-www -I
/home/phil/lilypond-git/input -I /home/phil/lilypond-git/Documentation -I
/home/phil/lilypond-git/Documentation/snippets -I
/home/phil/lilypond-git/input/regression/ -I
/home/phil/lilypond-git/Documentation/included/ -I
/home/phil/lilypond-git/build/mf/out/ -I /home/phil/lilypond-git/build/mf/out/
-I /home/phil/lilypond-git/Documentation/pictures -I
/home/phil/lilypond-git/build/Documentation/pictures/./out-www
--process='/home/phil/lilypond-git/build/out/bin/lilypond -I
/home/phil/lilypond-git/input/regression/ -I ./out-www -I
/home/phil/lilypond-git/input -I /home/phil/lilypond-git/Documentation -I
/home/phil/lilypond-git/Documentation/snippets -I
/home/phil/lilypond-git/input/regression/ -I
/home/phil/lilypond-git/Documentation/included/ -I
/home/phil/lilypond-git/build/mf/out/ -I /home/phil/lilypond-git/build/mf/out/
-I /home/phil/lilypond-git/Documentation/pictures -I
/home/phil/lilypond-git/build/Documentation/pictures/./out-www -dbackend=eps
--formats=ps,png,pdf -dinclude-eps-fonts -dgs-load-fonts --header=doctitle
--header=doctitlecs --header=doctitlede --header=doctitlees --header=doctitlefr
--header=doctitlehu --header=doctitleit --header=doctitleja --header=doctitlenl
--header=doctitlezh --header=texidoc --header=texidoccs --header=texidocde
--header=texidoces --header=texidocfr --header=texidochu --header=texidocit
--header=texidocja --header=texidocnl --header=texidoczh -dcheck-internal-types
-ddump-signatures -danti-alias-factor=2' --output=./out-www --format=texi-html
--verbose --lily-output-dir /home/phil/lilypond-git/build/out/lybook-db
out-www/collated-files.tely
For easier reading:
LILYPOND_VERSION=2.15.0
/usr/bin/python
/home/phil/lilypond-git/scripts/lilypond-book.py
-I /home/phil/lilypond-git/input/regression/
-I ./out-www -I /home/phil/lilypond-git/input
-I /home/phil/lilypond-git/Documentation
-I /home/phil/lilypond-git/Documentation/snippets
-I /home/phil/lilypond-git/input/regression/
-I /home/phil/lilypond-git/Documentation/included/
-I /home/phil/lilypond-git/build/mf/out/
-I /home/phil/lilypond-git/build/mf/out/
-I /home/phil/lilypond-git/Documentation/pictures
-I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
--process='/home/phil/lilypond-git/build/out/bin/lilypond
-I /home/phil/lilypond-git/input/regression/
-I ./out-www
-I /home/phil/lilypond-git/input
-I /home/phil/lilypond-git/Documentation
-I /home/phil/lilypond-git/Documentation/snippets
-I /home/phil/lilypond-git/input/regression/
-I /home/phil/lilypond-git/Documentation/included/
-I /home/phil/lilypond-git/build/mf/out/
-I /home/phil/lilypond-git/build/mf/out/
-I /home/phil/lilypond-git/Documentation/pictures
-I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
-dbackend=eps
--formats=ps,png,pdf
-dinclude-eps-fonts
-dgs-load-fonts
--header=doctitle
--header=doctitlecs
--header=doctitlede
--header=doctitlees
--header=doctitlefr
--header=doctitlehu
--header=doctitleit
--header=doctitleja
--header=doctitlenl
--header=doctitlezh
--header=texidoc
--header=texidoccs
--header=texidocde
--header=texidoces
--header=texidocfr
--header=texidochu
--header=texidocit
--header=texidocja
--header=texidocnl
--header=texidoczh
-dcheck-internal-types
-ddump-signatures
-danti-alias-factor=2'
--output=./out-www
--format=texi-html
--verbose
--lily-output-dir /home/phil/lilypond-git/build/out/lybook-db
out-www/collated-files.tely
So - lilypond-book runs on:
input/regression/out-www/collated-files.tely
Note the --verbose flag - this is from the make variable
LILYPOND_BOOK_VERBOSE which is added to the make variable
LILYPOND_BOOK_FLAGS.
Now found the invocation to write some of the image files. It's
like this:
Invoking `/home/phil/lilypond-git/build/out/bin/lilypond -I /home/phil/lilypond-git/input/regression/ -I ./out-www -I /home/phil/lilypond-git/input -I /home/phil/lilypond-git/Documentation -I
/home/phil/lilypond-git/Documentation/snippets -I /home/phil/lilypond-git/input/regression/ -I /home/phil/lilypond-git/Documentation/included/ -I /home/phil/lilypond-git/build/mf/out/ -I
/home/phil/lilypond-git/build/mf/out/ -I /home/phil/lilypond-git/Documentation/pictures -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www -dbackend=eps --formats=ps,png,pdf -dinclude-eps-fonts
-dgs-load-fonts --header=doctitle --header=doctitlecs --header=doctitlede --header=doctitlees --header=doctitlefr --header=doctitlehu --header=doctitleit --header=doctitleja --header=doctitlenl --header=doctitlezh
--header=texidoc --header=texidoccs --header=texidocde --header=texidoces --header=texidocfr --header=texidochu --header=texidocit --header=texidocja --header=texidocnl --header=texidoczh -dcheck-internal-types
-ddump-signatures -danti-alias-factor=2 -I "/home/phil/lilypond-git/build/out/lybook-db" -I "/home/phil/lilypond-git/build/input/regression" -I
"/home/phil/lilypond-git/input/regression" -I "/home/phil/lilypond-git/build/input/regression/out-www" -I "/home/phil/lilypond-git/input" -I
"/home/phil/lilypond-git/Documentation" -I "/home/phil/lilypond-git/Documentation/snippets" -I "/home/phil/lilypond-git/input/regression" -I
"/home/phil/lilypond-git/Documentation/included" -I "/home/phil/lilypond-git/build/mf/out" -I "/home/phil/lilypond-git/build/mf/out" -I
"/home/phil/lilypond-git/Documentation/pictures" -I "/home/phil/lilypond-git/build/Documentation/pictures/out-www" --formats=eps --verbose -deps-box-padding=3.000000 -dread-file-list
-dno-strip-output-dir "/home/phil/lilypond-git/build/out/lybook-db/snippet-names--415419468.ly"'
Note the --verbose. This causes 100s of lines of Lily debug output.
But at present I can't work out where the flag comes from. Later.
Here's how this looks spread out:
/home/phil/lilypond-git/build/out/bin/lilypond
-I /home/phil/lilypond-git/input/regression/
-I ./out-www -I /home/phil/lilypond-git/input
-I /home/phil/lilypond-git/Documentation
-I /home/phil/lilypond-git/Documentation/snippets
-I /home/phil/lilypond-git/input/regression/
-I /home/phil/lilypond-git/Documentation/included/
-I /home/phil/lilypond-git/build/mf/out/
-I /home/phil/lilypond-git/build/mf/out/
-I /home/phil/lilypond-git/Documentation/pictures
-I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
-dbackend=eps
--formats=ps,png,pdf
-dinclude-eps-fonts
-dgs-load-fonts
--header=doctitle
--header=doctitlecs
--header=doctitlede
--header=doctitlees
--header=doctitlefr
--header=doctitlehu
--header=doctitleit
--header=doctitleja
--header=doctitlenl
--header=doctitlezh
--header=texidoc
--header=texidoccs
--header=texidocde
--header=texidoces
--header=texidocfr
--header=texidochu
--header=texidocit
--header=texidocja
--header=texidocnl
--header=texidoczh
-dcheck-internal-types
-ddump-signatures
-danti-alias-factor=2
-I "/home/phil/lilypond-git/build/out/lybook-db"
-I "/home/phil/lilypond-git/build/input/regression"
-I "/home/phil/lilypond-git/input/regression"
-I "/home/phil/lilypond-git/build/input/regression/out-www"
-I "/home/phil/lilypond-git/input"
-I "/home/phil/lilypond-git/Documentation"
-I "/home/phil/lilypond-git/Documentation/snippets"
-I "/home/phil/lilypond-git/input/regression"
-I "/home/phil/lilypond-git/Documentation/included"
-I "/home/phil/lilypond-git/build/mf/out"
-I "/home/phil/lilypond-git/build/mf/out"
-I "/home/phil/lilypond-git/Documentation/pictures"
-I "/home/phil/lilypond-git/build/Documentation/pictures/out-www"
--formats=eps
--verbose
-deps-box-padding=3.000000
-dread-file-list
-dno-strip-output-dir
"/home/phil/lilypond-git/build/out/lybook-db/snippet-names--415419468.ly"'
_______________________________________________
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel
