----- Original Message -----
From: "Graham Percival" <gra...@percival-music.ca>
To: "Phil Holmes" <em...@philholmes.net>
Cc: "Devel" <lilypond-devel@gnu.org>
Sent: Friday, September 23, 2011 7:54 PM
Subject: Re: Big web page
On Fri, Sep 23, 2011 at 05:03:21PM +0100, Phil Holmes wrote:
You know, I'm wondering if this serves any purpose. It's a
humungous thing that takes an age to load and display, and for a
long time has had a load of missing images, bad links, etc. No-one
complained much and no-one really seemed to notice. Yet we lose the
images on web-examples and get a user report same day.
Given the shenanigans needed to create it, might it make sense to retire
it?
When the new website was first being built, there was a huge
outcry when I suggested not bothering with it.
I'd rather keep it around, at least for a bit longer, because it
serves as a canary. Is the doc build system as generic as we
need? no. (note that there's a few pictures in other manuals;
it's not just web-big-html that needs them) Is the doc build
system as well-organized as we need? of course not.
I think we can do a huge amount of work just by cleaning up the
"make website" process. Admittedly that doesn't include
web-big-html, but the increased understanding and simplifications
should have follow-on effects to "make doc".
Cheers,
- Graham
I thought I'd start with a rather easier to follow description of how make
website works. It's attached as a text file with no formatting as yet, for
quick thoughts. If wanted I could put it on Rietveld as is, or if it's
close enough to being useful, I could texify it first.
--
Phil Holmes
The Story of make website
The rule for make website is found in GNUmakefile.in:
website:
$(MAKE) config_make=$(config_make) \
top-src-dir=$(top-src-dir) \
-f $(top-src-dir)/make/website.make \
website
This translates as:
make --no-builtin-rules config_make=./config.make \
top-src-dir=/home/phil/lilypond-git \
-f /home/phil/lilypond-git/make/website.make \
website
which has the effect of setting the variables config_make and top-src-dir and
then processing the file git/make/website.make with the target of website.
website.make starts with the following:
ifeq ($(WEBSITE_ONLY_BUILD),1)
which checks to see whether the variable WEBSITE_ONLY_BUILD was set to one on
the command line. This is only done for standalone website builds, not in the
normal case. The result of the test determines the value of some variables
that are set. A number of other variables are set, in order to establish
locations of various files. An example is:
CREATE_VERSION=python $(script-dir)/create-version-itexi.py
The rule for website is:
website: website-texinfo website-css website-pictures website-examples web-post
cp $(SERVER_FILES)/favicon.ico $(OUT)/website
cp $(SERVER_FILES)/robots.txt $(OUT)/website
cp $(top-htaccess) $(OUT)/.htaccess
cp $(dir-htaccess) $(OUT)/website/.htaccess
so we see that this starts by running the rules for 5 other targets, then
finishes by copying some files. We'll cover that later - first
website-texinfo. That rule is:
website-texinfo: website-version website-xrefs website-bibs
for l in '' $(WEB_LANGS); do \
if test -n "$$l"; then \
langopt=--lang="$$l"; \
langsuf=.$$l; \
fi; \
$(TEXI2HTML) --prefix=index \
--split=section \
--I=$(top-src-dir)/Documentation/"$$l" \
--I=$(top-src-dir)/Documentation \
--I=$(OUT) \
$$langopt \
--init-file=$(texi2html-init-file) \
-D web_version \
--output=$(OUT)/"$$l" \
$(top-src-dir)/Documentation/"$$l"/web.texi ; \
ls $(OUT)/$$l/*.html | xargs grep -L 'UNTRANSLATED NODE: IGNORE ME' | sed
's!$(OUT)/'$$l'/!!g' | xargs $(MASS_LINK) --prepend-suffix="$$langsuf" hard
$(OUT)/$$l/ $(OUT)/website/ ; \
done
which therefore depends on website-version, website-xrefs & website-bibs.
website-version:
mkdir -p $(OUT)
$(CREATE_VERSION) $(top-src-dir) > $(OUT)/version.itexi
$(CREATE_WEBLINKS) $(top-src-dir) > $(OUT)/weblinks.itexi
which translates as:
mkdir -p out-website
python /home/phil/lilypond-git/scripts/build/create-version-itexi.py
/home/phil/lilypond-git > out-website/version.itexi
python /home/phil/lilypond-git/scripts/build/create-weblinks-itexi.py
/home/phil/lilypond-git > out-website/weblinks.itexi
So, we make out-website then send the output of create-version-itexi.py to out-website/version.itexi and create-weblinks-itexi.py to out-website/weblinks.itexi.
create-version-itexi.py parses the file VERSION in the top source dir. It
contains:
PACKAGE_NAME=LilyPond
MAJOR_VERSION=2
MINOR_VERSION=15
PATCH_LEVEL=13
MY_PATCH_LEVEL=
VERSION_STABLE=2.14.2
VERSION_DEVEL=2.15.12
currently. c-v-i.py parses this to:
@c ************************ Version numbers ************
@macro version
2.15.13
@end macro
@macro versionStable
2.14.2
@end macro
@macro versionDevel
2.15.12
@end macro
create-weblinks-itexi.py creates a load of texi macros (of the order of 1000)
similar to:
@macro manualStableGlossaryPdf
@uref{../doc/v2.14/Documentation/music-glossary.pdf,Music glossary.pdf}
@end macro.
It loads its languages from langdefs.py, and therefore outputs the following
unhelpful warning:
langdefs.py: warning: lilypond-doc gettext domain not found.
Next:
website-xrefs: website-version
for l in '' $(WEB_LANGS); do \
is the start of the rule, truncated for brevity. This loops through the
languages to be used on the website, processing some variables which I don't
fully understand, to run this command:
python /home/phil/lilypond-git/scripts/build/extract_texi_filenames.py \
-I /home/phil/lilypond-git/Documentation \
-I /home/phil/lilypond-git/Documentation/"$l" \
-I out-website -o out-website --split=node \
--known-missing-files=/home/phil/lilypond-git/scripts/build/website-known-missing-files.txt
\
-q \
/home/phil/lilypond-git/Documentation/"$l"/web.texi ;\
There's a good description of what extract_texi_filenames.py does at the top of
the script, but a shortened version is:
If this script is run on a file texifile.texi, it produces a file
texifile[.LANG].xref-map with tab-separated entries of the form
NODE\tFILENAME\tANCHOR. An example from web.nl.xref-map is:
Inleiding Introduction Introduction
e-t-f.py follows the includes from document to document. We know some have not
been created yet, and known-missing-files option tells e-t-f/py which these are.
It then does this:
for m in $(MANUALS); do \
to run e-t-f.py against all of the manuals, in each language. Next:
website-bibs: website-version
BSTINPUTS=$(top-src-dir)/Documentation/web \
$(WEB_BIBS) -s web \
-s $(top-src-dir)/Documentation/lily-bib \
-o $(OUT)/others-did.itexi \
$(quiet-flag) \
$(top-src-dir)/Documentation/web/others-did.bib
This is half the command. It runs bib2texi.py on 2 .bib files - others-did.bib
and we-wrote.bib. This converts bibliography files into texi files with bibtex.
Next the commands in the website-texinfo rule are run:
for l in '' $(WEB_LANGS); do \
run texi2html. This is the program that outputs the progress message (found in
Documentation/lilypond-texi2html.init):
Processing web site: []
It also outputs warning messages like:
WARNING: Unable to find node 'ÅeÅ¡enà potÞÃ' in book usage.
website-css:
cp $(top-src-dir)/Documentation/css/*.css $(OUT)/website
Copies 3 css files to out-website/website. Then:
website-pictures:
mkdir -p $(OUT)/website/pictures
if [ -d $(PICTURES) ]; \
then \
cp $(PICTURES)/* $(OUT)/website/pictures ; \
ln -sf website/pictures $(OUT)/pictures ;\
fi
which translates as:
if [ -d Documentation/pictures/out-www ]; \
then \
cp Documentation/pictures/out-www/*
out-website/website/pictures ; \
ln -sf website/pictures out-website/pictures ;\
fi
i.e. it copies the contents of build/Documentation/pictures/out-www/* to
out-website/website/pictures. Unfortunately, the pictures are only created
once make doc has been run, so an initial run of make website copies nothing,
and the pictures on the website (e.g. the logo) do not exist. Next:
website-examples:
mkdir -p $(OUT)/website/ly-examples
if [ -d $(EXAMPLES) ]; \
then \
cp $(EXAMPLES)/* $(OUT)/website/ly-examples ; \
fi
translates to:
mkdir -p out-website/website/ly-examples
if [ -d Documentation/web/ly-examples/out-www ]; \
then \
cp Documentation/web/ly-examples/out-www/*
out-website/website/ly-examples ; \
fi
This does the same with the LilyPond examples (found at
http://lilypond.org/examples.html). Again, these are actually only created by
make doc (and since they are generated from LilyPond source files, require a
working LilyPond exe made with make). So this does nothing initially. Then:
web-post:
$(WEB_POST) $(OUT)/website
which is:
python /home/phil/lilypond-git/scripts/build/website_post.py out-website/website
which describes itself as:
This is web_post.py. This script deals with translations in the "make website"
target.
It also does a number of other things, including adding the Google tracker code
and the language selection footer. We're now at the end of our story. The
final 4 lines of the recipe for website are:
cp $(SERVER_FILES)/favicon.ico $(OUT)/website
cp $(SERVER_FILES)/robots.txt $(OUT)/website
cp $(top-htaccess) $(OUT)/.htaccess
cp $(dir-htaccess) $(OUT)/website/.htaccess
The first translates as:
cp /home/phil/lilypond-git/Documentation/web/server/favicon.ico
out-website/website
so we see these are just copying the support files for the web server.
_______________________________________________
lilypond-devel mailing list
lilypond-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/lilypond-devel
