Re: Re-write of documentation tree build process
John Mandereau escreveu: > - decrease the number of arguments passed to add_html_footer() by > setting as much as possible global variables in add_html_footer.py, just > like current default_footer. > > When this is done, it'll be possible to i18n the footer and tha language > menu without making this script a mess again. Any thoughts? If it makes things cleaner, then go for it. It can hardly be worse than current add__html_footer. -- Han-Wen Nienhuys - [EMAIL PROTECTED] - http://www.xs4all.nl/~hanwen LilyPond Software Design -- Code for Music Notation http://www.lilypond-design.com ___ lilypond-devel mailing list lilypond-devel@gnu.org http://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Re-write of documentation tree build process
Jan Nieuwenhuizen wrote: > John Mandereau <[EMAIL PROTECTED]> writes: > > If I really wanted to rewrite it, I'd split it into: > > - a function that walks the input directories and returns directories > > and files lists > > - filtering these lists > > - mapping the lists with mkdir, hardlink and symlink standard library > > functions > > Yes, that makes sense. In mid-January, I'll have some time to work on this (and on the plan below). > As Han-Wen says, the millions of options is hardly a good thing about > rsinc. Mix that with the fact that the ordering of some options is > significant, rsync has intricate special regular expression syntax > '**' comes to mind, and trailing slashes sometimes have magic meaning, > all makes it next to impossible to predict what rsync will do, without > actually testing it. You're definitely right: I had to test 'rsync -rl' in GNUmakefile.in(install-WWW) to get the desired result, with a trailing slash for the source and no trailing slash for the destination. I think add_html_footer.py could be even more cleaned up this way: - drop do_file and its ton of arguments, move the file I/O code to the loop on the file list in add_html_footer(), and move text substitution code to small functions. As a consequence, all this mess would be easier to maintain: disabling a text substitution would mean commenting a single function call line, and the order of text substitutions would be clearer (target-specific or not, pre- or post-.html link processing, ...) - decrease the number of arguments passed to add_html_footer() by setting as much as possible global variables in add_html_footer.py, just like current default_footer. When this is done, it'll be possible to i18n the footer and tha language menu without making this script a mess again. Any thoughts? -- John Mandereau <[EMAIL PROTECTED]> ___ lilypond-devel mailing list lilypond-devel@gnu.org http://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Re-write of documentation tree build process
John Mandereau <[EMAIL PROTECTED]> writes: > Just done. I've spent so much time testing and debugging that I think > stable/2.10 is almost ready for a Christmas release :-) Thanks. > Ok. But I've scrapped remove_self_ref() because no HTML file in the docs > currently needs it. @WEB-TITLE@ substitution isn't used either, but > since it was less messy, I kept it. Very good. > After what Juergen wrote, it's safer not to change it. Sure, I should have verified before speaking ;-) > You remarks are very interesting, especially towards a programming > newbie like me, but I already spent a lot of time on this... IMHO, > programs like these buildscripts won't grow to thousands of lines of > code, so as long as they work, they can bear not totally perfect code. > Anyway, I take your remarks as general programming rules :-) Ok. > If I really wanted to rewrite it, I'd split it into: > - a function that walks the input directories and returns directories > and files lists > - filtering these lists > - mapping the lists with mkdir, hardlink and symlink standard library > functions Yes, that makes sense. > In my newbie mind, I thought that it would not possible to do what we > need for the docs i18n with the shell/make mix, or that it would be very > slow and disk space consuming, so Python was a good solution. After > having read rsync man page, I feel it may be wrong... As Han-Wen says, the millions of options is hardly a good thing about rsinc. Mix that with the fact that the ordering of some options is significant, rsync has intricate special regular expression syntax '**' comes to mind, and trailing slashes sometimes have magic meaning, all makes it next to impossible to predict what rsync will do, without actually testing it. Jan. -- Jan Nieuwenhuizen <[EMAIL PROTECTED]> | GNU LilyPond - The music typesetter http://www.xs4all.nl/~jantien | http://www.lilypond.org ___ lilypond-devel mailing list lilypond-devel@gnu.org http://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Re-write of documentation tree build process
John Mandereau escreveu: > In my newbie mind, I thought that it would not possible to do what we > need for the docs i18n with the shell/make mix, or that it would be very > slow and disk space consuming, so Python was a good solution. After > having read rsync man page, I feel it may be wrong... Unfortunately rsync is rather hard to use with its millions of options. I recommend against it. Also, some options are not supported on all platforms (eg. MacOS X) due to older versiosn. -- Han-Wen Nienhuys - [EMAIL PROTECTED] - http://www.xs4all.nl/~hanwen LilyPond Software Design -- Code for Music Notation http://www.lilypond-design.com ___ lilypond-devel mailing list lilypond-devel@gnu.org http://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Re-write of documentation tree build process
Jan Nieuwenhuizen wrote:
> John Mandereau <[EMAIL PROTECTED]> writes:
>
> > Should I send a patch again, or work according to your remarks and push
> > to stable/2.10 (and possibly master) if Jan agrees?
>
> Please push to stable/2.10.
Just done. I've spent so much time testing and debugging that I think
stable/2.10 is almost ready for a Christmas release :-)
> > - we don't add any custom header, so all stuff using the "header"
> > variable could be dropped.
>
> If it's not too much of a mess, keep the header stuff in. We might
> want to add a header.
Ok. But I've scrapped remove_self_ref() because no HTML file in the docs
currently needs it. @WEB-TITLE@ substitution isn't used either, but
since it was less messy, I kept it.
> > I've already experienced that '[.]' is bogus in some cases.
>
> I'm not aware of that?
Sorry, I was mistaken. There was a bug in my code not because of a '[.]'
but because of an unescaped '.' :-P
> > Does the 'EN' means that the page language is English? If so, it should
> > be modified to handle pages of any language.
>
> Yes, I think so.
After what Juergen wrote, it's safer not to change it.
> I have two remarks, but given the state add-html-footer was in, I
> hardly feel I can make any demands; you're free to ignore me.
You remarks are very interesting, especially towards a programming
newbie like me, but I already spent a lot of time on this... IMHO,
programs like these buildscripts won't grow to thousands of lines of
code, so as long as they work, they can bear not totally perfect code.
Anyway, I take your remarks as general programming rules :-)
> +mirrortree.hardlink_tree (input_roots = doc_dirs,
> + process_dirs = outdir,
> + strip_dir_names = outdir,
> + exclude_dirs = r'(fr|po|out|\w*?-root)(/|$)',
> + process_files =
> '.*[.](?:midi|pdf|png|txt|ly|signature)$|VERSION',
> + exclude_files = 'lily-[0-9a-f]+.*.pdf',
> + target_pattern = target_pattern,
> + targets = targets)
>
> in general, if a function takes much more 3 parameters, you may want
> to take a second look why that is. It may be to be too generic, or
> perhaps it tries to do too many things and can be split into several
> functions, or some parameters can more naturally be grouped in a
> class.
hardlink_tree is actually very generic and tries to do many things :
copying a tree of directories, hardlink files, find files and copy
symlinks, all that with 'out-www/' stripping.
The function takes 9 parameters, because I wrote it like a bunch of
shell commands within a makefile, with a bunch of variables. I tried to
make this awful function call a bit more readable with keywords
arguments.
If I really wanted to rewrite it, I'd split it into:
- a function that walks the input directories and returns directories
and files lists
- filtering these lists
- mapping the lists with mkdir, hardlink and symlink standard library
functions
> IIRC Han-Wen suggested to write Python scripts, because the mess of
> make and shell got unmaintainable, so adding to that would be unwise.
> Using python should result in a reduction of lines of code, and much
> clearer lines as well. I'm not sure that was achieved yet. Comparing
> hardlink_tree and say this snippet
>
> - for dir in $(outdir)/web-root/ ; do \
> - cd $$dir && \
> - for a in `find . -name out-www`; do \
> - rsync -a --link-dest $$a/ $$a/ $$a/.. ; \
> - rm -rf $$a ; \
> - done \
> - done
>
> (hardlink_tree does much more?). Having all logic in python is VERY
> NICE, but I would not hesitate to call system ('rsync ..') from python
> if that would dramatically simplify things.
In my newbie mind, I thought that it would not possible to do what we
need for the docs i18n with the shell/make mix, or that it would be very
slow and disk space consuming, so Python was a good solution. After
having read rsync man page, I feel it may be wrong...
--
John Mandereau <[EMAIL PROTECTED]>
___
lilypond-devel mailing list
lilypond-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Re-write of documentation tree build process
John Mandereau <[EMAIL PROTECTED]> writes:
> Should I send a patch again, or work according to your remarks and push
> to stable/2.10 (and possibly master) if Jan agrees?
Please push to stable/2.10.
> I've already junked a bit of clearly unused stuff, but I didn't try to
> check whether every bit of code is necessary to the HTML files of Lily
> docs. For instance, I wonder whether remove_self_ref() is really used.
> I'll try to use the Python debugger to find the unused bits of code.
> - could we scrap i18n() and move the code to do_file(), as we may add
> i18n in do_file too? (I'd like the footer to be translated, for
> instance).
Yes.
> - we don't add any custom header, so all stuff using the "header"
> variable could be dropped.
If it's not too much of a mess, keep the header stuff in. We might
want to add a header.
> - what do all these '[.]' in regular expressions mean? If they should
> match a dot in a string,
Yes.
> why not just use r'\.' ?
I never want to know how many \ to use in what kind of string. [.]
fixes that.
> I've already experienced that '[.]' is bogus in some cases.
I'm not aware of that?
> Does the 'EN' means that the page language is English? If so, it should
> be modified to handle pages of any language.
Yes, I think so.
> That can simply be done in mirrortree.hardlink_tree(), which could
> return the HTML files list. Let's go for that.
I have two remarks, but given the state add-html-footer was in, I
hardly feel I can make any demands; you're free to ignore me.
+mirrortree.hardlink_tree (input_roots = doc_dirs,
+ process_dirs = outdir,
+ strip_dir_names = outdir,
+ exclude_dirs = r'(fr|po|out|\w*?-root)(/|$)',
+ process_files =
'.*[.](?:midi|pdf|png|txt|ly|signature)$|VERSION',
+ exclude_files = 'lily-[0-9a-f]+.*.pdf',
+ target_pattern = target_pattern,
+ targets = targets)
in general, if a function takes much more 3 parameters, you may want
to take a second look why that is. It may be to be too generic, or
perhaps it tries to do too many things and can be split into several
functions, or some parameters can more naturally be grouped in a
class.
IIRC Han-Wen suggested to write Python scripts, because the mess of
make and shell got unmaintainable, so adding to that would be unwise.
Using python should result in a reduction of lines of code, and much
clearer lines as well. I'm not sure that was achieved yet. Comparing
hardlink_tree and say this snippet
- for dir in $(outdir)/web-root/ ; do \
- cd $$dir && \
- for a in `find . -name out-www`; do \
- rsync -a --link-dest $$a/ $$a/ $$a/.. ; \
- rm -rf $$a ; \
- done \
- done
(hardlink_tree does much more?). Having all logic in python is VERY
NICE, but I would not hesitate to call system ('rsync ..') from python
if that would dramatically simplify things.
Jan.
--
Jan Nieuwenhuizen <[EMAIL PROTECTED]> | GNU LilyPond - The music typesetter
http://www.xs4all.nl/~jantien | http://www.lilypond.org
___
lilypond-devel mailing list
lilypond-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Re-write of documentation tree build process
On Thu, 21 Dec 2006, John Mandereau wrote: ... > doctype = '\n' > s = doctype + s Does the 'EN' means that the page language is English? If so, it should be modified to handle pages of any language. No. The whole string is just the ID of the HTML DTD and thus does not change as long as you stick to HTML 4.01 Transitional). See: http://www.w3.org/TR/html4/struct/global.html#h-7.2 I am not sure about the exact meaning of the "EN" within the DTD ID string; you probably would have to look into ISO 8879; but that's not online available, as far as I know. Greetings, Juergen ___ lilypond-devel mailing list lilypond-devel@gnu.org http://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Re-write of documentation tree build process
Han-Wen Nienhuys wrote:
> John Mandereau escreveu:
> > Before the next release including this patch, may I also add a link to
> > the French tutorial in index.html.in?
>
> Sure.
>
> Here are some final remarks, and Jan will have a look too. If Jan is
> ok, please push to stable/2.10.
Should I send a patch again, or work according to your remarks and push
to stable/2.10 (and possibly master) if Jan agrees?
> >I've moved most stuff from GNUmakefile.in:local-WWW-post to
> >stepmake/bin/www_post.py.
>
> Good! Can you put them in buildscripts/
Yes, I'll do this.
> (the stepmake bin/ directory isn't used anymore AFAIK.)
It's still used for add-html-footer.py, but as it needs to be renamed to
add_html_footer.py because import doesn't accept dashes in filenames,
let's move it to buildscripts. Actually, I'd even like to give it a more
accurate name like html_postprocess.py.
> > -$(INSTALL) -m 755 -d $(DESTDIR)$(webdir)
> > - cp -a $(outdir)/web-root/ $(DESTDIR)$(webdir)/
> > + cp -a $(outdir)/offline-root/ $(DESTDIR)$(webdir)/
> >
>
> I tend to use rsync for this; on MacOS X, cp -a doesn't work.
OK. Speaking of portability, I don't know how symlinks and hard links
are supported on different platforms. I hope it is supposed to work on
all Unix flavors with file systems supporting symlinks and hardlinks;
does it include MacOS X? (according to the Python docs, os.symlink() and
os.link() are only available on Unix.) Similarly, I'm ignorant about
hardlink and symlink support in Cygwin.
> > +def build_pages_dict (filelist):
> > +"""Build dictionnary of available translations of each page"""
> > +pages_dict = {}
> > +for f in filelist:
> > +m = html_re.match (f)
> > +if m:
> > +g = m.groups()
>
> hmm, of course the code of add_html_footer is a complete mess, but
> you're excused as it is the combined turd of years of negligence of
> add-html-footer.py by Jan and me. Still, you're welcome to clean it up
> a bit if possible.
I've already junked a bit of clearly unused stuff, but I didn't try to
check whether every bit of code is necessary to the HTML files of Lily
docs. For instance, I wonder whether remove_self_ref() is really used.
I'll try to use the Python debugger to find the unused bits of code.
Btw, I'd like to know you opinion about these possible cleanups:
- could we scrap i18n() and move the code to do_file(), as we may add
i18n in do_file too? (I'd like the footer to be translated, for
instance).
- we don't add any custom header, so all stuff using the "header"
variable could be dropped.
- what do all these '[.]' in regular expressions mean? If they should
match a dot in a string, why not just use r'\.' ? I've already
experienced that '[.]' is bogus in some cases.
- quoting add_html_footer.py:
> if re.search ('(?i) doctype = ' Transitional//EN">\n'
> s = doctype + s
Does the 'EN' means that the page language is English? If so, it should
be modified to handle pages of any language.
> I think the wiki stuff can go
Yes, according to what Graham recently wrote on the "LilyPond wiki"
thread.
> > +#subst = globals ()
> > +#subst.update (locals())
> > +for k in page_flavors.keys():
> > +page_flavors[k] = page_flavors[k] % vars ()
> > +
> > +# urg
> > +# maybe find first node?
> > +fallback_web_title = '-- --'
> > +# ugh, python2.[12] re is broken.
> > +#m = re.match ('.*?\(.*?\)', s, re.DOTALL)
> > +m = re.match ('[.\n]*?([.\n]*?)', s)
>
> similarly, we don't bother with python 2.1 and 2.2 anymore.
> Jan, which is the proper one now?
Python 2.3, according to INSTALL. I've already cleaned up some
deprecated functions (e.g string.foo() functions -> string methods).
I'll have a deeper look at that.
> > diff --git a/stepmake/bin/mirrortree.py b/stepmake/bin/mirrortree.py
> > new file mode 100644
> > index 000..4bbac88
> > --- /dev/null
> > +++ b/stepmake/bin/mirrortree.py
[snip]
> > +map (os.mkdir, target_dirs)
> > +
>
> btw, what happens if target_dirs already exists?
It simply crashes. It shouldn't happen as target_dirs are deleted with
"rm -rf" before www_post.py is called.
> > diff --git a/stepmake/bin/www_post.py b/stepmake/bin/www_post.py
> > new file mode 100644
> > index 000..a2fd52e
> > --- /dev/null
> > +++ b/stepmake/bin/www_post.py
[snip]
> > + targets = targets)
> > +find_html = subprocess.Popen (['find'] + doc_dirs + ['-name', '*.html'],
> > + stdout = subprocess.PIPE,
> > + bufsize = -1)
> > +find_output = find_html.stdout.read ()
> > +n = find_html.wait ()
>
>
> os.popen (..command..).read ()
>
> Still more python is to use os.walk(), but it's a little more work.
That can simply be done in mirrortree.hardlink_tree(), which could
return the HTML files list. Let's go for that.
--
John Mandereau <[EMAIL PROTECTED]>
_
Re: Re-write of documentation tree build process
John Mandereau escreveu:
> Before the next release including this patch, may I also add a link to
> the French tutorial in index.html.in?
Sure.
Here are some final remarks, and Jan will have a look too. If Jan is
ok, please push to stable/2.10.
>I've moved most stuff from GNUmakefile.in:local-WWW-post to
>stepmake/bin/www_post.py.
Good! Can you put them in buildscripts/
(the stepmake bin/ directory isn't used anymore AFAIK.)
> -$(INSTALL) -m 755 -d $(DESTDIR)$(webdir)
> - cp -a $(outdir)/web-root/ $(DESTDIR)$(webdir)/
> + cp -a $(outdir)/offline-root/ $(DESTDIR)$(webdir)/
>
I tend to use rsync for this; on MacOS X, cp -a doesn't work.
> +# For online docs with content negotiation, issue `make web
> FINAL_TARGETS=online'
> +# For both online and offline docs, issue `make web FINAL_TARGETS="offline
> online"'
> +FINAL_TARGETS = offline
WEB_TARGETS
> +def build_pages_dict (filelist):
> +"""Build dictionnary of available translations of each page"""
> +pages_dict = {}
> +for f in filelist:
> +m = html_re.match (f)
> +if m:
> +g = m.groups()
hmm, of course the code of add_html_footer is a complete mess, but
you're excused as it is the combined turd of years of negligence of
add-html-footer.py by Jan and me. Still, you're welcome to clean it up
a bit if possible.
> +wiki_page = ('v%s.%s-' % (versiontup[0], versiontup[1]) + f)
> +wiki_page = re.sub ('out-www/', '', wiki_page)
> +wiki_page = re.sub ('/', '-', wiki_page)
> +wiki_page = re.sub (r'\.-', '', wiki_page)
> +wiki_page = re.sub ('.html', '', wiki_page)
> +
> +wiki_string = ''
> +
> +if wiki_base:
> +wiki_string = (r'''Read
> comments on this page, or
> +add one.''' %
> + { 'wiki_base': wiki_base,
> +'wiki_page': wiki_page})
> +
I think the wiki stuff can go
> +#subst = globals ()
> +#subst.update (locals())
> +for k in page_flavors.keys():
> +page_flavors[k] = page_flavors[k] % vars ()
> +
> +# urg
> +# maybe find first node?
> +fallback_web_title = '-- --'
> +# ugh, python2.[12] re is broken.
> +#m = re.match ('.*?\(.*?\)', s, re.DOTALL)
> +m = re.match ('[.\n]*?([.\n]*?)', s)
similarly, we don't bother with python 2.1 and 2.2 anymore.
Jan, which is the proper one now?
> diff --git a/stepmake/bin/langdefs.py b/stepmake/bin/langdefs.py
> new file mode 100644
> index 000..b6e4508
> --- /dev/null
> +++ b/stepmake/bin/langdefs.py
> @@ -0,0 +1,26 @@
> +def lang_file_name (p, langext, ext):
> +if langext != '':
> +return p + '.' + langext + ext
> +return p + ext
> +
> +class language_def:
class names look like LanguageDef
> diff --git a/stepmake/bin/mirrortree.py b/stepmake/bin/mirrortree.py
> new file mode 100644
> index 000..4bbac88
> --- /dev/null
> +++ b/stepmake/bin/mirrortree.py
> @@ -0,0 +1,79 @@
> [EMAIL PROTECTED]@
> +
> +import re
> +import os
> +
> +def new_link_path (link, dir, r):
> +l = link.split ('/')
> +d = dir.split ('/')
> +i = 0
> +while i < len(d) and l[i] == '..':
> +if r.match (d[i]):
> +del l[i]
> +else:
> +i += 1
> +return '/'.join ([x for x in l if not r.match (x)])
> +
> +def hardlink_tree (input_roots = [],
> + process_dirs = '.*',
> + strip_dir_names = '',
> + exclude_dirs = '',
> + process_files = '.*',
> + exclude_files = '',
> + target_pattern = '',
> + targets = ['.']):
> +"""Mirror trees for different targets by hardlinking files.
> +
> +Arguments:
> + input_roots=DIRLIST use DIRLIST as input tree roots list
> + process_dir=PATTERN only process files in directories named PATTERN
> + strip_dir_names=PATTERN strip directories names matching PATTERN
> + (write their content to parent)
> + exclude_dir=PATTERN don't recurse into directories named PATTERN
> + process_files=PATTERNfilters files which are hardlinked
> + exclude_files=PATTERNexclude files named PATTERN
> + target_pattern=STRINGuse STRING as target root directory name
> pattern
> + targets=DIRLIST mkdir each directory in DIRLIST and mirrors
> the tree into each
> +"""
> +process_files_re = re.compile (process_files)
> +exclude_dirs_re = re.compile (exclude_dirs)
> +exclude_files_re = re.compile (exclude_files)
> +process_dirs_re = re.compile (process_dirs)
> +strip_dir_names_re = re.compile (strip_dir_names)
> +do_strip_dir_names_re = re.compile ('/(?:' + strip_dir_names + ')')
> +
> +if not '%s' in target_pattern:
> +target_pattern += '%s'
> +target_dirs = [target_pattern % s for s in targets]
> +
> +map (os.mkdir, target_dirs)
> +
btw, what happens if target_dirs already exists?
> diff --git a/stepm
Re: Re-write of documentation tree build process
John Mandereau wrote: > Han-Wen Nienhuys wrote: > > > diff --git a/stepmake/bin/add-html-footer.py > > > b/stepmake/bin/add-html-footer.py > > > index 08ff5ff..9a28237 100644 > > > --- a/stepmake/bin/add-html-footer.py > > > +++ b/stepmake/bin/add-html-footer.py > > > @@ -13,7 +13,7 @@ import getopt > > > index_url='' > > > top_url='' > > > changelog_file='' > > > -content_negotiation = False > > > +targets = [] > > > > I think that this script could be cleaned up a lot if didn't try to do > > two targets in one go. Can you look into doing something like > > > > find online-root -name '*.html' | xargs $(footify) > > find offline-root -name '*.html' | xargs $(footify) > > It's easily doable if: > -- all .html are hardlinked in both targets by build-trees, so the > "specific files" (ie. HTML files) stuff could be dropped. > -- add-html-footer is called for each target; for each .html file, it > reads it, unlink it, process the page and write it. > So, let's change to that. Finally, doing one or both targets in add-html-footer doesn't change things much: i18n() has to take a target argument and may needs return several page flavors anyway, since the link tweaking (like '.html' stripping in href's) is done in this function. > I'm sending another patch according to your comments asap. Here's finally the patch, against current stable/2.10 branch. add-html-footer.py and build-trees.py are rewritten as modules, and renamed respectively to add_html_footer.py and mirrortree.py. I've moved most stuff from GNUmakefile.in:local-WWW-post to stepmake/bin/www_post.py. I've also put the language definitions in a new module langdefs.py, since it could be shared with other scripts. Thus, adding/disabling/enabling a language translation could be done by editing only this file -- currently, there are also language defs in html-gettext, and translated docs are enabled in Documentation/GNUmakefile. Before the next release including this patch, may I also add a link to the French tutorial in index.html.in? For the coming days/weeks, here's my to-do list for the docs i18n: 1) centralize language definitions and enabling (see above) 2) fix html-gettext.py so it really translates all node names 3) i18n add_html_footer, index.html.in and examples.html. This is certainly much work relatively to the word count of these items, but that's worth it IMO. 4) write TRANSLATION text file, giving hints to documentation editors and translators: technical aspects, style guidelines in addition to README.txt, ... If you don't think these tasks are too urgent (you already know I'm not very quick ;-), I'll work on them. Regarding French translation of the user manual, introduction.itely awaits for final polishing and validation, and Frédéric Chiasson has started translating basic-notation.itely. Cheers, -- John Mandereau <[EMAIL PROTECTED]> diff --git a/GNUmakefile.in b/GNUmakefile.in index 7f3dee5..b7c5775 100644 --- a/GNUmakefile.in +++ b/GNUmakefile.in @@ -49,7 +49,7 @@ doc: install-WWW: -$(INSTALL) -m 755 -d $(DESTDIR)$(webdir) - cp -a $(outdir)/web-root/ $(DESTDIR)$(webdir)/ + cp -a $(outdir)/offline-root/ $(DESTDIR)$(webdir)/ $(MAKE) -C Documentation/user local-install-WWW $(MAKE) -C Documentation/user install-info @@ -73,46 +73,18 @@ local-install: final-install: @true -web-ext = html midi pdf png txt ly signature - -# For docball, issue `make web CONTENT_NEGOTIATION=' -CONTENT_NEGOTIATION = --content-negotiation -footify = $(PYTHON) $(step-bindir)/add-html-footer.py --name $(PACKAGE_NAME) --version $(TOPLEVEL_VERSION) $(CONTENT_NEGOTIATION) -footifymail = MAILADDRESS='http://post.gmane.org/post.php?group=gmane.comp.gnu.lilypond.bugs' - +# For online docs with content negotiation, issue `make web FINAL_TARGETS=online' +# For both online and offline docs, issue `make web FINAL_TARGETS="offline online"' +FINAL_TARGETS = offline local-WWW-post: # need UTF8 setting in case this is hosted on a website. echo -e 'AddDefaultCharset utf-8\nAddCharset utf-8 .html\nAddCharset utf-8 .en\nAddCharset utf-8 .nl\nAddCharset utf-8 .txt\n' > $(top-build-dir)/.htaccess $(PYTHON) $(buildscript-dir)/mutopia-index.py -o $(outdir)/examples.html input/ - echo '' > $(outdir)/index.html - echo 'Redirecting to the documentation index...' >> $(outdir)/index.html - - cd $(top-build-dir) && $(FIND) . -name '*.html' -print | $(footifymail) xargs $(footify) - - cd $(top-build-dir) && find Documentation input \ - $(web-ext:%=-path '*/out-www/*.%' -or) -type l \ - | grep -v 'lily-[0-9a-f]*.*pdf' \ - | grep -v '/fr/' \ - > $(outdir)/weblist - ls $(outdir)/*.html >> $(outdir)/weblist - -## urg: this is too hairy, should write a python script to do this. - -## rewrite file names so we lose out-www - rm -rf $(outdir)/web-root/ - mkdir $(outdir)/web-root/ -## urg slow. - cat $(outdir)/weblist | (cd $(top-build-dir); tar -cf- -T- ) | \ - tar -C $(outdir)/web-root/ -xf - - for dir in $(outdir)/web-root/ ; do \ - cd
Re: Re-write of documentation tree build process
Han-Wen Nienhuys wrote: > you could even remove the buildtrees and (un)linking stuff for HTML, > by reading from the out-www/ dir directly and writing into FOO-webroot/ > directly Good idea! It would save creating .html hardlinks for nuts. It implies that add-html-footer strips 'out-www/' path components in the file name before writing it, which is of course doable. -- John Mandereau <[EMAIL PROTECTED]> ___ lilypond-devel mailing list lilypond-devel@gnu.org http://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Re-write of documentation tree build process
John Mandereau escreveu: >> >> find online-root -name '*.html' | xargs $(footify) >> find offline-root -name '*.html' | xargs $(footify) > > It's easily doable if: > -- all .html are hardlinked in both targets by build-trees, so the > "specific files" (ie. HTML files) stuff could be dropped. > -- add-html-footer is called for each target; for each .html file, it > reads it, unlink it, process the page and write it. > So, let's change to that. you could even remove the buildtrees and (un)linking stuff for HTML, by reading from the out-www/ dir directly and writing into FOO-webroot/ directly > > I'm sending another patch according to your comments asap. > cool! > Cheers -- Han-Wen Nienhuys - [EMAIL PROTECTED] - http://www.xs4all.nl/~hanwen LilyPond Software Design -- Code for Music Notation http://www.lilypond-design.com ___ lilypond-devel mailing list lilypond-devel@gnu.org http://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Re-write of documentation tree build process
Han-Wen Nienhuys wrote:
> First of all: do you think you could shove the entire
>
> local-WWW-post
>
> target into a python script? In that case, we don't need any awkward
> quoting, and can junk most of the command line decoding, as we could
> do
>
> import mirrortree
> mirrortree.miror_tree ('out-www/offline-webroot', {'ignore-dirs':
> 'foo.*(ext|') , 'lose-dirs': 'more-regexes'})
>
> (or use keyword arguments to pass options.)
Agreed. It would make build-trees.py twice shorter. There should also be
a langdefs.py module centralizing language definitions: it would be used
by add-html-footer.py, html-gettext.py, and Documentation/GNUmakefile.
> > diff --git a/stepmake/bin/add-html-footer.py
> > b/stepmake/bin/add-html-footer.py
> > index 08ff5ff..9a28237 100644
> > --- a/stepmake/bin/add-html-footer.py
> > +++ b/stepmake/bin/add-html-footer.py
> > @@ -13,7 +13,7 @@ import getopt
> > index_url=''
> > top_url=''
> > changelog_file=''
> > -content_negotiation = False
> > +targets = []
>
> I think that this script could be cleaned up a lot if didn't try to do
> two targets in one go. Can you look into doing something like
>
> find online-root -name '*.html' | xargs $(footify)
> find offline-root -name '*.html' | xargs $(footify)
It's easily doable if:
-- all .html are hardlinked in both targets by build-trees, so the
"specific files" (ie. HTML files) stuff could be dropped.
-- add-html-footer is called for each target; for each .html file, it
reads it, unlink it, process the page and write it.
So, let's change to that.
[...]
> Nowadays, optparse is the standard module for handling option
> parsing. Can you change to that? See lilypond-book.py
> (get_option_parser) for an example how to use.
It won't be necessary, as I'm going to re-write build-trees (renamed
into mirrortree) and add-html-footer as modules.
> > +for s in ['common_f', 'specific_f', 'excluded_d',
> > + 'excluded_f', 'process_d', 'lost_d_names']:
> > +exec s + '_re = re.compile (' + s + ')'
>
>
> Don't use exec. If you must, try
>
>
> locals()[s + "_re"] = ..
I only used that to avoid typing six times the same line of code.
> what's _f and _d? Write _dir or _file if you mean that.
Yes, I mean that.
> usually, list comprehensions are shorter and clearer,
>
> [p % s for p in targets]
Oh, yes , I didn't remember that possibility in Python.
I'm sending another patch according to your comments asap.
Cheers
--
John Mandereau <[EMAIL PROTECTED]>
___
lilypond-devel mailing list
lilypond-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Re-write of documentation tree build process
John Mandereau escreveu:
> Hello all,
>
> Here's a (long-awaited) patch against the stable/2.10 branch.
>
> 'make web' can now generate two documentation trees in
> /out-www:
>
> - one in online-root/, where .png and .html extensions are stripped in
> HTML files, for hosting on a web site with content negociation.
>
> - the other in /offline-root, for browsing the docs on a hard disk.
>
> Only one of these trees or both can be generated, depending on the
> FINAL_TARGETS variable:
>
> make FINAL_TARGETS=offline web # same as 'make web'
> make FINAL_TARGETS=online web
>
> only builds one tree, and
>
> make FINAL_TARGETS="offline online" web
>
> builds both.
>
> If this can be applied and a possible link to the French tutorial in
> index.html.in is added, Jan and Han-Wen may want to release 2.10.3 or
> 2.10.2-2, as the current docball is broken because of .html stripping
> (maybe the 2.11 docball is broken too).
Hello John,
thanks for your patch. Before it's applied, I think it should be
cleaned up a bit first.
First of all: do you think you could shove the entire
local-WWW-post
target into a python script? In that case, we don't need any awkward
quoting, and can junk most of the command line decoding, as we could
do
import mirrortree
mirrortree.miror_tree ('out-www/offline-webroot', {'ignore-dirs':
'foo.*(ext|') , 'lose-dirs': 'more-regexes'})
(or use keyword arguments to pass options.)
> diff --git a/stepmake/bin/add-html-footer.py b/stepmake/bin/add-html-footer.py
> index 08ff5ff..9a28237 100644
> --- a/stepmake/bin/add-html-footer.py
> +++ b/stepmake/bin/add-html-footer.py
> @@ -13,7 +13,7 @@ import getopt
> index_url=''
> top_url=''
> changelog_file=''
> -content_negotiation = False
> +targets = []
I think that this script could be cleaned up a lot if didn't try to do
two targets in one go. Can you look into doing something like
find online-root -name '*.html' | xargs $(footify)
find offline-root -name '*.html' | xargs $(footify)
if that's unpractical, it's probably better to do one target a time,
so you can remove the
for t in targets:
if t == 'online'
elif t == 'offline'
logic.
> +def help ():
> +sys.stdout.write (r"""Usage: build-trees [OPTIONS]... TARGETS
> +Build trees for different targets by hardlinking input trees.
> +
> +Options:
> + --input-root=DIR use DIR as input tree root (default is current
> directory,
> + multiple roots may be specified with a comma
> separator)
> + --target-pattern=STRING use STRING as target root directory name pattern
> + --lose-dir=PATTERNlose directories whose name matches PATTERN in
> copies
> + (write its content to parent)
is this to remove out-www from the paths? I'd call it
--strip-component or similar. The difference between --exclude-dir is
not clear.
> + --process-dir=PATTERN only process files in directories whose name
> matches PATTERN
"only process directories matching PATTERN"
> + --common-files=PATTERNfilters files commmon to all trees
> + (they are hardlinked instead of being copied).
> + --specific-files=PATTERN filters files specific to different trees
> + (regular files are only hardlinked to first
> target).
> + --exclude-dir=PATTERN don't recurse into directories whose name matches
> PATTERN
> + --exclude-files=PATTERN exclude files whose name matches PATTERN
> + --dump-sfl=FILE dump specific files list to FILE
"sfl " -> no abbreviations please.
> + -h, --helpprint this help
> +
> +PATTERN should be a Python regular expression.
> +Common and specific files which are symlinks are always copied to all
> targets.
> +""")
> +sys.exit (0)
> +
> +(options, targets) = getopt.getopt(sys.argv[1:], 'h', [
> +'input-root=', 'lose-dir=', 'common-files=', 'help', 'specific-files=',
> +'exclude-dir=', 'exclude-files=', 'dump-sfl=', 'process-dir=',
> 'target-pattern='])
Nowadays, optparse is the standard module for handling option
parsing. Can you change to that? See lilypond-book.py
(get_option_parser) for an example how to use.
> +input_roots = []
> +target_pattern = ''
> +
> +common_f = '.*'
> +specific_f = '.*'
> +excluded_d = ''
> +excluded_f = ''
> +process_d = '.*'
> +lost_d_names = ''
> +sfl_dump = ''
> +
> +#file_list_re = re.compile (r'(?:^| )(?:[^ ]*|".*?")(?: |$)')
> +
> +for opt in options:
> +o = opt[0]
> +a = opt[1]
> +if o == '--input-root':
> +input_roots = re.split (',', a)
> +elif o == '--lose-dir':
> +lost_d_names = a
> +elif o == '--common-files':
> +common_f = a
> +elif o == '--specific-files':
> +specific_f = a
> +elif o == '-h' or o == '--help':
> +help ()
> +elif o == '--exclude-dir':
> +excluded_d = a
> +elif o == '--exclude-files':
> +excluded_f = a
> +elif o == '--process-d
Re-write of documentation tree build process
Hello all, Here's a (long-awaited) patch against the stable/2.10 branch. 'make web' can now generate two documentation trees in /out-www: - one in online-root/, where .png and .html extensions are stripped in HTML files, for hosting on a web site with content negociation. - the other in /offline-root, for browsing the docs on a hard disk. Only one of these trees or both can be generated, depending on the FINAL_TARGETS variable: make FINAL_TARGETS=offline web # same as 'make web' make FINAL_TARGETS=online web only builds one tree, and make FINAL_TARGETS="offline online" web builds both. If this can be applied and a possible link to the French tutorial in index.html.in is added, Jan and Han-Wen may want to release 2.10.3 or 2.10.2-2, as the current docball is broken because of .html stripping (maybe the 2.11 docball is broken too). Cheers, -- John Mandereau <[EMAIL PROTECTED]> diff --git a/GNUmakefile.in b/GNUmakefile.in index 7f3dee5..37c47e4 100644 --- a/GNUmakefile.in +++ b/GNUmakefile.in @@ -49,7 +49,7 @@ doc: install-WWW: -$(INSTALL) -m 755 -d $(DESTDIR)$(webdir) - cp -a $(outdir)/web-root/ $(DESTDIR)$(webdir)/ + cp -a $(outdir)/offline-root/ $(DESTDIR)$(webdir)/ $(MAKE) -C Documentation/user local-install-WWW $(MAKE) -C Documentation/user install-info @@ -73,11 +73,12 @@ local-install: final-install: @true -web-ext = html midi pdf png txt ly signature +common-web-ext = midi|pdf|png|txt|ly|signature -# For docball, issue `make web CONTENT_NEGOTIATION=' -CONTENT_NEGOTIATION = --content-negotiation -footify = $(PYTHON) $(step-bindir)/add-html-footer.py --name $(PACKAGE_NAME) --version $(TOPLEVEL_VERSION) $(CONTENT_NEGOTIATION) +# For online docs with content negotiation, issue `make web FINAL_TARGETS=online' +# For both online and offline docs, issue `make web FINAL_TARGETS="offline online"' +FINAL_TARGETS = offline +footify = $(PYTHON) $(step-bindir)/add-html-footer.py --name $(PACKAGE_NAME) --version $(TOPLEVEL_VERSION) --targets "$(foreach i, $(FINAL_TARGETS), $(i)-root:$(i) )" footifymail = MAILADDRESS='http://post.gmane.org/post.php?group=gmane.comp.gnu.lilypond.bugs' @@ -86,33 +87,14 @@ local-WWW-post: # need UTF8 setting in case this is hosted on a website. echo -e 'AddDefaultCharset utf-8\nAddCharset utf-8 .html\nAddCharset utf-8 .en\nAddCharset utf-8 .nl\nAddCharset utf-8 .txt\n' > $(top-build-dir)/.htaccess $(PYTHON) $(buildscript-dir)/mutopia-index.py -o $(outdir)/examples.html input/ + $(foreach i, $(FINAL_TARGETS), rm -rf $(outdir)/$(i)-root &&) true + $(PYTHON) $(step-bindir)/build-trees.py --input-root Documentation,input --lose-dir out-www --process-dir out-www --common-files '.*[.](?:$(common-web-ext)$$)' --specific-files '.*[.]html$$' --exclude-dir '(fr|po|out)(/|$$)' --exclude-files 'lily-[0-9].*.pdf' --dump-sfl $(outdir)/html-list --target-pattern $(outdir)/%s-root $(FINAL_TARGETS) echo '' > $(outdir)/index.html echo 'Redirecting to the documentation index...' >> $(outdir)/index.html - - cd $(top-build-dir) && $(FIND) . -name '*.html' -print | $(footifymail) xargs $(footify) - - cd $(top-build-dir) && find Documentation input \ - $(web-ext:%=-path '*/out-www/*.%' -or) -type l \ - | grep -v 'lily-[0-9a-f]*.*pdf' \ - | grep -v '/fr/' \ - > $(outdir)/weblist - ls $(outdir)/*.html >> $(outdir)/weblist - -## urg: this is too hairy, should write a python script to do this. - -## rewrite file names so we lose out-www - rm -rf $(outdir)/web-root/ - mkdir $(outdir)/web-root/ -## urg slow. - cat $(outdir)/weblist | (cd $(top-build-dir); tar -cf- -T- ) | \ - tar -C $(outdir)/web-root/ -xf - - for dir in $(outdir)/web-root/ ; do \ - cd $$dir && \ - for a in `find . -name out-www`; do \ - rsync -a --link-dest $$a/ $$a/ $$a/.. ; \ - rm -rf $$a ; \ - done \ - done + cd $(outdir) && ls --format=single-column *.html >> html-list + $(foreach i, $(FINAL_TARGETS), echo $(TOPLEVEL_VERSION) > $(outdir)/$(i)-root/VERSION && \ + cp $(outdir)/*.html $(outdir)/$(i)-root/ &&) true + cd $(outdir) && cat html-list | $(footifymail) xargs $(footify) tree-prefix = $(outdir) tree-bin = $(tree-prefix)/bin diff --git a/stepmake/bin/add-html-footer.py b/stepmake/bin/add-html-footer.py index 08ff5ff..9a28237 100644 --- a/stepmake/bin/add-html-footer.py +++ b/stepmake/bin/add-html-footer.py @@ -13,7 +13,7 @@ import getopt index_url='' top_url='' changelog_file='' -content_negotiation = False +targets = [] package_name = '' package_version = '' @@ -69,7 +69,10 @@ Add header, footer and top of ChangLog f Options: --changelog=FILE use FILE as ChangeLog [ChangeLog] - --content-negotiation strip .html and .png from urls + --targets=TARGETS set targets +TARGETS is a space-separated list of PATH:TYPE +where PATH is the path prefix of HTML-FILE + TYPE is one of online, offline --footer=FILE use FILE as footer --header=FILE use FILE as header -h, --helpprint
