Re: advice for pre-generating documentation
Hello, to round this up: * Braden McDaniel wrote on Fri, Feb 12, 2010 at 05:28:26PM CET: On Fri, 2010-02-12 at 12:59 +0100, Stefano Lattarini wrote: At Friday 12 February 2010, Braden McDaniel bra...@endoframe.com wrote: Actually, EXTRA_DIST can pull in a whole subdirectory. Thank you for the information, I didn't know that (or I forgot it). I guess It's time for me to re-read the Automake documentation. Wildcards work there as well. Mmhhh... technically, you are right on this point too (as can be seen by reading any Automake-generated makefile), but I couldn't find anything about this feature in the manual... Are you sure that wildcards in EXTRA_DIST are supported by design rather than as a side-effect of implementation details? In the last case, it could be a bad idea to exploit that feature. I have no idea. It's worked that way for a long time. While in general I agree with the Automake philosophy regarding wildcards, Doxygen integration is one place where they make a whole lot of sense. I hope the Automake maintainers don't decide to break this. Wildcards in EXTRA_DIST are pretty much a must-not-break because too many packages depend on it. Of course the caveats mentioned in the FAQ apply. extra[567].test and distdir.test check semantics of putting a directory in EXTRA_DIST, subpkg3.test ensures files and directories may thus be shared by package and subpackage. I guess a test using an explicit wild card would be good as well, as I'm not entirely sure all make implementations handle wildcards in prerequisites correctly; the `distdir' target has `$(EXTRA_DIST)' as prerequisite. Cheers, Ralf
Re: advice for pre-generating documentation
At Saturday 20 February 2010, Ralf Wildenhues ralf.wildenh...@gmx.de wrote: Hello, to round this up: Wildcards in EXTRA_DIST are pretty much a must-not-break because too many packages depend on it. Of course the caveats mentioned in the FAQ apply. extra[567].test and distdir.test check semantics of putting a directory in EXTRA_DIST, subpkg3.test ensures files and directories may thus be shared by package and subpackage. I guess a test using an explicit wild card would be good as well, as I'm not entirely sure all make implementations handle wildcards in prerequisites correctly; the `distdir' target has `$(EXTRA_DIST)' as prerequisite. Cheers, Ralf Hello Ralf. If it's ok for you, I'm soon going to submit a (tentative) patch on the automake-patches list, which will add tests about the use wildcards in EXTRA_DIST. Regards, Stefano
Re: advice for pre-generating documentation
On Thu, Feb 11, 2010 at 5:08 PM, Gaetan Nadon mems...@videotron.ca wrote: generated using tools such as doxygen, asciidoc, xmlto, groff, and ps2pdf. I can state some reasons why generated docs and included in the tarball: This is interesting and many seem to agree here, but I think this is wrong. In my team we made bad experiences and now have the rules that: - a file either is autogen OR in CVS/SVN but not both - a file either is generated by make OR in srcdist - generated files (doxygen, libs) are in bindist only If someone wants to read the compiled documentation (be it HTML or PDF) or just use any other compiled object (such as a lib), he should use a bindist or a webbrowser. We use doxygen is most cases only for API documentation (I think this is the normal case), but our developers here usually seem never to read the HTML doc but read the .h (.java...) files. Including documentation may also be wrong when it is conditional, but this might be a special case. When CVS snaps or alike, let's say you check out trunk HEAD and run make dist, the generated documentation also might be invalid for example because required Change History information may not be filled or so. I think typically someone can expect valid documentation only from released versions. I think with doxygen it is difficult to get the decencies right, don't know if it even works, so how do you update the documentation? Are you re-generating it when running make dist? 1) convenience 2) the target platform building the tarball does not have the tool 3) the target platform building the tarball has the tool, but at the wrong version 4) the target may have the tool but different (compatible or incompatible) dependencies leading to different files be generated (might be less important for documentation). 3) unconditionally add html generated files in EXTRA_DIST: this will cause 'make dist' to fail if the platform creating a tarball does not have the doc tool. So in each srcdist you include all the HTML, PDF, man pages and whatever files generated, right? Is this the intended automake way? I just looked to some arbitrary lib and it has 8 MB sources but 20 MB doc (just HTML+PDF), so it would bloat up srcdist a lot... How to avoid to include outdated documentation? EXTRA_DIST = all the html and doxygen files listed here Does this mean to list all files individually here? In my example case I had to list 1422 files, wow... But how to maintain that? If anyone documents some struct somewhere, a new file will be generated, but how to know what to add to EXTRA_DIST? Do you create/use some include filelist.mak or so? oki, Steffen
Re: advice for pre-generating documentation
On Thu, Feb 11, 2010 at 6:06 PM, Andreas Jellinghaus a...@dungeon.inka.de wrote: also I wonder: what about builddir vs. sourcedir? how do you handle that? does automake handle that automaticaly? make does handle it (at least GNU Make, I don't know others): If you have in Makefile.am let's say: EXTRA_DIST = sourcefile.c sourcefile.o test: sourcefile.c sourcefile.o @echo $^ it will work even if sourcefile.o is generated from sourcefile.c; both will be in srcdist. Make will set things like $^ correctly because VPATH is considered, that means `make test' tells e.g.: stef...@host:/tmp/steffen/.../build/i386-gnulinux/test # make test ../../../../ccomm/test/sourcefile.c sourcefile.o -- cool aint??? :-) but I would ever put generated sources to EXTRA_DIST, because in the dist in this example file you would find: .../test/sourcefile.c .../test/sourcefile.o then if you compile from that dist with builddir != srcdir and deps enforce generation of sourcefile.o you end up with 2 sourcefile.o files: one in srcdir and other in builddir, which (IMHO) must never ever happen. oki, Steffen
Re: advice for pre-generating documentation
At Friday 12 February 2010, Braden McDaniel bra...@endoframe.com wrote: Actually, EXTRA_DIST can pull in a whole subdirectory. Thank you for the information, I didn't know that (or I forgot it). I guess It's time for me to re-read the Automake documentation. Wildcards work there as well. Mmhhh... technically, you are right on this point too (as can be seen by reading any Automake-generated makefile), but I couldn't find anything about this feature in the manual... Are you sure that wildcards in EXTRA_DIST are supported by design rather than as a side-effect of implementation details? In the last case, it could be a bad idea to exploit that feature. Regards, Stefano
Re: advice for pre-generating documentation
On Fri, 2010-02-12 at 12:59 +0100, Stefano Lattarini wrote: At Friday 12 February 2010, Braden McDaniel bra...@endoframe.com wrote: Actually, EXTRA_DIST can pull in a whole subdirectory. Thank you for the information, I didn't know that (or I forgot it). I guess It's time for me to re-read the Automake documentation. Wildcards work there as well. Mmhhh... technically, you are right on this point too (as can be seen by reading any Automake-generated makefile), but I couldn't find anything about this feature in the manual... Are you sure that wildcards in EXTRA_DIST are supported by design rather than as a side-effect of implementation details? In the last case, it could be a bad idea to exploit that feature. I have no idea. It's worked that way for a long time. While in general I agree with the Automake philosophy regarding wildcards, Doxygen integration is one place where they make a whole lot of sense. I hope the Automake maintainers don't decide to break this. But if they do, we still have dist-hook. -- Braden McDaniel bra...@endoframe.com
Re: advice for pre-generating documentation
On Fri, 2010-02-12 at 12:21 +0100, Steffen Dettmer wrote: On Thu, Feb 11, 2010 at 5:08 PM, Gaetan Nadon mems...@videotron.ca wrote: generated using tools such as doxygen, asciidoc, xmlto, groff, and ps2pdf. I can state some reasons why generated docs and included in the tarball: This is interesting and many seem to agree here, but I think this is wrong. I don't like it either, the makefile gets complicated and error prone. The main argument is that not all platforms have the tool so they would not be able to install the doc. I think there are other ways to handle such situation like publishing doc-only tarballs. The docs are not binaries and don't have to be built on each platform. If you can avoid putting doc generated files in the tarball, you will save yourself the trouble. In my team we made bad experiences and now have the rules that: - a file either is autogen OR in CVS/SVN but not both - a file either is generated by make OR in srcdist - generated files (doxygen, libs) are in bindist only If someone wants to read the compiled documentation (be it HTML or PDF) or just use any other compiled object (such as a lib), he should use a bindist or a webbrowser. We use doxygen is most cases only for API documentation (I think this is the normal case), but our developers here usually seem never to read the HTML doc but read the .h (.java...) files. Including documentation may also be wrong when it is conditional, but this might be a special case. When CVS snaps or alike, let's say you check out trunk HEAD and run make dist, the generated documentation also might be invalid for example because required Change History information may not be filled or so. I think typically someone can expect valid documentation only from released versions. I think with doxygen it is difficult to get the decencies right, don't know if it even works, so how do you update the documentation? Are you re-generating it when running make dist? 1) convenience 2) the target platform building the tarball does not have the tool 3) the target platform building the tarball has the tool, but at the wrong version 4) the target may have the tool but different (compatible or incompatible) dependencies leading to different files be generated (might be less important for documentation). 3) unconditionally add html generated files in EXTRA_DIST: this will cause 'make dist' to fail if the platform creating a tarball does not have the doc tool. So in each srcdist you include all the HTML, PDF, man pages and whatever files generated, right? Is this the intended automake way? I just looked to some arbitrary lib and it has 8 MB sources but 20 MB doc (just HTML+PDF), so it would bloat up srcdist a lot... How to avoid to include outdated documentation? EXTRA_DIST = all the html and doxygen files listed here Does this mean to list all files individually here? In my example case I had to list 1422 files, wow... But how to maintain that? If anyone documents some struct somewhere, a new file will be generated, but how to know what to add to EXTRA_DIST? Do you create/use some include filelist.mak or so? oki, Steffen signature.asc Description: This is a digitally signed message part
advice for pre-generating documentation
Hi, we generate some docs with doxygen, others with a shell script using wget and xslt to download our wiki and create local html files from that). that mechanism is currently enabled with --enable-doc, off by default as it is time-consuming and usualy not wanted. the problem I have is this: * I want people to checkout svn and compile and test the software without generating documentation. * if people want the docs, they should be able to create them too. * if I run make distcheck, I want the existing documentation to be included in the tar.gz file. * if I run make maintainer-clean, I want generate documentation to be removed, so I can diff between a modified checkout and an unmodified svn checkout. but our code to do all that is really ugly. so I wonder what the best way to implement something like this is. can you give me an advice? from my point of view documentation is on the same level as configure and makefile.in - something we don't keep in svn, but generate as first step after the checkout and then distribute as part of the tar.gz to users. is there some hook I can use to run custom commands easily from autoconf/make/libtool? maybe even in a way I can turn it on? (thus generate documentation only if asked) any better idea? thanks for your help. Regards, Andreas p.s. if you want to see the code we currently have: svn co http://www.opensc-project.org/svn/openct/trunk and look at the docs/ directory.
Re: advice for pre-generating documentation
On Thu, 2010-02-11 at 15:31 +0100, Andreas Jellinghaus wrote: Hi, we generate some docs with doxygen, others with a shell script using wget and xslt to download our wiki and create local html files from that). that mechanism is currently enabled with --enable-doc, off by default as it is time-consuming and usualy not wanted. the problem I have is this: * I want people to checkout svn and compile and test the software without generating documentation. * if people want the docs, they should be able to create them too. * if I run make distcheck, I want the existing documentation to be included in the tar.gz file. * if I run make maintainer-clean, I want generate documentation to be removed, so I can diff between a modified checkout and an unmodified svn checkout. but our code to do all that is really ugly. so I wonder what the best way to implement something like this is. can you give me an advice? from my point of view documentation is on the same level as configure and makefile.in - something we don't keep in svn, but generate as first step after the checkout and then distribute as part of the tar.gz to users. is there some hook I can use to run custom commands easily from autoconf/make/libtool? maybe even in a way I can turn it on? (thus generate documentation only if asked) any better idea? thanks for your help. We have the same challenge at X.Org. Documents from various sources are generated using tools such as doxygen, asciidoc, xmlto, groff, and ps2pdf. I can state some reasons why generated docs and included in the tarball: 1) convenience 2) the target platform building the tarball does not have the tool 3) the target platform building the tarball has the tool, but at the wrong version In addition, we must prevent a platform that does not have the doc tool to be able to create a tarball for distribution with missing generated documentation. This is accomplished by having 'make dist' fail while 'make all' and 'make install' works. You have stated the requirements very well, and I meant to ask the same question a little while ago. I can share how I do it in the makefile if it can help. 1) an AM_CONDITIONAL --enable-doc which defines ENABLE_DOC : this allows the builder to include the docs or not, for whatever reason. 2) an AM_CONDITIONAL --with-doxygen which defines HAVE_DOXYGEN: the default behaviour is to skip the docs if the tool is missing. A builder can use --without-doxygen if the tool is at the wrong version or if it fails. This way other documentation can be build. 3) unconditionally add html generated files in EXTRA_DIST: this will cause 'make dist' to fail if the platform creating a tarball does not have the doc tool. 4) Add output message telling builder why 'make dist' fails when doc tool is missing 5) maintainer-clean-local: cleans the generated files. Uses the automake concept of special tool which the platform building the tarball may not have. This prevents removing generating docs with a regular make clean by accident. A makefile would like this: if ENABLE_DOC if HAVE_DOXYGEN doc target: @command maintainer-clean-local: @clean generated file else doc target: @echo Tell builder docs cannot be generated. Install the tool or use --with-doxygen endif else doc target: @echo Tell builder docs cannot be generated. Use --enable-doc endif EXTRA_DIST = all the html and doxygen files listed here I don't see a reason for the code to be ugly, in the sense that automake can be used as designed. It does increase the complexity and the requirements must be understood. I am setting aside any doxygen specific issues in that statement. We have written macros for the 2 conditionals in configure.ac as they are used in several packages. Regards, Gaetan Regards, Andreas p.s. if you want to see the code we currently have: svn co http://www.opensc-project.org/svn/openct/trunk and look at the docs/ directory. signature.asc Description: This is a digitally signed message part
Re: advice for pre-generating documentation
Ah, thanks. I didn't know about EXTRA_DIST. Is there some mechanism I can put a wildcard into automake files? for example it would be easier to specify *.html, as the script downloads all wiki pages, and thus new pages would be added automaticaly that way. also I wonder: what about builddir vs. sourcedir? how do you handle that? if people have a tar.gz with pre-build documentation, it is in sourcedir. if people checkout svn and build in a seperate build root, the documentation is created in builddir. does automake handle that automaticaly? and is there a nice way to remove the documentation with make clean or make distclean if it was generated, but not remove it if it comes from the tar.gz? use the same if/else/endif around the CLEANFILES settings? Thanks! Regards, Andreas
Re: advice for pre-generating documentation
On Thu, 2010-02-11 at 18:06 +0100, Andreas Jellinghaus wrote: Ah, thanks. I didn't know about EXTRA_DIST. Is there some mechanism I can put a wildcard into automake files? for example it would be easier to specify *.html, as the script downloads all wiki pages, and thus new pages would be added automaticaly that way. Automake does not support or recommend wildcards: http://www.gnu.org/software/automake/manual/automake.html#Wildcards also I wonder: what about builddir vs. sourcedir? how do you handle that? I am not usre I understand the question and I am not the one who originally wrote the makefile for doxygen, here is an example of a doxygen file target: $(builddir)/doxygen.head: $(LN_S) $(srcdir)/doxygen.head $@ You need to test by doing a VPATH build (where the build directory is not the source directory. This is how 'make distcheck' builds. It will be easy to debug in a VPATH build. http://www.gnu.org/software/automake/manual/automake.html#VPATH-Builds if people have a tar.gz with pre-build documentation, it is in sourcedir. if people checkout svn and build in a seperate build root, the documentation is created in builddir. That's correct, if one choose to do a VPATH build. does automake handle that automaticaly? I don't think there is anything for Automake to handle. The docs may be different as the source code in the repo may not be the same as the one in the tarball. Using a tarball as the seed for development source and then downloading some or all of the code from svn is a development task. It's one of n ways of getting source and building. and is there a nice way to remove the documentation with make clean or make distclean if it was generated, but not remove it if it comes from the tar.gz? use the same if/else/endif around the CLEANFILES settings? Automake does not know whether you are working from a tarball or from a repo. Opinions will vary on this, but this is what we do: make clean: does the usual, but does not remove gen docs. The reason is that the tarball builder may not have doxygen to re-create. He wanted to clean the object code, not the docs! make maintainer-clean: does the clean, plus it invokes our maintainer-clean-local target and any file listed in the MAINTAINRECLEANFILES variable. This target was designed by Automake for situations like this where the builder may not have the tools to re-create some generated files. So the doxygen generated files get removed. This is the comment that it will issue: @echo This command is intended for maintainers to use @echo it deletes files that may require special tools to rebuild. The answer is yes, there is a nice way to remove the docs: make 'maintainer-clean'. And yes, there is a nice way to clean all objects code without removing the docs: 'make clean'. Whether the docs came from a tarball, svn or hand dropped on the disk is irrelevant and there is no way to know. The hard part is not the coding of the makefile but deciding on how it should work for the user. The way it works for one project may not be ideal for another one. Thanks! Regards, Andreas signature.asc Description: This is a digitally signed message part
Re: advice for pre-generating documentation
At Thursday 11 February 2010, Andreas Jellinghaus a...@dungeon.inka.de wrote: Hi, Hello Andreas. I'm by no means an expert on complex Automake usage, but maybe I can provide a hint or two. we generate some docs with doxygen, others with a shell script using wget and xslt to download our wiki and create local html files from that). that mechanism is currently enabled with --enable-doc, off by default as it is time-consuming and usualy not wanted. the problem I have is this: * I want people to checkout svn and compile and test the software without generating documentation. * if people want the docs, they should be able to create them too. * if I run make distcheck, I want the existing documentation to be included in the tar.gz file. Maybe a dist-hook can help here: - http://www.gnu.org/software/automake/manual/html_node/The-dist-Hook.html A dist-hook can be especially useful if you don't know in advance the names of all the files to be included in the distribution tarball, in which case EXTRA_DIST is useless (and if I recall correctly, Doxygen generates *a lot* of files when creating HTML output). * if I run make maintainer-clean, I want generate documentation to be removed, so I can diff between a modified checkout and an unmodified svn checkout. As already pointeided out by Gaetan, writing a `maintainer-clean-local' target can be useful here; you can find more information at: - http://www.gnu.org/software/automake/manual/html_node/Clean.html - http://www.gnu.org/software/automake/manual/html_node/Extending.html HTH, Stefano
Re: advice for pre-generating documentation
Hi Andreas, On 2/11/10 12:06 PM, Andreas Jellinghaus wrote: also I wonder: what about builddir vs. sourcedir? how do you handle that? if people have a tar.gz with pre-build documentation, it is in sourcedir. if people checkout svn and build in a seperate build root, the documentation is created in builddir. does automake handle that automaticaly? IMHO it seems confusing and error prone to treat those two cases separately. To me, a file is either located in srcdir or builddir, independent on whether I build from an svn checkout or from a tarball. For instance, the `Makefile.in' is always located in srcdir, also when it is created (built) in an svn checkout. You said before that you wanted to treat documentation as `Makefile.in' and then I would have the docs in srcdir. Thanks, Peter
Re: advice for pre-generating documentation
On Fri, 2010-02-12 at 00:03 +0100, Stefano Lattarini wrote: At Thursday 11 February 2010, Andreas Jellinghaus a...@dungeon.inka.de wrote: Hi, Hello Andreas. I'm by no means an expert on complex Automake usage, but maybe I can provide a hint or two. we generate some docs with doxygen, others with a shell script using wget and xslt to download our wiki and create local html files from that). that mechanism is currently enabled with --enable-doc, off by default as it is time-consuming and usualy not wanted. the problem I have is this: * I want people to checkout svn and compile and test the software without generating documentation. * if people want the docs, they should be able to create them too. * if I run make distcheck, I want the existing documentation to be included in the tar.gz file. Maybe a dist-hook can help here: - http://www.gnu.org/software/automake/manual/html_node/The-dist-Hook.html A dist-hook can be especially useful if you don't know in advance the names of all the files to be included in the distribution tarball, in which case EXTRA_DIST is useless (and if I recall correctly, Doxygen generates *a lot* of files when creating HTML output). Actually, EXTRA_DIST can pull in a whole subdirectory. Wildcards work there as well. That doesn't help you with install; but you can add an install-data-local hook for that. -- Braden McDaniel bra...@endoframe.com
