Re: Regarding the JAVA primary
* NightStrike wrote on Sat, Apr 24, 2010 at 02:08:43AM CEST: On Mon, Apr 19, 2010 at 2:25 PM, John Calcote john.calc...@gmail.com wrote: A problem I foresee is providing the globbing functionality to makefile commands. We'd almost need a new auxiliary script (like install-sh) to generate lists of files from such glob specs. Not sure yet from where the primary functionality would come -- perhaps a java utility, so that the same level of portability would be available to java builds as the source that's being built. That is, if someone uses the JAVA primary, he/she can expect to be required to have additional build functionality available, in the form of a JVM and javac compiler. Just a thought. You can resolve the globbing spec at automake time instead of make time. That way, Makefile.in still contains a static list. Would you consider the following semantics intuitive? edit Makefile.am ... run automake add another java file build fails due to new java file not being considered Remember that it would not be a good idea to rerun automake at each instance of 'make'. Here's another reason against wildcards in prerequisites: they are not portable to non-GNU make. Period. GNU make has $(wildcard ...) which is well-defined, and GNU make also most of the time does the right thing when you place a shell-style wildcard in a target or prerequisite list (but see http://www.gnu.org/software/make/manual/html_node/Wildcard-Pitfall.html for pitfalls!), but non-GNU make will normally not do what you expect. More precisely, while some non-GNU makes seem to expand wildcards in prerequisite lists, they don't actually match the expanded words against other targets listed in the makefile (tested with some versions of IRIX, AIX, and Solaris make). I found only FreeBSD make to do the matching besides GNU make. Tested with this script in a fresh directory: cat Makefile \EOF all: all1 all2 all3 echo updating $@ all1: newer* echo updating $@ all2: older* echo updating $@ all3: outdated* echo updating $@ new* old* outd*: echo updating $@ outd*: FORCE FORCE: EOF touch older1 older2 outdated1 outdated2 sleep 1 touch all1 all2 all3 sleep 1 touch newer1 newer2 make Only GNU and FreeBSD make will remake all1, outdated1, outdated2, and all2 before remaking all. The rest will only remake all1 before all. So, in case you have generated files, that'll just silently break your dependencies. I still need to rewrite this for an addition to the FAQ. Cheers, Ralf
Re: Regarding the JAVA primary
On Mon, Apr 19, 2010 at 2:25 PM, John Calcote john.calc...@gmail.com wrote: A problem I foresee is providing the globbing functionality to makefile commands. We'd almost need a new auxiliary script (like install-sh) to generate lists of files from such glob specs. Not sure yet from where the primary functionality would come -- perhaps a java utility, so that the same level of portability would be available to java builds as the source that's being built. That is, if someone uses the JAVA primary, he/she can expect to be required to have additional build functionality available, in the form of a JVM and javac compiler. Just a thought. You can resolve the globbing spec at automake time instead of make time. That way, Makefile.in still contains a static list. If you went this route, I'd be curious how quickly someone ports it to other primaries :)
Regarding the JAVA primary
Hi Ralf, I've been thinking a lot about the JAVA primary lately. It turns out that Automake's handling of Java sources is pretty efficient. Experiments indicate that building ~500 Java source files in a single command takes about 15 seconds on a 1.8 GHz CPU with 512 MB RAM. That same set of 500 sources, built individually can take upwards of 500 seconds - a 40 times increase in compile time. (GNU Make, O'Reilly) In other words, it's simply better to not manage individual source/object dependencies in Java. You win the battle, so to speak -- but you lose the war. Since the current implementation of the JAVA primary is not managing individual source/object dependencies (something that's difficult to do anyway because of inner and anonymous class definitions), would it not be prudent to remove the restriction regarding needing to specify all source files individually in Makefile.am -- at least for the JAVA primary? Builds in the Java world generally specify source files found within a subtree using a globbing mechanism, with optionally specified inclusions and exclusions. And the layout of that subtree defines the packages to which classes belong. Would it not be fair to say that all files found matching the source specification within a specified subtree are distributed within the directory layout to which they belong? In other words, distribution (for Java sources only) would include the same set of files specified in the Java source globbing pattern. Here's an example: sources = src/**/*.java exclude = src/examples/*.java Distributed Java sources would include everything defined by /sources/. Excluded files would be excluded from the build, but not from distribution. I'm not stealing this concept entirely from ant. Most of the Java IDE's (NetBeans, IntelliJ, Eclipse, etc) also use almost the same mechanism for specifying sources belonging to a particular build. I recognize that this is a significant deviation from existing Autotools methodology, but I'm not sure we can make any real forward progress in Autotools Java builds without making a few such concessions. A problem I foresee is providing the globbing functionality to makefile commands. We'd almost need a new auxiliary script (like install-sh) to generate lists of files from such glob specs. Not sure yet from where the primary functionality would come -- perhaps a java utility, so that the same level of portability would be available to java builds as the source that's being built. That is, if someone uses the JAVA primary, he/she can expect to be required to have additional build functionality available, in the form of a JVM and javac compiler. Just a thought. One thing we can do at this point is to define JAR and WAR primaries that build and install (in appropriate locations), .jar and .war files. I've got a few ideas I'll try to codify and send out shortly. John
Re: Regarding the JAVA primary
Hi John, * John Calcote wrote on Mon, Apr 19, 2010 at 08:25:22PM CEST: Since the current implementation of the JAVA primary is not managing individual source/object dependencies (something that's difficult to do anyway because of inner and anonymous class definitions), would it not be prudent to remove the restriction regarding needing to specify all source files individually in Makefile.am -- at least for the JAVA primary? I guess that would be possible, at least technically. Builds in the Java world generally specify source files found within a subtree using a globbing mechanism, with optionally specified inclusions and exclusions. And the layout of that subtree defines the packages to which classes belong. Would it not be fair to say that all files found matching the source specification within a specified subtree are distributed within the directory layout to which they belong? In other words, distribution (for Java sources only) would include the same set of files specified in the Java source globbing pattern. The issue is that it goes against one principle otherwise followed throughout Automake: the user cannot have junk files in her source nor her build tree that match the patterns. At least for some users, this is a feature; I for one have unfinished files in my source trees, or even in a version control branch. I do admit though that distributed version control makes this less relevant. I recognize that this is a significant deviation from existing Autotools methodology, but I'm not sure we can make any real forward progress in Autotools Java builds without making a few such concessions. We can have new semantics; in case of doubt we can enable them under a new Automake option only. A problem I foresee is providing the globbing functionality to makefile commands. We'd almost need a new auxiliary script (like install-sh) to generate lists of files from such glob specs. If it's not very complex it can be done in a makefile variable scriptlet. I haven't understood the JAR and WAR notes you wrote about. Cheers, Ralf
Re: Regarding the JAVA primary
On Mon, Apr 19, 2010 at 8:25 PM, John Calcote john.calc...@gmail.com wrote:
[...]
Builds in the Java world generally specify source files found
within a subtree using a globbing mechanism, with optionally
specified inclusions and exclusions.
Yes, they do. BTW, does anyone know why?
With some sarcasm someone could tell that it is done in this way
because with Java you need to make heaps of files (e.g. one for
every public exception), but maybe it has a good reason?
We use some very old and surely bad custom automake rules to
compile java sources. Ages ago we also had some wildcard (`find')
rules (inspired by ant assuming `this would be the good way to
go'). Those rules collected the files, but we changed them to use
a list of file names, which seemed much cleaner and solved some
issues (which of course could had been solved in other ways,
too).
One motivation for this change was that our make rules that
generated the exception source code files (many differing more or
less just in the name and some `String name ...') and people
forgot to add the files, so if existing for them they had been
built but not for others where the files were not updated (and
the new gensrc files were missing). This resulted in an
incomplete jar file and unfortunately when using this jar a
compilation error was flagged out in the wrong package... unit
tests did not helped because of course also missing in the
incomplete jars (and the make check scheme we used also used a
wildcard approach to collect the unit tests to be executed).
Strange things happend when switching between branches (where
typically the number and kind of exceptions changed etc).
I disliked that when by some problem almost all sources would get
lost in a sandbox (e.g. if switching to a bad / incomplete tag),
make (and even make check!) succeeded.
On `tree conflicts' (one changed a file, another moved it) it
could even happen to have two times the same functionality in a
jar...
To build a list of files why not open Makefile.am in $EDITOR,
like vim or emacs, and insert the file list here (in vim, you may
start a line `java_source = \' and on the next blank line use
`!!find . -name *.java -exec echo {} \\ ;'
which works except for the last file, which ends with a `\'.). No
need to make this at make run time, or is there any?
By this, cvs diff (or whichever SCM tool is used) easily shows
the included files which is easier to review I think.
Using wildcards IMHO means to logically `include' the directory
contents to the Makefile. I think you cannot even use it as
dependency (and thus I'd guess the jars would be resigned on each
make run, even if no file changed?).
What is the advantage of using find magic and make time?
How do you handle your java.in files?
How are you safe to get old generated files out the jar (if
removed from gensrc, they still are in builddir and the make
clean rule may not even take it any longer - how to notice?).
I'm afraid the find/wildcard approach only works for simple
builds - but those could also be done by ant I think...
I'm not stealing this concept entirely from ant.
(yeah strange tool that, but different topic :-))
oki,
Steffen
Re: Regarding the JAVA primary
Hi Steffen,
On 4/19/2010 1:22 PM, Steffen Dettmer wrote:
On Mon, Apr 19, 2010 at 8:25 PM, John Calcotejohn.calc...@gmail.com wrote:
[...]
Builds in the Java world generally specify source files found
within a subtree using a globbing mechanism, with optionally
specified inclusions and exclusions.
Yes, they do. BTW, does anyone know why?
With some sarcasm someone could tell that it is done in this way
because with Java you need to make heaps of files (e.g. one for
every public exception), but maybe it has a good reason?
We use some very old and surely bad custom automake rules to
compile java sources. Ages ago we also had some wildcard (`find')
rules (inspired by ant assuming `this would be the good way to
go'). Those rules collected the files, but we changed them to use
a list of file names, which seemed much cleaner and solved some
issues (which of course could had been solved in other ways,
too).
Actually, the only thing bad about the current JAVA primary make rules
is that the command line length may easily be exceeded with very large
file sets (in fact, with hundreds of files, it surely will be exceeded
on some platforms). But this problem can easily be fixed by dumping all
of the source files into a temporary text file (filename), and then by
using @filename on the javac command line.
One motivation for this change was that our make rules that
generated the exception source code files (many differing more or
less just in the name and some `String name ...') and people
forgot to add the files, so if existing for them they had been
built but not for others where the files were not updated (and
the new gensrc files were missing). This resulted in an
incomplete jar file and unfortunately when using this jar a
compilation error was flagged out in the wrong package... unit
tests did not helped because of course also missing in the
incomplete jars (and the make check scheme we used also used a
wildcard approach to collect the unit tests to be executed).
Strange things happend when switching between branches (where
typically the number and kind of exceptions changed etc).
Yes, Java does promote the use of many source files - not necessarily a
bad thing, as this could be seen to promote modular design, if used
correctly and wisely. I tend to use static inner classes to keep
relevant things together and to reduce the number of source files. The
nice thing about this approach is that it's generally fairly easy to see
where a class should be defined at the outer scope (and thus in its own
source file), or within another class.
I disliked that when by some problem almost all sources would get
lost in a sandbox (e.g. if switching to a bad / incomplete tag),
make (and even make check!) succeeded.
On `tree conflicts' (one changed a file, another moved it) it
could even happen to have two times the same functionality in a
jar...
To build a list of files why not open Makefile.am in $EDITOR,
like vim or emacs, and insert the file list here (in vim, you may
start a line `java_source = \' and on the next blank line use
`!!find . -name *.java -exec echo {} \\ ;'
which works except for the last file, which ends with a `\'.). No
need to make this at make run time, or is there any?
By this, cvs diff (or whichever SCM tool is used) easily shows
the included files which is easier to review I think.
Using wildcards IMHO means to logically `include' the directory
contents to the Makefile. I think you cannot even use it as
dependency (and thus I'd guess the jars would be resigned on each
make run, even if no file changed?).
What is the advantage of using find magic and make time?
How do you handle your java.in files?
How are you safe to get old generated files out the jar (if
removed from gensrc, they still are in builddir and the make
clean rule may not even take it any longer - how to notice?).
I'm afraid the find/wildcard approach only works for simple
builds - but those could also be done by ant I think...
All very good points - and all issues that Automake has been designed to
overcome. Many folks don't like the requirement that all source files
must be specified statically in Makefile.am. I'm not one of these
people. I tend to agree with you. For Java, I think it's a matter of
developer convenience that sources in a tree are specified with a
globbing pattern. It's not just that there are a lot of files, but also
that they tend to be spread around the source tree because they belong
to various packages that are defined by their location within the source
tree.
I can certainly see how we may want to stick with the Automake static
source file specification rules for the reasons you point out. In this
case, it becomes more of an evangelistic documentation issue. :) That
is, we might be wise to add a chapter to the Automake manual describing
the value that comes from Automake's position in this matter. Heaven
knows, we've answered the question on the
Re: Regarding the JAVA primary
* John Calcote wrote on Mon, Apr 19, 2010 at 09:52:51PM CEST: I can certainly see how we may want to stick with the Automake static source file specification rules for the reasons you point out. In this case, it becomes more of an evangelistic documentation issue. :) That is, we might be wise to add a chapter to the Automake manual describing the value that comes from Automake's position in this matter. Heaven knows, we've answered the question on the mailing list more than enough times! There already exists a FAQ entry on this. If it's not complete, patches are welcome to fix that. Thanks, Ralf
