Re: bug#9088: the jar fragment
Forgot the attachment.
# 'simulate' automake environment
jardir = $(datadir)/java
srcdir=../maktest
MKDIR_P=mkdir -p
VPATH=.:$(srcdir)
all: foo.jar
##
## The makefile.am `equivalent' stuff
##
## This user has chosen to make it gnu-make only.
##
jar_JARS=foo.jar
# sources, all relative to '.'
foo_jar_SOURCES = $(shell cd $(srcdir) ; find src -name '*.java')
# 'generated' sources
foo_jar_SOURCES += stuff.java
# resources all relative to '.'
foo_jar_RESOURCES_PREFIX = src res
foo_jar_RESOURCES = $(shell cd $(srcdir) ; find src res -name '*.txt' -o -name
'*.jpg' )
# 'generated' resources
foo_jar_RESOURCES += a.txt b.txt c.txt
foo_jar_MANIFEST = manifest
foo_jar_MAIN = au.notzed.stuff.Main
# build classpath
foo_jar_LIBADD = lib/jjmpeg.jar lib/test.jar
# "custom" targets
a.txt b.txt c.txt:
touch $@
stuff.java:
echo "class stuff { }" > stuff.java
##
## the jars.am fragment for foo.jar
##
# windows needs ;, of course ...
# automake.in generates this from LIBADD?
foo_jar_CLASSPATH=lib/jjmpeg.jar:lib/test.jar
foo_jar.stamp: $(foo_jar_SOURCES) $(foo_jar_LIBADD)
rm -rf build/classes/foo_jar
$(MKDIR_P) build/classes/foo_jar
clist=""; \
slist='$(foo_jar_SOURCES)' ; \
for n in $$slist ; do \
if test -f $$n ; then \
clist="$$clist $$n" ; \
else \
clist="$$clist $(srcdir)/$$n" ; \
fi ; \
done; \
echo compiling: $$clist; \
javac -cp $(foo_jar_CLASSPATH) -d build/classes/foo_jar $$clist
touch $@
# the complexity here is due to
# a) having to walk the 'VPATH' manually
# b) having to remap the relative roots of resources to the jar file root
# c) portable shell & tools
foo.jar: foo_jar.stamp $(foo_jar_RESOURCES) $(foo_jar_MANIFEST)
rlist='$(foo_jar_RESOURCES)'; \
clist=""; \
for n in $$rlist ; do \
found=0; \
for p in $(foo_jar_RESOURCES_PREFIX) ; do \
t=`echo $$n | sed "s@^$$p/@@"` ; \
if test $$t != $$n ; then \
if test -f $$n ; then \
clist="$$clist -C $$p $$t" ; \
else \
clist="$$clist -C $(srcdir)/$$p $$t" ; \
fi ;\
found=1 ; \
break ; \
fi ; \
done ; \
if test $$found -eq 0 ; then \
if test -f $$n ; then \
clist="$$clist $$n" ; \
else \
clist="$$clist $(srcdir)/$$n" ; \
fi ;\
fi ; \
done ; \
flags="cf" ; \
files="$@" ; \
if test -n "$(foo_jar_MANIFEST)" ; then \
flags="$${flags}m"; \
if test -f "$(foo_jar_MANIFEST)" ; then \
files="$$files $(foo_jar_MANIFEST)" ; \
else \
files="$$files $(srcdir)/$(foo_jar_MANIFEST)" ; \
fi; \
fi ; \
if test -n "$(foo_jar_MAIN)" ; then \
flags="$${flags}e" ;\
files="$$files $(foo_jar_MAIN)" ; \
fi; \
echo "jar $$flags $$files $$clist" ; \
jar $$flags $$files -C build/classes/foo_jar . $$clist || ( rv=$$?; rm
-f $@ ; exit $$rv )
Re: bug#9088: the jar fragment
Hi,
This is my current work on the '.jar' fragment. The rest is just there
to create a working makefile, but the focus of this post is solely on
the 'foo.jar' target fragment.
In-so-far as jar generation goes, this should be 'portable and feature
complete'.
1) Out of source building.
2) In-source tree/generated files
3) 'alternative-root relative' and 'makefile relative' files
4) optional in-source/generated manifest
5) optional main entry point
I have not used (to my knowledge) any gnu-makeisms in the
automake-template-part, and also aimed for portable sh and sed usage.
It should include all the dependencies it needs (at this point).
I presume 'spaces in file names' just isn't supported in general.
(the automake naming conventions and things like the 'build' dir are not
yet done, but are trivial to alter, I need to review all of your
provided information some more first).
It turned out that merging resources/sources wasn't a good idea: it
added complexity and lead to poor dependency behaviour (e.g. add a text
file - compile every jar), so I split them out again. And despite
suggesting it, I actually don't see any need for the SOURCES_PREFIX at
this point.
Because of the way the jar command-line works I decided to add direct
support for the manifest and 'main method' (entry point). I think this
works out cleaner/simpler (and is easier to use) than trying to allow
the user to supply a custom jar command line. _JARFLAGS could still be
used for a couple of aesoteric options.
Background/justification:
Generating a simple jar file is like tar:
jar cf foo.jar files ...
Adding a manifest requires strict ordering:
jar cfm foo.jar manifest.txt files ...
jar cmf manifest.txt foo.jar files ...
Same with also adding a main class entry point:
jar cfme foo.jar manifest.txt au.notzed.Main files ...
But this doesn't work:
jar c -f foo.jar -e au.notzed.Main -m manifest.txt files ...
i.e. the last example could be easily parametrised and allow a
'_JARFLAGS' variable to include these things, but as that isn't the way
jar works, it can't.
The shell script isn't very complicated, it's just bulky. I fully
indented it for clarity. The vpath walk is required several times, and
it has to look-up prefixes on every file. I basically followed how the
java.am did it.
The prefix handling stuff is about one of about 10 different solutions I
tried, everything from awk to sed to plain shell, case statements, etc.
None were any much smaller than the others and this one should be one
of the most portable, and unlike some it requires no automake.in setup.
I have another version which doesn't require sed for the prefix
handling, but i'm not sure how portable the substring/length stuff is (I
spent quite a while searching to try and find out, but search engines
aren't very useful for looking this technical stuff up any more).
It just needs the length and sub-string functions: prefix=${n:0:${#r}};
relname=${n:${#r}}
If those are ok to use it might be a bit more efficient/readable, not
needing to run sed.
!Z
Re: bug#9088: Java, JARS primary?
On 16/05/13 19:39, Stefano Lattarini wrote: > On 05/16/2013 05:57 AM, Michael Zucchi wrote: >> On 15/05/13 22:39, Stefano Lattarini wrote: >>> On 05/15/2013 01:52 PM, Michael Zucchi wrote: >>>> On 14/05/13 03:47, Stefano Lattarini wrote: >>>> >> Well if that's a requirement, then it just has to be added right? It's >> not impossible, it's only software ... >> > I failed to parse this. Can you rephrase? It means the problem is an economic one, not a technical one. It's a mostly facetious saying coined by a fellow work-mate which basically means 'we can do whatever you want (but it will cost you)'. Or broken down logically, the technical problem: a) We are solving a problem. b) We are using a general purpose computer to solve it. c) Any problem can be solved by a general purpose computer. d) Therefore, we can solve any problem. (where of course, problem is a computer related problem, which this clearly is) Here the cost is time that could be spent elsewhere, and maybe some lost/grey hairs. >>> Doing too much pre-processing in automake.in (that is, at Automake time) >>> would prevent us from being able to grasp GNU make specific stuff. So >>> let's try to keep such pre-processing at a minimum. >> >> I'm only talking about iterating a list of strings and printing them in >> various ways, i'm pretty sure it does that kind of thing already. >> > Yes, but the price is reduce flexibility at Automake time. Which might > be irrelevant or very important, depending on the expected usage patterns > of a feature. I'm lost here. You seem to be talking about something I haven't even written yet. > OK, then we need integration with javado and juinit. But first let's > try to get the jar stuff right (that seems already non-trivial). Only in-so-much as it isn't identical to a libtool library. And even then libtool gets a leg-up by putting all the messy stuff into another tool. A 'javatool' would likewise make most of the makefile trivial. I'm not proposing such a tool, useful as it might be. >> Not sure why you bothered pointing out the gnu make stuff every time >> when i stated that was a shortcoming for inclusion in automake. >> > Because it is not obvious to me how this usages will be later converted > to be portable. Ahh, I figured you'd be a portable makefile guru working on something like automake and could help work it out. I think with 'b.2' I can do away with the trickier stuff. In 20 odd years i've only ever used gnu make because it can be built anywhere by design. Well apart from maybe some sunos but I think gcc/make were the first tools we installed there anyway. And one windows 7 sdk makefile, but that doesn't even pretend to be a real make. >> My initial version listed every file, and used the gnu-make specific >> extensions in order to make it not "totally suck". But then I ran into >> problems with how to handle resources and rather than work on solving >> those in a way which could fit into an automake.am fragment, I thought >> i'd try the approach I had desired from the start. >> > Given my ignorance of Java, I'm fully ready to change my mind on > this; but then I need examples that are much more clear, complete, > self-explanatory and *working*. I think you should start perusing > the java tests in the 't/' subdirectory of the Automake source tree, > and start writing similar tests that show how you envision your > feature to work in different scenarios. At this stage, using GNU > make specific constructs is OK; I promise I won't bother you about > them anymore. Well the makefile works as much as it was designed to. I don't really understand how the unit tests work. It's a bit frustrating trying to make you enough of a java build expert in a handful of emails when i'm not one myself. Perhaps it's just a lot simpler than you expect - it has far fewer complexities than cross-platform c libraries. >> This particular target was all very simple until i added the LIBADD >> dependencies which meant i could no longer just use $^ as i had. >> >> I couldn't think of a way of having $^ just list .java files whilst >> still depending on the LIBADD stuff for running the compiler but if that >> was possible the problem would vanish. >> > Consider that $^ is not portable to all non-GNU makes, IIRC (but I'm not > 100% sure of this, I'll have to double check), so that we'd likely still > need to rewrite such idioms when we make the transition from your > prototype from the final implementation. Not a big deal right now, but > something worth kee
Re: bug#9088: Java, JARS primary?
On 15/05/13 22:39, Stefano Lattarini wrote: > On 05/15/2013 01:52 PM, Michael Zucchi wrote: >> On 14/05/13 03:47, Stefano Lattarini wrote: >> >>> Instead, let's start implementing something *correct*, in line with >>> the Java philosophy, and with a clean API. We'll think about enhancing >>> it when (and if!) the need arise. And then most of the objections are from trying the java way. Oh well. >> Seems the way to go. >> >> On that, here are a few more thoughts on java's specific way of doing >> things ... >> >> files vs dirs >> -- >> >> java projects/build systems tend to have a fairly basic mechanism to >> define the source tree(s) which would be nice to emulate, but may not >> fit well with the portability requirements of automake. >> >> They simply define the root directory and let the build system do the rest. >> >> Not necessary but would be nice. Or put another way it would basically >> suck if it required every source file to be listed explicitly and auto* >> having to be re-run every time one is added. >> > But then, how could make handle the case of generated java files? I.e., > if half of your files are generated by (say) an awk+sed incantation, > make has to know it must build them before trying to call the java compiler. > But see below. Well if that's a requirement, then it just has to be added right? It's not impossible, it's only software ... Either explicitly listed separately or handled automatically, it's quite easy to tell a file from a directory. >> In a GNU makefile I can >> just use $(shell find) (or other fancier stuff), but for automake? >> > The solution is simpler than it seems at first sight: we should just Actually it isn't quite that simple. I did consider this. More below about resources. > For an example of how this approach could work, take a look at the > EXTRA_DIST support (and to the test 't/extra-dist-wildcards-gnu.sh'). Although ... one can specify directories to EXTRA_DIST (even if not recommended). EXTA_DIST is defined relative to the makefile it is in, but that isn't good enough. It works fine for a root makefile since well, its' generating the source tree and that is by definition relative to the root makefile. For jars it needs to be relative to a another location which is relative to the makefile. The bit about nobase in the manual has an impractical solution of listing every directory separately. Somehow the makefile needs this information, either by listing it or by it's location, it cannot determine it automatically. More detail and an example well below. >> makefile location >> - >> >> C projects often have a make file per directory, java has the build >> script at the top-level at least, outside of the source trees. >> >> This is desirable. >> > In some ways, this is desirable also for C projects actually; see the > article "Recursive Make Considered Harmful" from Peter Miller: > > <http://miller.emu.id.au/pmiller/books/rmch/> Yeah well that's just an opinion. I don't like it myself because it makes emacs a total pain to use and I personally consider that pretty important. >> For the resource files the Makefile needs to resolve the resource root, >> and strip it out of the names used for the dependencies. So either it >> needs a (set of) explicit resource_root, resource_root_files variables, >> or it just needs to be defined as the root dir and handled either in the >> makefile fragment or in automake.in. >> > Doing too much pre-processing in automake.in (that is, at Automake time) > would prevent us from being able to grasp GNU make specific stuff. So > let's try to keep such pre-processing at a minimum. I'm only talking about iterating a list of strings and printing them in various ways, i'm pretty sure it does that kind of thing already. It wont be running find or walking directory trees since that doesn't provide anything useful at 'make' time. >> javadoc, jni, check/test, etc >> >> >> Some built-in support would be good. javadoc is easy, check/test should >> run junit tests I guess although i'm not too familiar with junit, >> > Or we might leave that to the use from now (they can use the 'check-local' > target), and maybe implement it in a later Automake version, once an usage > pattern has emerged (and most importantly, *if* there actually is any > request in that direction). javadoc+test are pretty important in the java world, and of course documentation is a critical component for any free
Re: bug#9088: Java, JARS primary?
On 14/05/13 03:47, Stefano Lattarini wrote: > Instead, let's start implementing something *correct*, in line with > the Java philosophy, and with a clean API. We'll think about enhancing > it when (and if!) the need arise. Seems the way to go. On that, here are a few more thoughts on java's specific way of doing things ... files vs dirs -- java projects/build systems tend to have a fairly basic mechanism to define the source tree(s) which would be nice to emulate, but may not fit well with the portability requirements of automake. They simply define the root directory and let the build system do the rest. Not necessary but would be nice. Or put another way it would basically suck if it required every source file to be listed explicitly and auto* having to be re-run every time one is added. In a GNU makefile I can just use $(shell find) (or other fancier stuff), but for automake? source vs data -- A source tree can also include non-source data which is copied into the jar file verbatim at the same relative path location. At least this is how ant does it and given the way these resources are resolved at runtime makes the most sense. Again I think this needs to be specified by root directory not file. It could just hang off the _SOURCES or use _RESOURCES. makefile location - C projects often have a make file per directory, java has the build script at the top-level at least, outside of the source trees. This is desirable. For the compiled code this is simple enough since the makefile will compile every file every time and the compiler will dump the class files where it's told. For the resource files the Makefile needs to resolve the resource root, and strip it out of the names used for the dependencies. So either it needs a (set of) explicit resource_root, resource_root_files variables, or it just needs to be defined as the root dir and handled either in the makefile fragment or in automake.in. annotation processors - I think these can be handled by using _JAVACFLAGS or something similar. They can generate source for new classes at compile time, but this is all handled by javac in separate locations so make doesn't need to know about it. javadoc, jni, check/test, etc Some built-in support would be good. javadoc is easy, check/test should run junit tests I guess although i'm not too familiar with junit, jni will need some thought but should be limited in scope (support for javah basically, maybe not enough need to include it in automake so long as the dev can depend on the jar). first cut - I've attached a pretty nasty bit of unportable makefile which attempts some of the above to see how it might be done. Even with gnu make stuff it still needs some support from automake.in to handle multiple resource directories, and it doesn't handle everything quite properly. The build steps from this are simple: $ rm foo_jar.stamp ; make -f Auto.mak javac -cp lib/jjmpeg.jar:lib/test.jar -sourcepath src -d build/classes/foo_jar src/z/util/d.java src/z/util/c.java src/z/a.java src/z/b.java touch foo_jar.stamp jar cf foo.jar -C build/classes/foo_jar . \ -C ./src ./z/util/util-names.txt \ -C ./res ./foo.txt $ Regards, Michael # default is /usr/share/java/package ? jardir = $(datadir)/java/@PACKAGE@ srcdir=. all: foo.jar jar_JARS=foo.jar foo_jar_SOURCES = src # multiple resources dirs foo_jar_RESOURCES = src res # same as in the way one would add a in-package libtool lib, but this is in the build jar path foo_jar_LIBADD = lib/jjmpeg.jar lib/test.jar ## foo.jar # jar.am template, could handle multi-valued foo_jar_SOURCES foo_jar_SOURCES_list=$(shell cd $(srcdir) ; find $(foo_jar_SOURCES) -name '*.java') # automake.in generated, the find exclusions are not nearly enough to handle version control foo_jar_RESOURCES_list0=$(shell cd $(srcdir)/src; find . -type f -a \! -name '*.java' \! -name '*~') foo_jar_RESOURCES_list1=$(shell cd $(srcdir)/res; find . -type f -a \! -name '*.java' \! -name '*~') # windows needs ;, of course ... # automake.in generates this from LIBADD? foo_jar_CLASSPATH=lib/jjmpeg.jar:lib/test.jar foo_jar.stamp: $(foo_jar_SOURCES_list) $(foo_jar_LIBADD) @-rm -rf build/classes/foo_jar @-mkdir -p build/classes/foo_jar javac -cp $(foo_jar_CLASSPATH) -sourcepath src -d build/classes/foo_jar $(filter %.java,$^) touch $@ foo.jar: foo_jar.stamp $(patsubst %,src/%,$(foo_jar_RESOURCES_list0)) $(patsubst %,res/%,$(foo_jar_RESOURCES_list1)) jar cf $@ -C build/classes/foo_jar . \ $(patsubst %,-C $(srcdir)/src %,$(foo_jar_RESOURCES_list0)) \ $(patsubst %,-C $(srcdir)/res %,$(foo_jar_RESOURCES_list1)) .PHONY javadoc: javadoc-foo_jar javadoc-foo_jar: exit 1 build javadoc stuff, not done yet
Re: bug#9088: Java, JARS primary?
On 13/05/13 02:28, Stefano Lattarini wrote: > On 05/12/2013 06:29 AM, Michael Zucchi wrote: >> Using $? will not suffice as a dependency check, as it's trivially easy >> to create an example which will compile ok after a change but create a >> broken jar. e.g. add a new abstract method to an abstract class but >> forget to fix all sub-classes. >> > I don't really follow here, sorry (likely because I know almost nothing > about Java). Do you mean that, if you have a bunch of .java files that > get compiled into a single jar, and you change just one of these files, > you also need to recompile all the other ones in order not to risk ending > up with a broken and/or inconsistent jar? If it is so, that's awful :-( Well if you were only going on the timestamps of the changed files it would not work. I'm not sure it's really that awful - it's done in C because that's how C is compiled, and compilation is so slow. If it was made fast enough the need for all the considerable complexity of creating and managing dependencies would go away. For a point of reference with a small library i have, a build of 104 java files with 22KLOC takes under 3s on my workstation. Building them "one source file at a time" (see next quoted block) takes about a minute. For big projects code that doesn't change often can be placed in it's own jar negating the need to rebuild that part every time downstream code changes - although one does have to rebuild everything downstream when that jar does. Hmm, now i think about it there is a possible solution to the dependency problem but i'm not sure it's worth it and it has some caveats. The main one being that stale classes could still make it to the jar (so you would have to manually make clean before make install), and it also needs a dependency tool. Because of the strict rules on java package names (they must match the directory they file is in), the import statements could be used to generate some coarse-but-accurate dependencies. As anything in the same package does not need an import statement, one would have to assume every class in the same package is included (i.e. ostensibly the same directory, but it may span multiple trees). Actually it isn't quite that simple - as you can just use fully qualified names without any import statement, so the dependency tool would have to scan the whole file. However, assuming these coarse dependencies could be generated reliably ... the following might work. Lets say the project is something like src/z/a.java imports c src/z/b.java src/z/util/c.java src/z/util/d.java A deps file is created with the cross dependencies for each file. It is simply everything in the same package, and then everything within the project that a given file imports. the deps file: src/z/a.java src/z/b.java src/z/a.java src/z/util/c.java src/z/util/c.java src/z/util/d.java i.e. src/z/*.java, src/z/utill/*.java, then dependencies for any files the Makefile: # this is so plain package names can be used VPATH=src JAVASRC=z/a.java z/b.java z/util/c.java z/util/d.java all: z.jar build.stamp: $(JAVASRC) @-mkdir classes @-rm buildsrc for n in $? ; do grep $$n deps >> buildsrc ; done javac -sourcepath src -d classes @buildsrc touch build.stamp rm buildsrc # just classes - in reality it needs to deal with other files z.jar: build.stamp jar cf $@ -C classes . So it takes the modified list of files and expands it to include any forward requirements. So for example, touch src/z/util/c.java will rebuild a, c, and d - all the classes which might possibly have something to with c. This simple example has some bugs in that it might over-build (e.g. touch a.java will build a, b as required but also c which is not) but that is fixable. Given the caveats, is it worth it? >> So without compiler support for dependency generation I think the only >> practical solution will be to build all files every time. Even the >> sub-directory holding the classes will probably need to be wiped away >> otherwise the jar could contain extraneous classes no longer generated >> from the corresponding source which would probably not be a good thing. > Couldn't we put the *.class files obtained compiling a foo.java file > into a (say) 'foo.d' directory, and remove & rebuild only that directory > whenever foo.java changes? You would have to then invoke the compiler individually on each file, which is at least 1-3 orders of magnitude slower than just compiling every file at the same time. And it wouldn't have the desired result anyway, if it can't find any required .class files it finds the corresponding source and builds them into the target location as well so then build order becomes an issue. I can go into more detail if you like ... One might ask why use make then, but there is more to real projects than a bunch of java source files that need compiling. Michael
Re: Java, JARS primary?
Hi again, I (mostly) just have an observation to add to the bug tracker discussion on the dependency generation. Using $? will not suffice as a dependency check, as it's trivially easy to create an example which will compile ok after a change but create a broken jar. e.g. add a new abstract method to an abstract class but forget to fix all sub-classes. So without compiler support for dependency generation I think the only practical solution will be to build all files every time. Even the sub-directory holding the classes will probably need to be wiped away otherwise the jar could contain extraneous classes no longer generated from the corresponding source which would probably not be a good thing. I have had a bit of a look at automake.in and some of the .am files - it seems to me it would not be any use using the existing in built language code as that is designed for 1:1 source:object compilation. But before I get too bogged down in that I think I will first try to create a simple Makefile with the required features for discussion, and then worry about how to generate it. Most of it should be straightforward apart from deciding on conventions. Regards, Michael
Re: Java, JARS primary?
Hi Stefano, On 10/05/13 22:15, Stefano Lattarini wrote: > So, if you are willing to go ahead, you might want to clone the Automake > git repository, read the HACKING file, and start perusing the files > 'bin/automake.in' and 'lib/am/*.am' for "inspiration". Thanks for all this information. I write perl like I write C, but i've been using it occasionally for nearly 20 years so there's some hope :) I think I will have to have a bit of a poke around before I can contribute anything meaningful to a discussion on the details. To start with i'll just look at the current suggestions on the bug. In the event I can't get anywhere I will also let you and the list know i've given up and not leave it dangling. Cheers, Michael
Java, JARS primary?
Hi list, I recently added a tiny bit of java+jni to an auto* configured project, and during the process came across some discussion from about two years ago about improving Java support. As documented on: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9088 So, it seems the JAVA deprecation was carried out, but not the JARS implementation. I presume that was mainly due to lack of interest together with critical-path personnel constraints? Unfortunately as much as ant (really) sucks, I can't see the Java world changing it's use of horrid tools, and the GNU world still seems to shun Java too - so interest may never pick up. However, assuming it hasn't gone anywhere beyond that bug report, where would one begin were one to try to tackle this? m4 mostly just gives me the willies so i'm not sure I can help much but if I waste my time it's only my time wasted. Cheers, Michael
