On 15/05/13 22:39, Stefano Lattarini wrote: > On 05/15/2013 01:52 PM, Michael Zucchi wrote: >> On 14/05/13 03:47, Stefano Lattarini wrote: >> >>> Instead, let's start implementing something *correct*, in line with >>> the Java philosophy, and with a clean API. We'll think about enhancing >>> it when (and if!) the need arise.
And then most of the objections are from trying the java way. Oh well. >> Seems the way to go. >> >> On that, here are a few more thoughts on java's specific way of doing >> things ... >> >> files vs dirs >> -------------- >> >> java projects/build systems tend to have a fairly basic mechanism to >> define the source tree(s) which would be nice to emulate, but may not >> fit well with the portability requirements of automake. >> >> They simply define the root directory and let the build system do the rest. >> >> Not necessary but would be nice. Or put another way it would basically >> suck if it required every source file to be listed explicitly and auto* >> having to be re-run every time one is added. >> > But then, how could make handle the case of generated java files? I.e., > if half of your files are generated by (say) an awk+sed incantation, > make has to know it must build them before trying to call the java compiler. > But see below. Well if that's a requirement, then it just has to be added right? It's not impossible, it's only software ... Either explicitly listed separately or handled automatically, it's quite easy to tell a file from a directory. >> In a GNU makefile I can >> just use $(shell find) (or other fancier stuff), but for automake? >> > The solution is simpler than it seems at first sight: we should just Actually it isn't quite that simple. I did consider this. More below about resources. > For an example of how this approach could work, take a look at the > EXTRA_DIST support (and to the test 't/extra-dist-wildcards-gnu.sh'). Although ... one can specify directories to EXTRA_DIST (even if not recommended). EXTA_DIST is defined relative to the makefile it is in, but that isn't good enough. It works fine for a root makefile since well, its' generating the source tree and that is by definition relative to the root makefile. For jars it needs to be relative to a another location which is relative to the makefile. The bit about nobase in the manual has an impractical solution of listing every directory separately. Somehow the makefile needs this information, either by listing it or by it's location, it cannot determine it automatically. More detail and an example well below. >> makefile location >> ----------------- >> >> C projects often have a make file per directory, java has the build >> script at the top-level at least, outside of the source trees. >> >> This is desirable. >> > In some ways, this is desirable also for C projects actually; see the > article "Recursive Make Considered Harmful" from Peter Miller: > > <http://miller.emu.id.au/pmiller/books/rmch/> Yeah well that's just an opinion. I don't like it myself because it makes emacs a total pain to use and I personally consider that pretty important. >> For the resource files the Makefile needs to resolve the resource root, >> and strip it out of the names used for the dependencies. So either it >> needs a (set of) explicit resource_root, resource_root_files variables, >> or it just needs to be defined as the root dir and handled either in the >> makefile fragment or in automake.in. >> > Doing too much pre-processing in automake.in (that is, at Automake time) > would prevent us from being able to grasp GNU make specific stuff. So > let's try to keep such pre-processing at a minimum. I'm only talking about iterating a list of strings and printing them in various ways, i'm pretty sure it does that kind of thing already. It wont be running find or walking directory trees since that doesn't provide anything useful at 'make' time. >> javadoc, jni, check/test, etc >> ------------------------ >> >> Some built-in support would be good. javadoc is easy, check/test should >> run junit tests I guess although i'm not too familiar with junit, >> > Or we might leave that to the use from now (they can use the 'check-local' > target), and maybe implement it in a later Automake version, once an usage > pattern has emerged (and most importantly, *if* there actually is any > request in that direction). javadoc+test are pretty important in the java world, and of course documentation is a critical component for any free software. jni is just a personal need and I am familiar with how it works. I also bring up jni because this is one area where java build tools really don't stack up at all and automake some non-zero chance of becoming a primary contender. I'm afraid based on the state of things that 'waiting for a request' will be a rather long wait ... the horse has quite bolted on the 'build tool' front in the last few years for all languages, and one can only surmise it was due to the complexity/inadequacy of the ones we already had. It's hard to argue that auto* wasn't part of the reason many of these tools exist in the first place (i doubt anyone remember imake). Pity all the new ones tend to suck in their own ways not least of which being they all still suffer from the learning curve issue. >> first cut >> --------- >> >> I've attached a pretty nasty bit of unportable makefile which attempts >> some of the above to see how it might be done. >> > Comments for that are below. I understand this is just a preliminary and > rough implementation, so please don't allow my doubts and criticism blow > your morale. Not sure why you bothered pointing out the gnu make stuff every time when i stated that was a shortcoming for inclusion in automake. > BTW, while writing this first draft, have taken into consideration all > the comments in <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9088>, > as well as the extensive and (AFAICS) *working* code at > <https://github.com/stumbleupon/opentsdb/blob/6059488/Makefile#L207>? > We cannot allow all those earlier thoughts and efforts go wasted ... Yeah I looked at both of those, and the java.am stuff. >> The build steps from this are simple: >> >> $ rm foo_jar.stamp ; make -f Auto.mak >> javac -cp lib/jjmpeg.jar:lib/test.jar -sourcepath src -d >> build/classes/foo_jar src/z/util/d.java src/z/util/c.java src/z/a.java >> src/z/b.java >> touch foo_jar.stamp >> jar cf foo.jar -C build/classes/foo_jar . \ >> -C ./src ./z/util/util-names.txt \ >> -C ./res ./foo.txt >> $ >> >> Regards, >> Michael > >> ===== Auto.mak ===== >> >> # default is /usr/share/java/package ? >> jardir = $(datadir)/java/@PACKAGE@ >> > Or simply '$(datadir)/java'; then we can pre-define pkgjardir > to be '$(datadir)/java/$(PACKAGE)'. Something similar is > already done for datadir, includedir, libdir, and libexecdir: > <http://www.gnu.org/software/automake/manual/automake.html#index-pkglibdir-131> I don't really know on this one. I'm just looking at where java stuff is installed on my fedora system - although i never use any of that because it's easier/safer (configuration wise) to just include all the jars in my own packages particularly when supporting cross-platform applications. I guess any jar installed into /usr/share/java would want a version in it's name. >> srcdir=. >> > Just a reminder: we must also be prepared to handle VPATH builds, that > is, '$(srcdir)' != '.'; no need to think about this right away, but it > should be made to work before the feature is integrated in Automake (so > we want a design that will allow it to work!) srcdir is just set so that i have a working standalone makefile but can still follow some automake conventions inside of it to demonstrate the principle. I was going to add a comment to that effect but thought it might've been a little ... obvious. I thought i stated that I think it should support a vpath build, but I haven't yet tested that. So yes, I have already thought about it and it's something i'm quite well aware of. >> all: foo.jar >> >> jar_JARS=foo.jar >> >> foo_jar_SOURCES = src >> > NAK. We should make the use list all the Java files, while (as said > before) being able to support GNU make extensions in the definition > of this variable. Unless you have strong objections to this idea; > objections which I'd like to hear ASAP. Well it was just an alternative. This is the alternative I would like, and I'm pretty sure every normal java developer would expect. If they have to manually list every source file every time a new one is added and then re-run autostuff, they wouldn't bother ever considering automake as it would just totally suck. So, totally sucking is my only real objection. My initial version listed every file, and used the gnu-make specific extensions in order to make it not "totally suck". But then I ran into problems with how to handle resources and rather than work on solving those in a way which could fit into an automake.am fragment, I thought i'd try the approach I had desired from the start. I have some more thoughts lower, and in general they work either way. >> # same as in the way one would add a in-package libtool lib, but >> # this is in the build jar path >> foo_jar_LIBADD = lib/jjmpeg.jar lib/test.jar >> > Sounds good. But maybe s/LIBADD/JARADD/? Could be anything. I followed the way libtool does it. Previous comments in the bug seemed to suggest the desirability of sticking with existing names, so it wasn't a totally arbitrary choice. > In addition, "mkdir -p" is unfortunately not multithred-safe on all platforms. > We have code that should work around the issue; to take advantage of that, > just use the $(MKDIR_P) macro instead of "mkdir -p". ok no worries. >> javac -cp $(foo_jar_CLASSPATH) -sourcepath src -d build/classes/foo_jar >> $(filter %.java,$^) >> > We cannot unilaterally squat on a common file/directry name like "build" > here. No need to fix that right now though; we can easily return on the > issue later. This is just what ant uses, and not surprisingly, it could be anything. As a developer I would prefer it wasn't hidden but it's kind of here nor there. No java tools I know ever hide it, it's a critical path in the build process and one can just run code from there, ignoring any jar file. I think a documented/makefile usable location will be needed for tools that work on the class files, but I don't have familarity with those. > Also, the '$(filter %.java,$^)' is GNU make specific; it should be juse > substituted by $(foo_jar_SOURCES), no? Yes I know I was using gnu make extensions. Well, no, $(foo_jar_SOURCES) can't be used, at least not directly. It doesn't expand VPATH as the dependencies do but it could (obviously!) just use a similar approach as in the current java.am file. Although that also uses the expanded dependencies so it would still need to filter out .java files. This particular target was all very simple until i added the LIBADD dependencies which meant i could no longer just use $^ as i had. I couldn't think of a way of having $^ just list .java files whilst still depending on the LIBADD stuff for running the compiler but if that was possible the problem would vanish. >> touch $@ >> > >> foo.jar: foo_jar.stamp $(patsubst %,src/%,$(foo_jar_RESOURCES_list0)) >> $(patsubst %,res/%,$(foo_jar_RESOURCES_list1)) >> jar cf $@ -C build/classes/foo_jar . \ >> $(patsubst %,-C $(srcdir)/src %,$(foo_jar_RESOURCES_list0)) \ >> $(patsubst %,-C $(srcdir)/res %,$(foo_jar_RESOURCES_list1)) >> > This is too much GNU make specific. Moreover, I have zero knowledge about > how the "Java resources" are expected to work, so I don't know how to > suggest fixes or improvements. Do you have links to noob-understendable > documentation that explains in enough detail what Java resources are and > how they are suppose to work? Yes I know I was using gnu make extensions. As previously explained they're typically just every file in the source tree which doesn't end in '.java' but they could be from (mutiple) other trees. I never read any documentation about this I just picked it up osmoticly[sic] from using netbeans. Unlike other languages, java files really must end in '.java', so the distinction is easy. They can be anything that is app-specific but there are also others which are to support java meta-data such as service plugin hooks. But at the end of the day they're just files in a directory tree. Obviously this means that a jar may also be purely data with no code whatsoever. Newbie: It's a zip file of a directory tree of files. It can contain anything a zip can. It is definitely a hard requirement that they be able to be placed into the source tree in addition to other locations. And this is why ... e.g. a source tree: a/a.java a/a.txt jar: a/a.class a/a.txt a.java fragment: InputStream a.class.getResourceAsStream("a.txt"); i.e. a class has direct access to these resources relative to it's own class. They are just loaded from the zip at runtime. The problem is this: a) the jar needs to depend on the resources, they need to be defined relative to the makefile b) to include the files in the jar command, they need to be specified as they appear in the jar file, i.e. relative to the jar root. c) with no a-priori knowledge of the root directory in which the resources reside, how can these two independent file lists be created? And support multiple directory trees? This problem does not arise as such with the building of the java source because the compiler is already doing the translation for you, and the individual .class files are not tracked by make. However, the problem does arise with extra class files which might be included/created somewhere else. c is not an issue with EXTRA_DIST because the root directory is known a-priori due to it being where the Makefile is. Basically say I have this: foo_jar_RESOURCES=src/a/a.txt src/b/b.txt res/c/c.txt I need two results from this input (taking into account out of source builds with the abstract use of $(srcdir) here, regardless of whether this is the correct automake foo or not): 1) dependencies: $(srcdir)/src/a/a.txt $(srcdir)/src/a/a.txt $(srcdir)/res/c/c.txt - this is just handled by VPATH already and so happens automagically 'for free'. 2) relative paths (and their corresponding root): root $(srcdir)/src files a/a.txt b/b.txt root $(srcdir)/res files c/c.txt Actually the end result I need explicitly for jar is this: -C $(srcdir)/src a/a.txt \ -C $(srcdir)/src b/b.txt \ -C $(srcdir)/res c/c.txt However this can be in a file rather than a command argument (and doesn't need the newline escapes then). Ideally jar could be used directly for this, alternatively a staging area could be used but this would slow the build down/waste disk space. Possible solutions (not exhaustive, not suggestions): a) As included in the makefile so far. Supply the roots and let the script handle everything. It has enough information to do everything required. This is how every java coder on the planet will want and expect it to work. b) specify each resource root using multiple variables. Possible non-suggestions: b.0) foo_bar_RESOURCES = src res ../android/res foo_bar_src_RESOURCES = a/a.txt b/b.txt foo_bar_res_RESOURCES = c/c.txt foo_bar____android_res_RESOURCES = pics/bob.jpg (obvious problems with relative paths with this idea) b.1) foo_bar_RESOURCES_anything = src foo_bar_RESOURCES_anything_FILES = a/a.txt b/b.txt foo_bar_RESOURCES_blah = res foo_bar_RESOURCES_blah_FILES = c/c.txt etc. b.2) foo_bar_RESOURCESDIRS = src res foo_bar_RESOURCES = src/a/a.txt src/b/b.txt root/c/c.txt (use regex/sed to strip either root: it can be safely assumed they are independent trees so this should be easy) b.3) Something more implicit like the way nobase_ works, but i don't know how you'd define the root in that case. There may be automake.in conventions which make these or other arrangements more desirable. But whatever it is this basic information is required in the end: a root and a list relative to that root. c) Define everything twice. The 'jar' name (plus root, somehow), and the 'dependency name'. d) force individual makefiles into the 'src' or 'res' roots. Then require everything to be copied into a staging area. Then use another makefile to create the actual jar. e) List every source dir and target dir by hand (example given in the nobase_ part of the manual S7.3 - node 'Alternative'). a) nicest to user. b) bearable. c) sucks, messy d) slow, messy, painful to use. e) ugh. Whatever it is the same mechanism should probably be used for multiple source trees and pre-existing/make-generated class trees. Actually just using SOURCES alone would work for everything (i.e. no _RESOURCES stuff, but it still needs to know the relative root), and just filter out '.java' for the resources/extra classes. This is assuming one never wanted a .java source file in such a generated jar, which I think is a fairly reasonable assumption as these are about code not source. > >> .PHONY javadoc: javadoc-foo_jar >> javadoc-foo_jar: >> exit 1 >> build javadoc stuff, not done yet >> > > Is see that we still lack clean and installation rules. They are quite > important, and should be introduced ASAP. These are just trivialities. I thought i'd get the hard problems out of the way before giving you a final solution you wouldn't accept anyway - i know how this works. I was worried the fragment presented wasn't complete enough but now i'm glad I didn't spend more time on it before feedback. You seem to have got quite hung up on the gnu make stuff even though i stated that wasn't a consideration at this point. Obviously anything it does can be implemented using sed or grep, and java tools allow arguments to be placed into files for better scalability. Questions then arise: what of such basic tools can be used, and what limitations on filenames must be adhered to should they be used for argument lists (if this is in hacking just leave it at that). Even if every file was listed by hand these capabilities are still going to be required. I notice that automake tends to just use `` instead: aren't command line limits a portability issue anymore, or is it just not likely a problem with c source? !Z