bug#9088: Java, JARS primary?
On 05/12/2013 06:29 AM, Michael Zucchi wrote: Hi again, I (mostly) just have an observation to add to the bug tracker discussion on the dependency generation. Using $? will not suffice as a dependency check, as it's trivially easy to create an example which will compile ok after a change but create a broken jar. e.g. add a new abstract method to an abstract class but forget to fix all sub-classes. I don't really follow here, sorry (likely because I know almost nothing about Java). Do you mean that, if you have a bunch of .java files that get compiled into a single jar, and you change just one of these files, you also need to recompile all the other ones in order not to risk ending up with a broken and/or inconsistent jar? If it is so, that's awful :-( So without compiler support for dependency generation I think the only practical solution will be to build all files every time. Even the sub-directory holding the classes will probably need to be wiped away otherwise the jar could contain extraneous classes no longer generated from the corresponding source which would probably not be a good thing. Couldn't we put the *.class files obtained compiling a foo.java file into a (say) 'foo.d' directory, and remove rebuild only that directory whenever foo.java changes? I have had a bit of a look at automake.in and some of the .am files - it seems to me it would not be any use using the existing in built language code as that is designed for 1:1 source:object compilation. Maybe we can steal some code from the existing _JAVA primary though, were that makes sense? But before I get too bogged down in that I think I will first try to create a simple Makefile with the required features for discussion, and then worry about how to generate it. This is the sanest approach, yes. You might also write some tests on the expected behaviours of this Makefile, and we could later re-use them in our testsuite. Most of it should be straightforward apart from deciding on conventions. Regards, Michael Thanks, Stefano
bug#9088: Java, JARS primary?
Stefano Lattarini stefano.lattar...@gmail.com writes: On 05/12/2013 06:29 AM, Michael Zucchi wrote: Hi again, I (mostly) just have an observation to add to the bug tracker discussion on the dependency generation. Using $? will not suffice as a dependency check, as it's trivially easy to create an example which will compile ok after a change but create a broken jar. e.g. add a new abstract method to an abstract class but forget to fix all sub-classes. I don't really follow here, sorry (likely because I know almost nothing about Java). Do you mean that, if you have a bunch of .java files that get compiled into a single jar, and you change just one of these files, you also need to recompile all the other ones in order not to risk ending up with a broken and/or inconsistent jar? If it is so, that's awful :-( Think of it like changing a header file in C to change the definition of a struct. If you don't recompile all of the C source that references that struct, you get broken code. The same thing is true in Java, except that Java doesn't have separate files for interfaces versus implementations (or at least doesn't mandate it; that's a coding style that some people use and other people don't). Java development environments like Eclipse figure out the necessary dependencies and know what *.java files need to be recompiled to pick up changes (and detect errors). If one is using a command-line compiler that can't generate similar sorts of dependency information, it's usually best to just rebuild all the *.java files that make up a JAR if any of them have changed, to ensure that no errors have been introduced and no internal APIs have changed. -- Russ Allbery (r...@stanford.edu) http://www.eyrie.org/~eagle/
Re: install-strip variant that strips then installs?
On Tue, May 7, 2013 at 5:12 PM, Rhys Ulerich rhys.uler...@gmail.com wrote: I gather that 'make install-strip' installs and then strips binaries. Is there some variant that reverses the order? If not, any recommendations for how to write one in an Automake-compliant manner? My unstripped binaries are absurdly large and my installation directory is NFS-mounted. So I get to pay lots of network overhead to install what eventually becomes O(100MB) of binaries because the unstripped copy is O(1.5GB). Thanks, Rhys This seems like a good idea to me. Is there any reason why the order couldn't be reversed? The only problem I can think of is that make install-strip isn't expected to modify the binaries in the build directory, and the user might conceivably be relying on them being unstripped (for some obscure reason). If that could be a problem, perhaps a solution is to have a separate strip rule which could be run. You could try writing a rule yourself in your Makefile.am to strip the binaries. You could use the bin_PROGRAMS make variable that is set in the output Makefile.
Re: bug#9088: Java, JARS primary?
On 05/12/2013 06:29 AM, Michael Zucchi wrote: Hi again, I (mostly) just have an observation to add to the bug tracker discussion on the dependency generation. Using $? will not suffice as a dependency check, as it's trivially easy to create an example which will compile ok after a change but create a broken jar. e.g. add a new abstract method to an abstract class but forget to fix all sub-classes. I don't really follow here, sorry (likely because I know almost nothing about Java). Do you mean that, if you have a bunch of .java files that get compiled into a single jar, and you change just one of these files, you also need to recompile all the other ones in order not to risk ending up with a broken and/or inconsistent jar? If it is so, that's awful :-( So without compiler support for dependency generation I think the only practical solution will be to build all files every time. Even the sub-directory holding the classes will probably need to be wiped away otherwise the jar could contain extraneous classes no longer generated from the corresponding source which would probably not be a good thing. Couldn't we put the *.class files obtained compiling a foo.java file into a (say) 'foo.d' directory, and remove rebuild only that directory whenever foo.java changes? I have had a bit of a look at automake.in and some of the .am files - it seems to me it would not be any use using the existing in built language code as that is designed for 1:1 source:object compilation. Maybe we can steal some code from the existing _JAVA primary though, were that makes sense? But before I get too bogged down in that I think I will first try to create a simple Makefile with the required features for discussion, and then worry about how to generate it. This is the sanest approach, yes. You might also write some tests on the expected behaviours of this Makefile, and we could later re-use them in our testsuite. Most of it should be straightforward apart from deciding on conventions. Regards, Michael Thanks, Stefano
Re: bug#9088: Java, JARS primary?
Stefano Lattarini stefano.lattar...@gmail.com writes: On 05/12/2013 06:29 AM, Michael Zucchi wrote: Hi again, I (mostly) just have an observation to add to the bug tracker discussion on the dependency generation. Using $? will not suffice as a dependency check, as it's trivially easy to create an example which will compile ok after a change but create a broken jar. e.g. add a new abstract method to an abstract class but forget to fix all sub-classes. I don't really follow here, sorry (likely because I know almost nothing about Java). Do you mean that, if you have a bunch of .java files that get compiled into a single jar, and you change just one of these files, you also need to recompile all the other ones in order not to risk ending up with a broken and/or inconsistent jar? If it is so, that's awful :-( Think of it like changing a header file in C to change the definition of a struct. If you don't recompile all of the C source that references that struct, you get broken code. The same thing is true in Java, except that Java doesn't have separate files for interfaces versus implementations (or at least doesn't mandate it; that's a coding style that some people use and other people don't). Java development environments like Eclipse figure out the necessary dependencies and know what *.java files need to be recompiled to pick up changes (and detect errors). If one is using a command-line compiler that can't generate similar sorts of dependency information, it's usually best to just rebuild all the *.java files that make up a JAR if any of them have changed, to ensure that no errors have been introduced and no internal APIs have changed. -- Russ Allbery (r...@stanford.edu) http://www.eyrie.org/~eagle/
Re: install-strip variant that strips then installs?
On 05/08/2013 02:12 AM, Rhys Ulerich wrote: I gather that 'make install-strip' installs and then strips binaries. Is there some variant that reverses the order? If not, any recommendations for how to write one in an Automake-compliant manner? Hi Rhys, I'm tempted to believe the DESTDIR feature could be useful here with something like make install-strip DESTDIR=`pwd`/tmp and then copy files under 'tmp' into your final destination. Cheers, Peter
Re: bug#9088: Java, JARS primary?
On 13/05/13 02:28, Stefano Lattarini wrote: On 05/12/2013 06:29 AM, Michael Zucchi wrote: Using $? will not suffice as a dependency check, as it's trivially easy to create an example which will compile ok after a change but create a broken jar. e.g. add a new abstract method to an abstract class but forget to fix all sub-classes. I don't really follow here, sorry (likely because I know almost nothing about Java). Do you mean that, if you have a bunch of .java files that get compiled into a single jar, and you change just one of these files, you also need to recompile all the other ones in order not to risk ending up with a broken and/or inconsistent jar? If it is so, that's awful :-( Well if you were only going on the timestamps of the changed files it would not work. I'm not sure it's really that awful - it's done in C because that's how C is compiled, and compilation is so slow. If it was made fast enough the need for all the considerable complexity of creating and managing dependencies would go away. For a point of reference with a small library i have, a build of 104 java files with 22KLOC takes under 3s on my workstation. Building them one source file at a time (see next quoted block) takes about a minute. For big projects code that doesn't change often can be placed in it's own jar negating the need to rebuild that part every time downstream code changes - although one does have to rebuild everything downstream when that jar does. Hmm, now i think about it there is a possible solution to the dependency problem but i'm not sure it's worth it and it has some caveats. The main one being that stale classes could still make it to the jar (so you would have to manually make clean before make install), and it also needs a dependency tool. Because of the strict rules on java package names (they must match the directory they file is in), the import statements could be used to generate some coarse-but-accurate dependencies. As anything in the same package does not need an import statement, one would have to assume every class in the same package is included (i.e. ostensibly the same directory, but it may span multiple trees). Actually it isn't quite that simple - as you can just use fully qualified names without any import statement, so the dependency tool would have to scan the whole file. However, assuming these coarse dependencies could be generated reliably ... the following might work. Lets say the project is something like src/z/a.java imports c src/z/b.java src/z/util/c.java src/z/util/d.java A deps file is created with the cross dependencies for each file. It is simply everything in the same package, and then everything within the project that a given file imports. the deps file: src/z/a.java src/z/b.java src/z/a.java src/z/util/c.java src/z/util/c.java src/z/util/d.java i.e. src/z/*.java, src/z/utill/*.java, then dependencies for any files the Makefile: # this is so plain package names can be used VPATH=src JAVASRC=z/a.java z/b.java z/util/c.java z/util/d.java all: z.jar build.stamp: $(JAVASRC) @-mkdir classes @-rm buildsrc for n in $? ; do grep $$n deps buildsrc ; done javac -sourcepath src -d classes @buildsrc touch build.stamp rm buildsrc # just classes - in reality it needs to deal with other files z.jar: build.stamp jar cf $@ -C classes . So it takes the modified list of files and expands it to include any forward requirements. So for example, touch src/z/util/c.java will rebuild a, c, and d - all the classes which might possibly have something to with c. This simple example has some bugs in that it might over-build (e.g. touch a.java will build a, b as required but also c which is not) but that is fixable. Given the caveats, is it worth it? So without compiler support for dependency generation I think the only practical solution will be to build all files every time. Even the sub-directory holding the classes will probably need to be wiped away otherwise the jar could contain extraneous classes no longer generated from the corresponding source which would probably not be a good thing. Couldn't we put the *.class files obtained compiling a foo.java file into a (say) 'foo.d' directory, and remove rebuild only that directory whenever foo.java changes? You would have to then invoke the compiler individually on each file, which is at least 1-3 orders of magnitude slower than just compiling every file at the same time. And it wouldn't have the desired result anyway, if it can't find any required .class files it finds the corresponding source and builds them into the target location as well so then build order becomes an issue. I can go into more detail if you like ... One might ask why use make then, but there is more to real projects than a bunch of java source files that need compiling. Michael
