Re: bug#9088: Java, JARS primary?

2013-05-12 Thread Michael Zucchi
On 13/05/13 02:28, Stefano Lattarini wrote:
> On 05/12/2013 06:29 AM, Michael Zucchi wrote:

>> Using $? will not suffice as a dependency check, as it's trivially easy
>> to create an example which will compile ok after a change but create a
>> broken jar. e.g. add a new abstract method to an abstract class but
>> forget to fix all sub-classes.
>>
> I don't really follow here, sorry (likely because I know almost nothing
> about Java).  Do you mean that, if you have a bunch of .java files that
> get compiled into a single jar, and you change just one of these files,
> you also need to recompile all the other ones in order not to risk ending
> up with a broken and/or inconsistent jar?  If it is so, that's awful :-(

Well if you were only going on the timestamps of the changed files it
would not work.

I'm not sure it's really that awful - it's done in C because that's how
C is compiled, and compilation is so slow.  If it was made fast enough
the need for all the considerable complexity of creating and managing
dependencies would go away.

For a point of reference with a small library i have, a build of 104
java files with 22KLOC takes under 3s on my workstation.  Building them
"one source file at a time" (see next quoted block) takes about a minute.

For big projects code that doesn't change often can be placed in it's
own jar negating the need to rebuild that part every time downstream
code changes - although one does have to rebuild everything downstream
when that jar does.

Hmm, now i think about it there is a possible solution to the dependency
problem but i'm not sure it's worth it and it has some caveats.  The
main one being that stale classes could still make it to the jar (so you
would have to manually make clean before make install), and it also
needs a dependency tool.

Because of the strict rules on java package names (they must match the
directory they file is in), the import statements could be used to
generate some coarse-but-accurate dependencies.  As anything in the same
package does not need an import statement, one would have to assume
every class in the same package is included (i.e. ostensibly the same
directory, but it may span multiple trees).

Actually it isn't quite that simple - as you can just use fully
qualified names without any import statement, so the dependency tool
would have to scan the whole file.

However, assuming these coarse dependencies could be generated reliably
... the following might work.

Lets say the project is something like

src/z/a.java imports c
src/z/b.java
src/z/util/c.java
src/z/util/d.java

A deps file is created with the cross dependencies for each file.  It is
simply everything in the same package, and then everything within the
project that a given file imports.

the deps file:

src/z/a.java src/z/b.java
src/z/a.java src/z/util/c.java
src/z/util/c.java src/z/util/d.java

i.e.
src/z/*.java, src/z/utill/*.java, then dependencies for any files

the Makefile:

# this is so plain package names can be used
VPATH=src

JAVASRC=z/a.java z/b.java z/util/c.java z/util/d.java

all: z.jar

build.stamp: $(JAVASRC)
@-mkdir classes
@-rm buildsrc
for n in $? ; do grep $$n deps >> buildsrc ; done
javac -sourcepath src -d classes @buildsrc
touch build.stamp
rm buildsrc

# just classes - in reality it needs to deal with other files
z.jar: build.stamp
jar cf $@ -C classes .

So it takes the modified list of files and expands it to include any
forward requirements.

So for example, touch src/z/util/c.java will rebuild a, c, and d - all
the classes which might possibly have something to with c.

This simple example has some bugs in that it might over-build (e.g.
touch a.java will build a, b as required but also c which is not) but
that is fixable.

Given the caveats, is it worth it?

>> So without compiler support for dependency generation I think the only
>> practical solution will be to build all files every time.  Even the
>> sub-directory holding the classes will probably need to be wiped away
>> otherwise the jar could contain extraneous classes no longer generated
>> from the corresponding source which would probably not be a good thing.

> Couldn't we put the *.class files obtained compiling a foo.java file
> into a (say) 'foo.d' directory, and remove & rebuild only that directory
> whenever foo.java changes?

You would have to then invoke the compiler individually on each file,
which is at least 1-3 orders of magnitude slower than just compiling
every file at the same time.  And it wouldn't have the desired result
anyway, if it can't find any required .class files it finds the
corresponding source and builds them into the target location as well so
then build order becomes an issue.

I can go into more detail if you like ...

One might ask why use make then, but there is more to real projects than
a bunch of java source files that need compiling.

 Michael



handling nested AM_CONDITIONAL via m4 vars

2013-05-12 Thread Mike Frysinger
the AM_CONDITIONAL documentation states it must always be executed.  that's 
fine.  but what if i want to be lazy?

for example, i have code like so:
AC_DEFUN([_FOKER_DEFINE], [dnl
AC_DEFINE([FOKER_]$1[_]m4_toupper($3), [1], [Define is the ]$2[ is ]$3)
])
AC_DEFUN([_FOKER_ARCH], [_FOKER_DEFINE([ARCH], [architecture], $1)])

AS_CASE([$host_cpu],
[alpha*], [_FOKER_ARCH([Alpha])],
[aarch64*],   [_FOKER_ARCH([AArch64])],
[arm*],   [_FOKER_ARCH([ARM])],
[bfin],   [_FOKER_ARCH([bfin])],
[ia64],   [_FOKER_ARCH([IA64])],
[mips*],  [_FOKER_ARCH([MIPS])],
[hppa*],  [_FOKER_ARCH([PARISC])],
[i?86],   [_FOKER_ARCH([X86])],
[powerpc*],   [_FOKER_ARCH([PowerPC])],
[s390*],  [_FOKER_ARCH([s390])],
[sh*],[_FOKER_ARCH([SH])],
[sparc*], [_FOKER_ARCH([SPARC])],
[x86_64], [_FOKER_ARCH([x86_64])],
[*],  [AC_MSG_WARN([unsupported architecture])],
)

i want each branch to expand into both an AC_DEFINE and an AM_CONDITIONAL.  
rather than rewrite this, i thought why not use m4 vars to accumulate at 
aclocal time, and then expand them afterwards.  this way i don't have to give 
up my pretty case statement.

what i have now is:
dnl Setup AC_DEFINE with FOKER_ARCH_$ARCH, set the shell var FOKER_ARCH
dnl to $ARCH, and push $ARCH onto the _FOKER_CONDS stack.
AC_DEFUN([_FOKER_DEFINE], [dnl
AC_DEFINE([FOKER_]$1[_]m4_toupper($3), [1], [Define is the ]$2[ is ]$3)
[FOKER_]$1=[FOKER_]$1[_]m4_toupper($3)
pushdef([_FOKER_CONDS], [FOKER_]$1[_]m4_toupper($3))
])
AC_DEFUN([_FOKER_ARCH], [_FOKER_DEFINE([ARCH], [architecture], $1)])
dnl Initialize the shell var FOKER_ARCH=unknown
AC_DEFUN([_FOKER_CONDS_INIT], [dnl
AC_MSG_CHECKING([for known host ]$1)
[FOKER_]$1='unknown'
])
dnl Pop all the arches off the _FOKER_CONDS stack and call AM_CONDITIONAL for
dnl each one.
AC_DEFUN([_FOKER_CONDS_EXPAND], [dnl
ifdef([_FOKER_CONDS], [dnl
AM_CONDITIONAL(_FOKER_CONDS, test $[FOKER_]$1 = _FOKER_CONDS)
popdef([_FOKER_CONDS])
_FOKER_CONDS_EXPAND($1)
], [dnl
AC_MSG_RESULT($[FOKER_]$1)
])
])

_FOKER_CONDS_INIT([ARCH])
...existing AS_CASE as above...
_FOKER_CONDS_EXPAND([ARCH])

this seems to accomplish what i want -- AM_CONDITIONAL is expanded outside of 
the case statement w/out having to duplicate every entry.  however, my m4 
skills are beginner at best, so i'm wondering if people have suggestions for 
how to do this better.  i also wouldn't be surprised if my quoting is off.

not exactly an automake question, but close enough :)
-mike


signature.asc
Description: This is a digitally signed message part.


Re: install-strip variant that strips then installs?

2013-05-12 Thread Peter Johansson

On 05/08/2013 02:12 AM, Rhys Ulerich wrote:

I gather that 'make install-strip' installs and then strips binaries.
Is there some variant that reverses the order?  If not, any
recommendations for how to write one in an Automake-compliant manner?



Hi Rhys,

I'm tempted to believe the DESTDIR feature could be useful here with 
something like


make install-strip DESTDIR=`pwd`/tmp

and then copy files under 'tmp' into your final destination.

Cheers,
Peter




Re: bug#9088: Java, JARS primary?

2013-05-12 Thread Russ Allbery
Stefano Lattarini  writes:
> On 05/12/2013 06:29 AM, Michael Zucchi wrote:

>> Hi again,
>> 
>> I (mostly) just have an observation to add to the bug tracker discussion
>> on the dependency generation.
>> 
>> Using $? will not suffice as a dependency check, as it's trivially easy
>> to create an example which will compile ok after a change but create a
>> broken jar. e.g. add a new abstract method to an abstract class but
>> forget to fix all sub-classes.

> I don't really follow here, sorry (likely because I know almost nothing
> about Java).  Do you mean that, if you have a bunch of .java files that
> get compiled into a single jar, and you change just one of these files,
> you also need to recompile all the other ones in order not to risk ending
> up with a broken and/or inconsistent jar?  If it is so, that's awful :-(

Think of it like changing a header file in C to change the definition of a
struct.  If you don't recompile all of the C source that references that
struct, you get broken code.

The same thing is true in Java, except that Java doesn't have separate
files for interfaces versus implementations (or at least doesn't mandate
it; that's a coding style that some people use and other people don't).
Java development environments like Eclipse figure out the necessary
dependencies and know what *.java files need to be recompiled to pick up
changes (and detect errors).  If one is using a command-line compiler that
can't generate similar sorts of dependency information, it's usually best
to just rebuild all the *.java files that make up a JAR if any of them
have changed, to ensure that no errors have been introduced and no
internal APIs have changed.

-- 
Russ Allbery (r...@stanford.edu) 



Re: bug#9088: Java, JARS primary?

2013-05-12 Thread Stefano Lattarini
On 05/12/2013 06:29 AM, Michael Zucchi wrote:
> Hi again,
> 
> I (mostly) just have an observation to add to the bug tracker discussion
> on the dependency generation.
> 
> Using $? will not suffice as a dependency check, as it's trivially easy
> to create an example which will compile ok after a change but create a
> broken jar. e.g. add a new abstract method to an abstract class but
> forget to fix all sub-classes.
>
I don't really follow here, sorry (likely because I know almost nothing
about Java).  Do you mean that, if you have a bunch of .java files that
get compiled into a single jar, and you change just one of these files,
you also need to recompile all the other ones in order not to risk ending
up with a broken and/or inconsistent jar?  If it is so, that's awful :-(

> So without compiler support for dependency generation I think the only
> practical solution will be to build all files every time.  Even the
> sub-directory holding the classes will probably need to be wiped away
> otherwise the jar could contain extraneous classes no longer generated
> from the corresponding source which would probably not be a good thing.
>
Couldn't we put the *.class files obtained compiling a foo.java file
into a (say) 'foo.d' directory, and remove & rebuild only that directory
whenever foo.java changes?

> I have had a bit of a look at automake.in and some of the .am files - it
> seems to me it would not be any use using the existing in built language
> code as that is designed for 1:1 source:object compilation.
>
Maybe we can steal some code from the existing _JAVA primary though, were
that makes sense?

> But before I get too bogged down in that I think I will first try to
> create a simple Makefile with the required features for discussion, and
> then worry about how to generate it.
>
This is the sanest approach, yes.  You might also write some tests on the
expected behaviours of this Makefile, and we could later re-use them in
our testsuite.

> Most of it should be straightforward apart from deciding on conventions.
> 
> Regards,
>  Michael
> 

Thanks,
  Stefano



Re: install-strip variant that strips then installs?

2013-05-12 Thread Gavin Smith
On Tue, May 7, 2013 at 5:12 PM, Rhys Ulerich  wrote:
> I gather that 'make install-strip' installs and then strips binaries.
> Is there some variant that reverses the order?  If not, any
> recommendations for how to write one in an Automake-compliant manner?
>
> My unstripped binaries are absurdly large and my installation
> directory is NFS-mounted.  So I get to pay lots of network overhead to
> install what eventually becomes O(100MB) of binaries because the
> unstripped copy is O(1.5GB).
>
> Thanks,
> Rhys
>

This seems like a good idea to me. Is there any reason why the order
couldn't be reversed?

The only problem I can think of is that make install-strip isn't
expected to modify the binaries in the build directory, and the user
might conceivably be relying on them being unstripped (for some
obscure reason). If that could be a problem, perhaps a solution is to
have a separate "strip" rule which could be run.

You could try writing a rule yourself in your Makefile.am to strip the
binaries. You could use the bin_PROGRAMS make variable that is set in
the output Makefile.