Re: bug#9088: Java, JARS primary?

[Stefano Lattarini]

On 05/17/2013 05:50 AM, Michael Zucchi wrote:
> On 16/05/13 19:39, Stefano Lattarini wrote:
>> On 05/16/2013 05:57 AM, Michael Zucchi wrote:
>>> On 15/05/13 22:39, Stefano Lattarini wrote:
 On 05/15/2013 01:52 PM, Michael Zucchi wrote:
> On 14/05/13 03:47, Stefano Lattarini wrote:
>
>>> Well if that's a requirement, then it just has to be added right?  It's
>>> not impossible, it's only software ...
>>>
>> I failed to parse this.  Can you rephrase?
> 
> It means the problem is an economic one, not a technical one.
> 
> It's a mostly facetious saying coined by a fellow work-mate which
> basically means 'we can do whatever you want (but it will cost you)'.
> 
> Or broken down logically, the technical problem:
> 
> a) We are solving a problem.
> b) We are using a general purpose computer to solve it.
> c) Any problem can be solved by a general purpose computer.
> d) Therefore, we can solve any problem.
> 
> (where of course, problem is a computer related problem, which this
> clearly is)
> 
> Here the cost is time that could be spent elsewhere, and maybe some
> lost/grey hairs.
> 
 Doing too much pre-processing in automake.in (that is, at Automake time)
 would prevent us from being able to grasp GNU make specific stuff.  So
 let's try to keep such pre-processing at a minimum.
>>>
>>> I'm only talking about iterating a list of strings and printing them in
>>> various ways, i'm pretty sure it does that kind of thing already.
>>>
>> Yes, but the price is reduce flexibility at Automake time.  Which might
>> be irrelevant or very important, depending on the expected usage patterns
>> of a feature.
> 
> I'm lost here.  You seem to be talking about something I haven't even
> written yet.
>
My point was that, if you need to parse and analyze the content of a
make variable (let's call it FOO) at automake runtime rather than at
make runtime, the contents of that variable will have to be rather
"static"; that is, you lose at least two abilities:

  1. Even if you just assume your users will use GNU make, you won't
 be able to employ GNU make specific functions in the definition
 of FOO, because automake doesn't grasp them anyway.

  2. Your users will likely not be able to override that variable
 at make runtime (which might be nice for debugging or testing
 purpose).

Now, depending on the situation, the intended uses for a feature, etc.,
the loss of those two abilities can either be irrelevant, merely annoying,
or tragic.

> [SNIP]

>>> My initial version listed every file, and used the gnu-make specific
>>> extensions in order to make it not "totally suck".   But then I ran into
>>> problems with how to handle resources and rather than work on solving
>>> those in a way which could fit into an automake.am fragment, I thought
>>> i'd try the approach I had desired from the start.
>>>
>> Given my ignorance of Java, I'm fully ready to change my mind on
>> this; but then I need examples that are much more clear, complete,
>> self-explanatory and *working*.  I think you should start perusing
>> the java tests in the 't/' subdirectory of the Automake source tree,
>> and start writing similar tests that show how you envision your
>> feature to work in different scenarios.  At this stage, using GNU
>> make specific constructs is OK; I promise I won't bother you about
>> them anymore.
> 
> Well the makefile works as much as it was designed to.
>
What I mean is that I'd like to have a test script in 't/', say
't/java-new.sh', that demonstrate how your features work, and that
I can run with a simple "make check TESTS=t/java-new".  That kind
of automation is going great lengths toward making me more willing
to try out your code, think about possible problems, and clarify my
doubts (hopefully).

> I don't really understand how the unit tests work.
>
You can try starting from an existing test, and modify it as fit.  If
you have doubts, take a look at the 't/README' file.  If you get stuck,
just ask here for help.

> It's a bit frustrating trying to make you enough of a java build expert
> in a handful of emails when i'm not one myself.  Perhaps it's just a lot
> simpler than you expect - it has far fewer complexities than
> cross-platform c libraries.
>
I sincerely hope so ;-)  I think most of the issues here are just due to
newbie misunderstanding on my part.  Just be patient.

>>> This particular target was all very simple until i added the LIBADD
>>> dependencies which meant i could no longer just use $^ as i had.
>>>
>>> I couldn't think of a way of having $^ just list .java files whilst
>>> still depending on the LIBADD stuff for running the compiler but if that
>>> was possible the problem would vanish.
>>>
>> Consider that $^ is not portable to all non-GNU makes, IIRC (but I'm not
>> 100% sure of this, I'll have to double check), so that we'd likely still
>> need to rewrite such idioms when we make the transition from your
>> prototype from the final implementation.  Not a big deal ri

Re: bug#9088: Java, JARS primary?

[Michael Zucchi]

On 16/05/13 19:39, Stefano Lattarini wrote:
> On 05/16/2013 05:57 AM, Michael Zucchi wrote:
>> On 15/05/13 22:39, Stefano Lattarini wrote:
>>> On 05/15/2013 01:52 PM, Michael Zucchi wrote:
 On 14/05/13 03:47, Stefano Lattarini wrote:

>> Well if that's a requirement, then it just has to be added right?  It's
>> not impossible, it's only software ...
>>
> I failed to parse this.  Can you rephrase?

It means the problem is an economic one, not a technical one.

It's a mostly facetious saying coined by a fellow work-mate which
basically means 'we can do whatever you want (but it will cost you)'.

Or broken down logically, the technical problem:

a) We are solving a problem.
b) We are using a general purpose computer to solve it.
c) Any problem can be solved by a general purpose computer.
d) Therefore, we can solve any problem.

(where of course, problem is a computer related problem, which this
clearly is)

Here the cost is time that could be spent elsewhere, and maybe some
lost/grey hairs.

>>> Doing too much pre-processing in automake.in (that is, at Automake time)
>>> would prevent us from being able to grasp GNU make specific stuff.  So
>>> let's try to keep such pre-processing at a minimum.
>>
>> I'm only talking about iterating a list of strings and printing them in
>> various ways, i'm pretty sure it does that kind of thing already.
>>
> Yes, but the price is reduce flexibility at Automake time.  Which might
> be irrelevant or very important, depending on the expected usage patterns
> of a feature.

I'm lost here.  You seem to be talking about something I haven't even
written yet.

> OK, then we need integration with javado and juinit.  But first let's
> try to get the jar stuff right (that seems already non-trivial).

Only in-so-much as it isn't identical to a libtool library.

And even then libtool gets a leg-up by putting all the messy stuff into
another tool.  A 'javatool' would likewise make most of the makefile
trivial.  I'm not proposing such a tool, useful as it might be.

>> Not sure why you bothered pointing out the gnu make stuff every time
>> when i stated that was a shortcoming for inclusion in automake.
>>
> Because it is not obvious to me how this usages will be later converted
> to be portable.

Ahh, I figured you'd be a portable makefile guru working on something
like automake and could help work it out.  I think with 'b.2' I can do
away with the trickier stuff.

In 20 odd years i've only ever used gnu make because it can be built
anywhere by design.  Well apart from maybe some sunos but I think
gcc/make were the first tools we installed there anyway.  And one
windows 7 sdk makefile, but that doesn't even pretend to be a real make.

>> My initial version listed every file, and used the gnu-make specific
>> extensions in order to make it not "totally suck".   But then I ran into
>> problems with how to handle resources and rather than work on solving
>> those in a way which could fit into an automake.am fragment, I thought
>> i'd try the approach I had desired from the start.
>>
> Given my ignorance of Java, I'm fully ready to change my mind on
> this; but then I need examples that are much more clear, complete,
> self-explanatory and *working*.  I think you should start perusing
> the java tests in the 't/' subdirectory of the Automake source tree,
> and start writing similar tests that show how you envision your
> feature to work in different scenarios.  At this stage, using GNU
> make specific constructs is OK; I promise I won't bother you about
> them anymore.

Well the makefile works as much as it was designed to.

I don't really understand how the unit tests work.

It's a bit frustrating trying to make you enough of a java build expert
in a handful of emails when i'm not one myself.  Perhaps it's just a lot
simpler than you expect - it has far fewer complexities than
cross-platform c libraries.

>> This particular target was all very simple until i added the LIBADD
>> dependencies which meant i could no longer just use $^ as i had.
>>
>> I couldn't think of a way of having $^ just list .java files whilst
>> still depending on the LIBADD stuff for running the compiler but if that
>> was possible the problem would vanish.
>>
> Consider that $^ is not portable to all non-GNU makes, IIRC (but I'm not
> 100% sure of this, I'll have to double check), so that we'd likely still
> need to rewrite such idioms when we make the transition from your
> prototype from the final implementation.  Not a big deal right now, but
> something worth keeping in mind.

Really?  Bummer.  How does one get the list of requirements then, or is
make pretty much hard-coded for C with a single source file using $> Actually the end result I need explicitly for jar is this:
>>
>>  -C $(srcdir)/src a/a.txt \
>>  -C $(srcdir)/src b/b.txt \
>>  -C $(srcdir)/res c/c.txt
>>
> But how can the code know, in general, to only strip the first component
> of the resources file?  That is, why:
> 
> -C res c/c.txt

Re: bug#9088: Java, JARS primary?

[Stefano Lattarini]

On 05/16/2013 05:57 AM, Michael Zucchi wrote:
> On 15/05/13 22:39, Stefano Lattarini wrote:
>> On 05/15/2013 01:52 PM, Michael Zucchi wrote:
>>> On 14/05/13 03:47, Stefano Lattarini wrote:
>>>
 Instead, let's start implementing something *correct*, in line with
 the Java philosophy, and with a clean API.  We'll think about enhancing
 it when (and if!) the need arise.
> 
> And then most of the objections are from trying the java way.  Oh well.
> 
>>> Seems the way to go.
>>>
>>> On that, here are a few more thoughts on java's specific way of doing
>>> things ...
>>>
>>> files vs dirs
>>> --
>>>
>>> java projects/build systems tend to have a fairly basic mechanism to
>>> define the source tree(s) which would be nice to emulate, but may not
>>> fit well with the portability requirements of automake.
>>>
>>> They simply define the root directory and let the build system do the rest.
>>>
>>> Not necessary but would be nice.  Or put another way it would basically
>>> suck if it required every source file to be listed explicitly and auto*
>>> having to be re-run every time one is added.
>>>
>> But then, how could make handle the case of generated java files?  I.e.,
>> if half of your files are generated by (say) an awk+sed incantation,
>> make has to know it must build them before trying to call the java compiler.
>> But see below.
> 
> Well if that's a requirement, then it just has to be added right?  It's
> not impossible, it's only software ...
>
I failed to parse this.  Can you rephrase?

> Either explicitly listed separately or handled automatically, it's quite
> easy to tell a file from a directory.
>
Ah, OK, so you mean something like the following:

   jar_JARS = foo.jar
   # All .java files in directory 'foo' ...
   foo_jar_SOURCES = foo/
   # ... plus this one that is not expected to be present at automake
   # time, but will be generated by make.
   foo_jar_SOURCES += foo/generated.java

>>> In a GNU makefile I can
>>> just use $(shell find) (or other fancier stuff), but for automake?
>>>
>> The solution is simpler than it seems at first sight: we should just
> 
> Actually it isn't quite that simple.  I did consider this. More below
> about resources.
> 
>> For an example of how this approach could work, take a look at the
>> EXTRA_DIST support (and to the test 't/extra-dist-wildcards-gnu.sh').
> 
> Although ... one can specify directories to EXTRA_DIST (even if not
> recommended).
>
> EXTA_DIST is defined relative to the makefile it is in,
>
Correct.

> but that isn't good enough.  It works fine for a root makefile since well,
> it's generating the source tree and that is by definition relative to the
> root makefile.
>
I'm lost again.  EXTRA_DIST is routinely used in subdir Makefiles ...

> For jars it needs to be relative to a another location
> which is relative to the makefile.
>
Failed to parse, sorry (and my utter ignorance about the Java world
is not helping me out here).

> The bit about nobase in the manual has an impractical solution of
> listing every directory separately.
>
But 'nobase_' is only needed for files that are going to be installed.
In our case, no .java, .class or resource file is going to be installed;
they are going to be used to create the .jar files, and *that* is going
to be installed.

> Somehow the makefile needs this information, either by listing it or by
> it's location, it cannot determine it automatically.  More detail and an
> example well below.
> 
>>> makefile location
>>> -
>>>
>>> C projects often have a make file per directory, java has the build
>>> script at the top-level at least, outside of the source trees.
>>>
>>> This is desirable.
>>>
>> In some ways, this is desirable also for C projects actually; see the
>> article "Recursive Make Considered Harmful" from Peter Miller:
>>
>> 
> 
> Yeah well that's just an opinion.  I don't like it myself because it
> makes emacs a total pain to use and I personally consider that pretty
> important.
> 
>>> For the resource files the Makefile needs to resolve the resource root,
>>> and strip it out of the names used for the dependencies.  So either it
>>> needs a (set of) explicit resource_root, resource_root_files variables,
>>> or it just needs to be defined as the root dir and handled either in the
>>> makefile fragment or in automake.in.
>>>
>> Doing too much pre-processing in automake.in (that is, at Automake time)
>> would prevent us from being able to grasp GNU make specific stuff.  So
>> let's try to keep such pre-processing at a minimum.
> 
> I'm only talking about iterating a list of strings and printing them in
> various ways, i'm pretty sure it does that kind of thing already.
>
Yes, but the price is reduce flexibility at Automake time.  Which might
be irrelevant or very important, depending on the expected usage patterns
of a feature.

> It wont be running find or walking directory trees since that doesn't
> provide an

Re: bug#9088: Java, JARS primary?

[Michael Zucchi]

On 15/05/13 22:39, Stefano Lattarini wrote:
> On 05/15/2013 01:52 PM, Michael Zucchi wrote:
>> On 14/05/13 03:47, Stefano Lattarini wrote:
>>
>>> Instead, let's start implementing something *correct*, in line with
>>> the Java philosophy, and with a clean API.  We'll think about enhancing
>>> it when (and if!) the need arise.

And then most of the objections are from trying the java way.  Oh well.

>> Seems the way to go.
>>
>> On that, here are a few more thoughts on java's specific way of doing
>> things ...
>>
>> files vs dirs
>> --
>>
>> java projects/build systems tend to have a fairly basic mechanism to
>> define the source tree(s) which would be nice to emulate, but may not
>> fit well with the portability requirements of automake.
>>
>> They simply define the root directory and let the build system do the rest.
>>
>> Not necessary but would be nice.  Or put another way it would basically
>> suck if it required every source file to be listed explicitly and auto*
>> having to be re-run every time one is added.
>>
> But then, how could make handle the case of generated java files?  I.e.,
> if half of your files are generated by (say) an awk+sed incantation,
> make has to know it must build them before trying to call the java compiler.
> But see below.

Well if that's a requirement, then it just has to be added right?  It's
not impossible, it's only software ...

Either explicitly listed separately or handled automatically, it's quite
easy to tell a file from a directory.

>> In a GNU makefile I can
>> just use $(shell find) (or other fancier stuff), but for automake?
>>
> The solution is simpler than it seems at first sight: we should just

Actually it isn't quite that simple.  I did consider this. More below
about resources.

> For an example of how this approach could work, take a look at the
> EXTRA_DIST support (and to the test 't/extra-dist-wildcards-gnu.sh').

Although ... one can specify directories to EXTRA_DIST (even if not
recommended).

EXTA_DIST is defined relative to the makefile it is in, but that isn't
good enough.  It works fine for a root makefile since well, its'
generating the source tree and that is by definition relative to the
root makefile.  For jars it needs to be relative to a another location
which is relative to the makefile.

The bit about nobase in the manual has an impractical solution of
listing every directory separately.

Somehow the makefile needs this information, either by listing it or by
it's location, it cannot determine it automatically.  More detail and an
example well below.

>> makefile location
>> -
>>
>> C projects often have a make file per directory, java has the build
>> script at the top-level at least, outside of the source trees.
>>
>> This is desirable.
>>
> In some ways, this is desirable also for C projects actually; see the
> article "Recursive Make Considered Harmful" from Peter Miller:
> 
> 

Yeah well that's just an opinion.  I don't like it myself because it
makes emacs a total pain to use and I personally consider that pretty
important.

>> For the resource files the Makefile needs to resolve the resource root,
>> and strip it out of the names used for the dependencies.  So either it
>> needs a (set of) explicit resource_root, resource_root_files variables,
>> or it just needs to be defined as the root dir and handled either in the
>> makefile fragment or in automake.in.
>>
> Doing too much pre-processing in automake.in (that is, at Automake time)
> would prevent us from being able to grasp GNU make specific stuff.  So
> let's try to keep such pre-processing at a minimum.

I'm only talking about iterating a list of strings and printing them in
various ways, i'm pretty sure it does that kind of thing already.  It
wont be running find or walking directory trees since that doesn't
provide anything useful at 'make' time.

>> javadoc, jni, check/test, etc
>> 
>>
>> Some built-in support would be good.  javadoc is easy, check/test should
>> run junit tests I guess although i'm not too familiar with junit,
>>
> Or we might leave that to the use from now (they can use the 'check-local'
> target), and maybe implement it in a later Automake version, once an usage
> pattern has emerged (and most importantly, *if* there actually is any
> request in that direction).

javadoc+test are pretty important in the java world, and of course
documentation is a critical component for any free software.  jni is
just a personal need and I am familiar with how it works.  I also bring
up jni because this is one area where java build tools really don't
stack up at all and automake some non-zero chance of becoming a primary
contender.

I'm afraid based on the state of things that 'waiting for a request'
will be a rather long wait ... the horse has quite bolted on the 'build
tool' front in the last few years for all languages, and one can only
surmise it was due to the com

Re: bug#9088: Java, JARS primary?

[Stefano Lattarini]

On 05/15/2013 01:52 PM, Michael Zucchi wrote:
> On 14/05/13 03:47, Stefano Lattarini wrote:
> 
>> Instead, let's start implementing something *correct*, in line with
>> the Java philosophy, and with a clean API.  We'll think about enhancing
>> it when (and if!) the need arise.
> 
> Seems the way to go.
> 
> On that, here are a few more thoughts on java's specific way of doing
> things ...
> 
> files vs dirs
> --
> 
> java projects/build systems tend to have a fairly basic mechanism to
> define the source tree(s) which would be nice to emulate, but may not
> fit well with the portability requirements of automake.
> 
> They simply define the root directory and let the build system do the rest.
> 
> Not necessary but would be nice.  Or put another way it would basically
> suck if it required every source file to be listed explicitly and auto*
> having to be re-run every time one is added.
>
But then, how could make handle the case of generated java files?  I.e.,
if half of your files are generated by (say) an awk+sed incantation,
make has to know it must build them before trying to call the java compiler.
But see below.

> In a GNU makefile I can
> just use $(shell find) (or other fancier stuff), but for automake?
>
The solution is simpler than it seems at first sight: we should just
implement the new feature in a way that can handle the use of GNU make
specific functions, without requiring too much cooperation from Automake
(that means we have to delegate most of the work at make runtime).
So, people only interested in having their Makefile work with GNU make
only will be able to use all the needed tricks with "$(shell find ...)"
or "$(wildcard ...)", while people interested in wider portability we'll
have to bite the bullet and explicitly list all their java source files.
A sort of graceful degradation.

For an example of how this approach could work, take a look at the
EXTRA_DIST support (and to the test 't/extra-dist-wildcards-gnu.sh').


> source vs data
> --
> 
> A source tree can also include non-source data which is copied into the
> jar file verbatim at the same relative path location.  At least this is
> how ant does it and given the way these resources are resolved at
> runtime makes the most sense.
> 
> Again I think this needs to be specified by root directory not file.  It
> could just hang off the _SOURCES or use _RESOURCES.
>
Seems reasonable.

> makefile location
> -
> 
> C projects often have a make file per directory, java has the build
> script at the top-level at least, outside of the source trees.
>
> This is desirable.
>
In some ways, this is desirable also for C projects actually; see the
article "Recursive Make Considered Harmful" from Peter Miller:



And recently, the GNU world has started moving towards making
non-recursive make-based build system a first class citizen.
For example, important GNU packages like Coreutils and Bison has
been converted to a non-recursive build system; and Automake has
been enhanced to make its use in non-recursive setups easier and
more convenient:

   

> For the compiled code this is simple enough since the makefile will
> compile every file every time and the compiler will dump the class files
> where it's told.
> 
> For the resource files the Makefile needs to resolve the resource root,
> and strip it out of the names used for the dependencies.  So either it
> needs a (set of) explicit resource_root, resource_root_files variables,
> or it just needs to be defined as the root dir and handled either in the
> makefile fragment or in automake.in.
> 
Doing too much pre-processing in automake.in (that is, at Automake time)
would prevent us from being able to grasp GNU make specific stuff.  So
let's try to keep such pre-processing at a minimum.

> annotation processors
> -
> 
> I think these can be handled by using _JAVACFLAGS or something similar.
>  They can generate source for new classes at compile time, but this is
> all handled by javac in separate locations so make doesn't need to know
> about it.
> 
Sounds good.

> javadoc, jni, check/test, etc
> 
> 
> Some built-in support would be good.  javadoc is easy, check/test should
> run junit tests I guess although i'm not too familiar with junit,
>
Or we might leave that to the use from now (they can use the 'check-local'
target), and maybe implement it in a later Automake version, once an usage
pattern has emerged (and most importantly, *if* there actually is any
request in that direction).

> jni will need some thought but should be limited in scope (support for
> javah basically, maybe not enough need to include it in automake so long as
> the dev can depend on the jar).
>
Sorry, I don't know enough on JNI to comment on this.  I understand that
it is a way to include platform-specific code in a Java application, but
I 

Re: bug#9088: Java, JARS primary?

[Michael Zucchi]

On 14/05/13 03:47, Stefano Lattarini wrote:

> Instead, let's start implementing something *correct*, in line with
> the Java philosophy, and with a clean API.  We'll think about enhancing
> it when (and if!) the need arise.

Seems the way to go.

On that, here are a few more thoughts on java's specific way of doing
things ...

files vs dirs
--

java projects/build systems tend to have a fairly basic mechanism to
define the source tree(s) which would be nice to emulate, but may not
fit well with the portability requirements of automake.

They simply define the root directory and let the build system do the rest.

Not necessary but would be nice.  Or put another way it would basically
suck if it required every source file to be listed explicitly and auto*
having to be re-run every time one is added.  In a GNU makefile I can
just use $(shell find) (or other fancier stuff), but for automake?

source vs data
--

A source tree can also include non-source data which is copied into the
jar file verbatim at the same relative path location.  At least this is
how ant does it and given the way these resources are resolved at
runtime makes the most sense.

Again I think this needs to be specified by root directory not file.  It
could just hang off the _SOURCES or use _RESOURCES.

makefile location
-

C projects often have a make file per directory, java has the build
script at the top-level at least, outside of the source trees.

This is desirable.

For the compiled code this is simple enough since the makefile will
compile every file every time and the compiler will dump the class files
where it's told.

For the resource files the Makefile needs to resolve the resource root,
and strip it out of the names used for the dependencies.  So either it
needs a (set of) explicit resource_root, resource_root_files variables,
or it just needs to be defined as the root dir and handled either in the
makefile fragment or in automake.in.

annotation processors
-

I think these can be handled by using _JAVACFLAGS or something similar.
 They can generate source for new classes at compile time, but this is
all handled by javac in separate locations so make doesn't need to know
about it.

javadoc, jni, check/test, etc


Some built-in support would be good.  javadoc is easy, check/test should
run junit tests I guess although i'm not too familiar with junit, jni
will need some thought but should be limited in scope (support for javah
basically, maybe not enough need to include it in automake so long as
the dev can depend on the jar).

first cut
-

I've attached a pretty nasty bit of unportable makefile which attempts
some of the above to see how it might be done.  Even with gnu make stuff
it still needs some support from automake.in to handle multiple resource
directories, and it doesn't handle everything quite properly.

The build steps from this are simple:

$ rm foo_jar.stamp ; make -f Auto.mak
javac -cp lib/jjmpeg.jar:lib/test.jar -sourcepath src -d
build/classes/foo_jar src/z/util/d.java src/z/util/c.java src/z/a.java
src/z/b.java
touch foo_jar.stamp
jar cf foo.jar -C build/classes/foo_jar . \
-C ./src ./z/util/util-names.txt \
-C ./res ./foo.txt
$

Regards,
 Michael

# default is /usr/share/java/package ?
jardir = $(datadir)/java/@PACKAGE@
srcdir=.

all: foo.jar

jar_JARS=foo.jar

foo_jar_SOURCES = src
# multiple resources dirs
foo_jar_RESOURCES = src res
# same as in the way one would add a in-package libtool lib, but this is in the 
build jar path
foo_jar_LIBADD = lib/jjmpeg.jar lib/test.jar

## foo.jar

# jar.am template, could handle multi-valued foo_jar_SOURCES
foo_jar_SOURCES_list=$(shell cd $(srcdir) ; find $(foo_jar_SOURCES) -name 
'*.java')
# automake.in generated, the find exclusions are not nearly enough to handle 
version control
foo_jar_RESOURCES_list0=$(shell cd $(srcdir)/src; find . -type f -a \! -name 
'*.java' \! -name '*~')
foo_jar_RESOURCES_list1=$(shell cd $(srcdir)/res; find . -type f -a \! -name 
'*.java' \! -name '*~')

# windows needs ;, of course ...
# automake.in generates this from LIBADD?
foo_jar_CLASSPATH=lib/jjmpeg.jar:lib/test.jar

foo_jar.stamp: $(foo_jar_SOURCES_list) $(foo_jar_LIBADD)
@-rm -rf build/classes/foo_jar
@-mkdir -p build/classes/foo_jar
javac -cp $(foo_jar_CLASSPATH) -sourcepath src -d build/classes/foo_jar 
$(filter %.java,$^)
touch $@

foo.jar: foo_jar.stamp $(patsubst %,src/%,$(foo_jar_RESOURCES_list0)) 
$(patsubst %,res/%,$(foo_jar_RESOURCES_list1))
jar cf $@ -C build/classes/foo_jar . \
$(patsubst %,-C $(srcdir)/src %,$(foo_jar_RESOURCES_list0)) \
$(patsubst %,-C $(srcdir)/res %,$(foo_jar_RESOURCES_list1))

.PHONY javadoc: javadoc-foo_jar
javadoc-foo_jar:
exit 1
build javadoc stuff, not done yet

Re: bug#9088: Java, JARS primary?

[Stefano Lattarini]

Given all the rationales given by Russ and Micheal, I think we should
drop the idea of having "smart dependencies" for the moment being (I
suggested that because, in my ignorance of Java, I thought they would
be easy to implement).

Instead, let's start implementing something *correct*, in line with
the Java philosophy, and with a clean API.  We'll think about enhancing
it when (and if!) the need arise.

Thanks,
  Stefano

Re: bug#9088: Java, JARS primary?

[Michael Zucchi]

On 13/05/13 02:28, Stefano Lattarini wrote:
> On 05/12/2013 06:29 AM, Michael Zucchi wrote:

>> Using $? will not suffice as a dependency check, as it's trivially easy
>> to create an example which will compile ok after a change but create a
>> broken jar. e.g. add a new abstract method to an abstract class but
>> forget to fix all sub-classes.
>>
> I don't really follow here, sorry (likely because I know almost nothing
> about Java).  Do you mean that, if you have a bunch of .java files that
> get compiled into a single jar, and you change just one of these files,
> you also need to recompile all the other ones in order not to risk ending
> up with a broken and/or inconsistent jar?  If it is so, that's awful :-(

Well if you were only going on the timestamps of the changed files it
would not work.

I'm not sure it's really that awful - it's done in C because that's how
C is compiled, and compilation is so slow.  If it was made fast enough
the need for all the considerable complexity of creating and managing
dependencies would go away.

For a point of reference with a small library i have, a build of 104
java files with 22KLOC takes under 3s on my workstation.  Building them
"one source file at a time" (see next quoted block) takes about a minute.

For big projects code that doesn't change often can be placed in it's
own jar negating the need to rebuild that part every time downstream
code changes - although one does have to rebuild everything downstream
when that jar does.

Hmm, now i think about it there is a possible solution to the dependency
problem but i'm not sure it's worth it and it has some caveats.  The
main one being that stale classes could still make it to the jar (so you
would have to manually make clean before make install), and it also
needs a dependency tool.

Because of the strict rules on java package names (they must match the
directory they file is in), the import statements could be used to
generate some coarse-but-accurate dependencies.  As anything in the same
package does not need an import statement, one would have to assume
every class in the same package is included (i.e. ostensibly the same
directory, but it may span multiple trees).

Actually it isn't quite that simple - as you can just use fully
qualified names without any import statement, so the dependency tool
would have to scan the whole file.

However, assuming these coarse dependencies could be generated reliably
... the following might work.

Lets say the project is something like

src/z/a.java imports c
src/z/b.java
src/z/util/c.java
src/z/util/d.java

A deps file is created with the cross dependencies for each file.  It is
simply everything in the same package, and then everything within the
project that a given file imports.

the deps file:

src/z/a.java src/z/b.java
src/z/a.java src/z/util/c.java
src/z/util/c.java src/z/util/d.java

i.e.
src/z/*.java, src/z/utill/*.java, then dependencies for any files

the Makefile:

# this is so plain package names can be used
VPATH=src

JAVASRC=z/a.java z/b.java z/util/c.java z/util/d.java

all: z.jar

build.stamp: $(JAVASRC)
@-mkdir classes
@-rm buildsrc
for n in $? ; do grep $$n deps >> buildsrc ; done
javac -sourcepath src -d classes @buildsrc
touch build.stamp
rm buildsrc

# just classes - in reality it needs to deal with other files
z.jar: build.stamp
jar cf $@ -C classes .

So it takes the modified list of files and expands it to include any
forward requirements.

So for example, touch src/z/util/c.java will rebuild a, c, and d - all
the classes which might possibly have something to with c.

This simple example has some bugs in that it might over-build (e.g.
touch a.java will build a, b as required but also c which is not) but
that is fixable.

Given the caveats, is it worth it?

>> So without compiler support for dependency generation I think the only
>> practical solution will be to build all files every time.  Even the
>> sub-directory holding the classes will probably need to be wiped away
>> otherwise the jar could contain extraneous classes no longer generated
>> from the corresponding source which would probably not be a good thing.

> Couldn't we put the *.class files obtained compiling a foo.java file
> into a (say) 'foo.d' directory, and remove & rebuild only that directory
> whenever foo.java changes?

You would have to then invoke the compiler individually on each file,
which is at least 1-3 orders of magnitude slower than just compiling
every file at the same time.  And it wouldn't have the desired result
anyway, if it can't find any required .class files it finds the
corresponding source and builds them into the target location as well so
then build order becomes an issue.

I can go into more detail if you like ...

One might ask why use make then, but there is more to real projects than
a bunch of java source files that need compiling.

 Michael

Re: bug#9088: Java, JARS primary?

[Russ Allbery]

Stefano Lattarini  writes:
> On 05/12/2013 06:29 AM, Michael Zucchi wrote:

>> Hi again,
>> 
>> I (mostly) just have an observation to add to the bug tracker discussion
>> on the dependency generation.
>> 
>> Using $? will not suffice as a dependency check, as it's trivially easy
>> to create an example which will compile ok after a change but create a
>> broken jar. e.g. add a new abstract method to an abstract class but
>> forget to fix all sub-classes.

> I don't really follow here, sorry (likely because I know almost nothing
> about Java).  Do you mean that, if you have a bunch of .java files that
> get compiled into a single jar, and you change just one of these files,
> you also need to recompile all the other ones in order not to risk ending
> up with a broken and/or inconsistent jar?  If it is so, that's awful :-(

Think of it like changing a header file in C to change the definition of a
struct.  If you don't recompile all of the C source that references that
struct, you get broken code.

The same thing is true in Java, except that Java doesn't have separate
files for interfaces versus implementations (or at least doesn't mandate
it; that's a coding style that some people use and other people don't).
Java development environments like Eclipse figure out the necessary
dependencies and know what *.java files need to be recompiled to pick up
changes (and detect errors).  If one is using a command-line compiler that
can't generate similar sorts of dependency information, it's usually best
to just rebuild all the *.java files that make up a JAR if any of them
have changed, to ensure that no errors have been introduced and no
internal APIs have changed.

-- 
Russ Allbery (r...@stanford.edu) 

Re: bug#9088: Java, JARS primary?

[Stefano Lattarini]

On 05/12/2013 06:29 AM, Michael Zucchi wrote:
> Hi again,
> 
> I (mostly) just have an observation to add to the bug tracker discussion
> on the dependency generation.
> 
> Using $? will not suffice as a dependency check, as it's trivially easy
> to create an example which will compile ok after a change but create a
> broken jar. e.g. add a new abstract method to an abstract class but
> forget to fix all sub-classes.
>
I don't really follow here, sorry (likely because I know almost nothing
about Java).  Do you mean that, if you have a bunch of .java files that
get compiled into a single jar, and you change just one of these files,
you also need to recompile all the other ones in order not to risk ending
up with a broken and/or inconsistent jar?  If it is so, that's awful :-(

> So without compiler support for dependency generation I think the only
> practical solution will be to build all files every time.  Even the
> sub-directory holding the classes will probably need to be wiped away
> otherwise the jar could contain extraneous classes no longer generated
> from the corresponding source which would probably not be a good thing.
>
Couldn't we put the *.class files obtained compiling a foo.java file
into a (say) 'foo.d' directory, and remove & rebuild only that directory
whenever foo.java changes?

> I have had a bit of a look at automake.in and some of the .am files - it
> seems to me it would not be any use using the existing in built language
> code as that is designed for 1:1 source:object compilation.
>
Maybe we can steal some code from the existing _JAVA primary though, were
that makes sense?

> But before I get too bogged down in that I think I will first try to
> create a simple Makefile with the required features for discussion, and
> then worry about how to generate it.
>
This is the sanest approach, yes.  You might also write some tests on the
expected behaviours of this Makefile, and we could later re-use them in
our testsuite.

> Most of it should be straightforward apart from deciding on conventions.
> 
> Regards,
>  Michael
> 

Thanks,
  Stefano