Re: automake -vs- huge projects

[Tom Tromey]

> "adl" == Alexandre Duret-Lutz <[EMAIL PROTECTED]> writes:

adl> Furthermore, generally it does not work to compile both the .o
adl> and .lo objects of a source file (in the last example Automake
adl> is expected to warn that these files are being built both with
adl> and without Libtool), so it sounds safe to remove the rules
adl> which are not used.

This would be a great change, but unfortunately it only gets us down
to 3M, which is still about 6x too large.

Tom

Re: automake -vs- huge projects

[Tom Tromey]

> "adl" == Alexandre Duret-Lutz <[EMAIL PROTECTED]> writes:

adl> Couldn't we use the (existing) .java.o: inference rule in this
adl> case?  Actually, is there a difference between `%.o: %.java' and
adl> `.java.o:' beside portability?  -- I'm not asking about the
adl> general % construction, just about this precise case where both
adl> sides are expected to be in the same directory.

My recollection is that we tried this and ran into real problems with
it.  I don't remember what they are any more.

There's an old thread on this:

  http://sources.redhat.com/ml/automake/1999-04/threads.html#00033

At this point I still believed that these suffix rules would work.  I
couldn't find the point where we changed things.

adl> So the simplest part of the fix would be to disable the output
adl> of explicit rule for subdirectory sources without per-target
adl> flags when subdir-objects is used.

Yeah.  That would help us a lot, but...

adl> The other side of the coin is that dependency tracking will not
adl> work anymore, because the dependency stuff for subdir/X.o should
adl> go into subdir/.deps/X.Po but the default suffix rule will put
adl> it in ./.deps/subdir/X.Po.  I don't see an easy way to fix this,
adl> unless we add some clumsy shell computation in the suffix rules
adl> (while this can probably be folded into depcomp when it is used,
adl> it does not seem to fit well in fastdep rules).

Hmm, maybe this is the issue from way back.

We definitely want to keep dependency tracking.  This is pretty
important.

fastdep would be great, since we know we'll always be using gcc.  If
somehow dropping fastdep would get the Makefile.in to a reasonable
size, though, I'd be in favor of that.

Alternatively running sed or whatever once or twice before the
compilation isn't going to hurt as much as having a 9M Makefile.in.
So we could just do the rewrite in the suffix rule and pay the cost.

adl> Note that this issue is unrelated to the %.o:%.java vs. .java.o:
adl> choice.

Not completely, since GNU make might give us handy macros that would
let us do this transformation in-line.

Tom

Re: automake -vs- huge projects

[Alexandre Duret-Lutz]

>>> "Thomas" == Thomas Fitzsimmons <[EMAIL PROTECTED]> writes:

[...]
 Thomas> One specifc problem I noticed is that automake is
 Thomas> generating three explicit rules for each target.  For
 Thomas> example, for one .java file, these rules are generated:
[...]
 Thomas> I would only expect rules for .lo files, since we are
 Thomas> generating libtool libraries.  Getting rid of the other
 Thomas> targets would cut out *a lot* of the bulk.

Indeed!

I think we can construct cases that will break if any of the .o
or .lo rules are left out, for instance

  bin_PROGRAMS = foo
  foo_SOURCES = a.c b.c
  foo_LDADD = @MOREFOO@
  lib_LTLIBRARIES = libbar.la
  libbar_la_SOURCES = 1.c 2.c
  libbar_la_LIBADD = @MOREBAR@

now think about @MOREFOO@ being substituted into 1.o or 2.o, and
@MOREBAR@ being replaced by a.la or b.la...  However it seems
fair to require that the above snippet be rewritten as

  bin_PROGRAMS = foo
  foo_SOURCES = a.c b.c
  foo_LDADD = @MOREFOO@
  EXTRA_foo_SOURCES = 1.c 2.c
  lib_LTLIBRARIES = libbar.la
  libbar_la_SOURCES = 1.c 2.c
  libbar_la_LDADD =  @MOREBAR@
  EXTRA_libbar_la_SOURCES = a.c b.c

like the manual says (oh well, I hope it says so).  And in this
second example it is clear which rules Automake needs to output.

Furthermore, generally it does not work to compile both the .o
and .lo objects of a source file (in the last example Automake
is expected to warn that these files are being built both with
and without Libtool), so it sounds safe to remove the rules
which are not used.
-- 
Alexandre Duret-Lutz

Re: automake -vs- huge projects

[Bob Friesenhahn]

On Tue, 16 Dec 2003, Alexandre Duret-Lutz wrote:

> >>> "Bob" == Bob Friesenhahn <[EMAIL PROTECTED]> writes:
>
>  Bob> Another thing that would help reduce Makefile size is to
>  Bob> introduce synonyms for subdirectory paths.
>
> Better: LZW compression using Makefile variables.  Any taker?  

Is this your way of poking fun at my idea? :-(

Bob
==
Bob Friesenhahn
[EMAIL PROTECTED]
http://www.simplesystems.org/users/bfriesen

Re: automake -vs- huge projects

[Alexandre Duret-Lutz]

>>> "Bob" == Bob Friesenhahn <[EMAIL PROTECTED]> writes:

 Bob> Another thing that would help reduce Makefile size is to
 Bob> introduce synonyms for subdirectory paths.

Better: LZW compression using Makefile variables.  Any taker?  
-- 
Alexandre Duret-Lutz

cialis is the new drug

[Vito Butcher]

Save over 80% on
New official drugs approved by FDA
Come In and Visit us

Re: automake -vs- huge projects

[Alexandre Duret-Lutz]

>>> "Tom" == Tom Tromey <[EMAIL PROTECTED]> writes:

[...]
 Tom> We have to use subdir-objects
[...]
 Tom> Also, we use a single top-level Makefile.am
[...]
 Tom> The problem is, automake generates an explicit rule for each
 Tom> compilation.  Our resulting Makefile.in is nearly 9 megabytes.  This
 Tom> is really much too large -- compare to 200K with automake 1.4.

 Tom> One idea we had for a fix is to introduce a new "gnu-make" option that
 Tom> would allow automake to generate code relying on GNU make.  Then we
 Tom> could replace all those rules with a single "%.o: %.java".

Couldn't we use the (existing) .java.o: inference rule in this
case?  Actually, is there a difference between `%.o: %.java' and
`.java.o:' beside portability?  -- I'm not asking about the
general % construction, just about this precise case where both
sides are expected to be in the same directory.

I've done some limited testing (with GNU make, BSD make,
OSF1/Tru64 Make, and HP-UX make) and .src.dest: rules appear to
work for files in subdirectory even in VPATH configurations.  I
wonder if there are Make implementations where this does not
work.

So the simplest part of the fix would be to disable the output
of explicit rule for subdirectory sources without per-target
flags when subdir-objects is used.

The other side of the coin is that dependency tracking will not
work anymore, because the dependency stuff for subdir/X.o should
go into subdir/.deps/X.Po but the default suffix rule will put
it in ./.deps/subdir/X.Po.  I don't see an easy way to fix this,
unless we add some clumsy shell computation in the suffix rules
(while this can probably be folded into depcomp when it is used,
it does not seem to fit well in fastdep rules).  Note that this
issue is unrelated to the %.o:%.java vs. .java.o: choice.

[...]

-- 
Alexandre Duret-Lutz

Autoconf 2.59 available from ftp.gnu.org

[Paul Eggert]

Autoconf 2.59 is now available from the following URLs:

ftp://ftp.gnu.org/gnu/autoconf/autoconf-2.59.tar.gz
ftp://ftp.gnu.org/gnu/autoconf/autoconf-2.59.tar.bz2

Here are the MD5 checksums:

d4d45eaa1769d45e59dcb131a4af17a0  autoconf-2.59.tar.gz
1ee40f7a676b3cfdc0e3f7cd81551b5f  autoconf-2.59.tar.bz2

There is only one major change in Autoconf 2.59:

** ac_abs_builddir etc.
  Absolute paths were actually relative in 2.58.

Autoconf 2.59 was originally released on 2003-11-12; see
.
Unfortunately it wasn't copied to ftp.gnu.org until today due to
problems that we had with the new ftp.gnu.org upload procedures.
Sorry about the delay.

Re: automake -vs- huge projects

[Bob Friesenhahn]

I would like to point out that my previous gripe about the unnecessary
translation of Automake target names (particularly when the target
includes a path), and the subsequent proposal to add user-level target
synonyms, dovetails nicely with the need to reduce the Makefile size.
The synonyms can help eliminate the size increase caused by the
directory path itself.  For typical subdirectory path lengths, this
could result in a 40% Makefile size reduction.

Bob
==
Bob Friesenhahn
[EMAIL PROTECTED]
http://www.simplesystems.org/users/bfriesen

Re: automake -vs- huge projects

[Bob Friesenhahn]

Another thing that would help reduce Makefile size is to introduce
synonyms for subdirectory paths.  For example,

  java/awt/print/$(DEPDIR)/

could be reduced to $(am_f123).  This would of course obfusticate the
Makefile, so it should be made optional, or allow the user to specify
part of the name so it is still comprehensible.

Bob
==
Bob Friesenhahn
[EMAIL PROTECTED]
http://www.simplesystems.org/users/bfriesen

Re: automake -vs- huge projects

[Thomas Fitzsimmons]

On Tue, 16 Dec 2003 09:49:44 -0700, Tom Tromey wrote:

> Tom Fitzsimmons (CCd) has been working on upgrading libgcj to use
> newer auto* tools.  This has gone swimmingly, except one problem with
> automake.
> 
> A little background.  libgcj is pretty big.  It has 2,243 ".java"
> files at the moment.  Previously it has been using its own slightly
> hacked automake 1.4.  It used to use its own "%" rules to handle
> compiling .java (since 1.4 couldn't do this).  It is part of GCC,
> which recently decided as a project that requiring GNU make is ok.
> 
> 
> We have to use subdir-objects, both because nobody wants 2000 .o
> files in "." and because we have unavoidable basename clashes between
> .java files.
> 
> Also, we use a single top-level Makefile.am, as it is way more
> convenient.
> 
> The problem is, automake generates an explicit rule for each
> compilation.  Our resulting Makefile.in is nearly 9 megabytes.  This
> is really much too large -- compare to 200K with automake 1.4.
> 

One specifc problem I noticed is that automake is generating three
explicit rules for each target.  For example, for one .java file, these
rules are generated:

java/awt/print/Pageable.o: java/awt/print/Pageable.java
@am__fastdepGCJ_TRUE@   if $(GCJ) $(AM_GCJFLAGS) $(GCJFLAGS) -MT 
java/awt/print/Pageable.o -MD -MP -MF "java/awt/print/$(DEPDIR)/Pageable.Tpo" \
@am__fastdepGCJ_TRUE@ -c -o java/awt/print/Pageable.o `test -f 
'java/awt/print/Pageable.java' || echo '$(srcdir)/'`java/awt/print/Pageable.java; \
@am__fastdepGCJ_TRUE@   then mv -f "java/awt/print/$(DEPDIR)/Pageable.Tpo" 
"java/awt/print/$(DEPDIR)/Pageable.Po"; \
@am__fastdepGCJ_TRUE@   else rm -f "java/awt/print/$(DEPDIR)/Pageable.Tpo"; exit 1; \
@am__fastdepGCJ_TRUE@   fi
@AMDEP_TRUE@@am__fastdepGCJ_FALSE@  source='java/awt/print/Pageable.java' 
object='java/awt/print/Pageable.o' libtool=no @AMDEPBACKSLASH@
@AMDEP_TRUE@@am__fastdepGCJ_FALSE@  depfile='java/awt/print/$(DEPDIR)/Pageable.Po' 
tmpdepfile='java/awt/print/$(DEPDIR)/Pageable.TPo' @AMDEPBACKSLASH@
@AMDEP_TRUE@@am__fastdepGCJ_FALSE@  $(GCJDEPMODE) $(depcomp) @AMDEPBACKSLASH@
@am__fastdepGCJ_FALSE@  $(GCJ) $(AM_GCJFLAGS) $(GCJFLAGS) -c -o 
java/awt/print/Pageable.o `test -f 'java/awt/print/Pageable.java' || echo 
'$(srcdir)/'`java/awt/print/Pageable.java

java/awt/print/Pageable.obj: java/awt/print/Pageable.java
@am__fastdepGCJ_TRUE@   if $(GCJ) $(AM_GCJFLAGS) $(GCJFLAGS) -MT 
java/awt/print/Pageable.obj -MD -MP -MF "java/awt/print/$(DEPDIR)/Pageable.Tpo" \
@am__fastdepGCJ_TRUE@ -c -o java/awt/print/Pageable.obj `if test -f 
'java/awt/print/Pageable.java'; then $(CYGPATH_W) 'java/awt/print/Pageable.java'; else 
$(CYGPATH_W) '$(srcdir)/java/awt/print/Pageable.java'; fi`; \
@am__fastdepGCJ_TRUE@   then mv -f "java/awt/print/$(DEPDIR)/Pageable.Tpo" 
"java/awt/print/$(DEPDIR)/Pageable.Po"; \
@am__fastdepGCJ_TRUE@   else rm -f "java/awt/print/$(DEPDIR)/Pageable.Tpo"; exit 1; \
@am__fastdepGCJ_TRUE@   fi
@AMDEP_TRUE@@am__fastdepGCJ_FALSE@  source='java/awt/print/Pageable.java' 
object='java/awt/print/Pageable.obj' libtool=no @AMDEPBACKSLASH@
@AMDEP_TRUE@@am__fastdepGCJ_FALSE@  depfile='java/awt/print/$(DEPDIR)/Pageable.Po' 
tmpdepfile='java/awt/print/$(DEPDIR)/Pageable.TPo' @AMDEPBACKSLASH@
@AMDEP_TRUE@@am__fastdepGCJ_FALSE@  $(GCJDEPMODE) $(depcomp) @AMDEPBACKSLASH@
@am__fastdepGCJ_FALSE@  $(GCJ) $(AM_GCJFLAGS) $(GCJFLAGS) -c -o 
java/awt/print/Pageable.obj `if test -f 'java/awt/print/Pageable.java'; then 
$(CYGPATH_W) 'java/awt/print/Pageable.java'; else $(CYGPATH_W) 
'$(srcdir)/java/awt/print/Pageable.java'; fi`

java/awt/print/Pageable.lo: java/awt/print/Pageable.java
@am__fastdepGCJ_TRUE@   if $(LIBTOOL) --mode=compile $(GCJ) $(AM_GCJFLAGS) $(GCJFLAGS) 
-MT java/awt/print/Pageable.lo -MD -MP -MF "java/awt/print/$(DEPDIR)/Pageable.Tpo" \
@am__fastdepGCJ_TRUE@ -c -o java/awt/print/Pageable.lo `test -f 
'java/awt/print/Pageable.java' || echo '$(srcdir)/'`java/awt/print/Pageable.java; \
@am__fastdepGCJ_TRUE@   then mv -f "java/awt/print/$(DEPDIR)/Pageable.Tpo" 
"java/awt/print/$(DEPDIR)/Pageable.Plo"; \
@am__fastdepGCJ_TRUE@   else rm -f "java/awt/print/$(DEPDIR)/Pageable.Tpo"; exit 1; \
@am__fastdepGCJ_TRUE@   fi
@AMDEP_TRUE@@am__fastdepGCJ_FALSE@  source='java/awt/print/Pageable.java' 
object='java/awt/print/Pageable.lo' libtool=yes @AMDEPBACKSLASH@
@AMDEP_TRUE@@am__fastdepGCJ_FALSE@  
depfile='java/awt/print/$(DEPDIR)/Pageable.Plo' 
tmpdepfile='java/awt/print/$(DEPDIR)/Pageable.TPlo' @AMDEPBACKSLASH@
@AMDEP_TRUE@@am__fastdepGCJ_FALSE@  $(GCJDEPMODE) $(depcomp) @AMDEPBACKSLASH@
@am__fastdepGCJ_FALSE@  $(LIBTOOL) --mode=compile $(GCJ) $(AM_GCJFLAGS) $(GCJFLAGS) -c 
-o java/awt/print/Pageable.lo `test -f 'java/awt/print/Pageable.java' || echo 
'$(srcdir)/'`java/awt/print/Pageable.java

I would only expect rules for .lo files, since we are generating libtool
libraries.  Getting rid of the other targets would cut out *a lot* of the
bulk.

Tom

Re: automake -vs- huge projects

[Bob Friesenhahn]

On 16 Dec 2003, Paul D. Smith wrote:

> %% Bob Friesenhahn <[EMAIL PROTECTED]> writes:
>
>   bf> Per-subdirectory rules and definitions can be added in order to
>   bf> significantly reduce the amount of redundant code, and to
>   bf> re-enable the capability to usefully override parts of the default
>   bf> Makefile.in.
>
> Not if you want to continue to generate portable makefiles.
>
> There is no way in POSIX make (for example) to generate a target in a
> subdirectory using a suffix rule.

Bummer.  I wasn't sure about that.  At the very least, an effort could
be made to decrease the Makefile size by making maximum use of macro
substitutions.

Bob
==
Bob Friesenhahn
[EMAIL PROTECTED]
http://www.simplesystems.org/users/bfriesen

Re: automake -vs- huge projects

[Paul D. Smith]

%% Bob Friesenhahn <[EMAIL PROTECTED]> writes:

  bf> Per-subdirectory rules and definitions can be added in order to
  bf> significantly reduce the amount of redundant code, and to
  bf> re-enable the capability to usefully override parts of the default
  bf> Makefile.in.

Not if you want to continue to generate portable makefiles.

There is no way in POSIX make (for example) to generate a target in a
subdirectory using a suffix rule.

-- 
---
 Paul D. Smith <[EMAIL PROTECTED]>  Find some GNU make tips at:
 http://www.gnu.org  http://make.paulandlesley.org
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist

Re: automake -vs- huge projects

[Bob Friesenhahn]

On 16 Dec 2003, Tom Tromey wrote:
>
> The problem is, automake generates an explicit rule for each
> compilation.  Our resulting Makefile.in is nearly 9 megabytes.  This
> is really much too large -- compare to 200K with automake 1.4.

I have observed the same problem myself.  It is not language
dependent.  At some point (perhaps when subdirectory builds were
introduced) Automake diverged from using common rules, and started to
insert per-target rules. It seems that a bit more smarts can be used
to get close to automake 1.4 efficiency.  Per-subdirectory rules and
definitions can be added in order to significantly reduce the amount
of redundant code, and to re-enable the capability to usefully
override parts of the default Makefile.in.

> One idea we had for a fix is to introduce a new "gnu-make" option that
> would allow automake to generate code relying on GNU make.  Then we
> could replace all those rules with a single "%.o: %.java".

This doesn't seem to be necessary.

Bob
==
Bob Friesenhahn
[EMAIL PROTECTED]
http://www.simplesystems.org/users/bfriesen

automake -vs- huge projects

[Tom Tromey]

Tom Fitzsimmons (CCd) has been working on upgrading libgcj to use
newer auto* tools.  This has gone swimmingly, except one problem with
automake.

A little background.  libgcj is pretty big.  It has 2,243 ".java"
files at the moment.  Previously it has been using its own slightly
hacked automake 1.4.  It used to use its own "%" rules to handle
compiling .java (since 1.4 couldn't do this).  It is part of GCC,
which recently decided as a project that requiring GNU make is ok.


We have to use subdir-objects, both because nobody wants 2000 .o
files in "." and because we have unavoidable basename clashes between
.java files.

Also, we use a single top-level Makefile.am, as it is way more
convenient.

The problem is, automake generates an explicit rule for each
compilation.  Our resulting Makefile.in is nearly 9 megabytes.  This
is really much too large -- compare to 200K with automake 1.4.


One idea we had for a fix is to introduce a new "gnu-make" option that
would allow automake to generate code relying on GNU make.  Then we
could replace all those rules with a single "%.o: %.java".

I thought this probably wouldn't be a popular idea, but figured I'd
send it past you just in case.  We definitely don't want to have a
local fork of automake any more -- it is too much of a pain, and GCC
as a whole is trying to update and standardize on one set of tools for
the whole tree.

Any other ideas for how to fix this problem?

In absence of a real fix, another option for us is to just hand-write
our Makefile.  I'd really rather not do that, either.  Though I
suppose if we start from Automake's output it won't be too awful -- we
can still preserve dependency tracking and the like.  The long term
maintenance on this is likely to be hard though.

Tom