Re: allowing users to add source files without rerunning the autotools?

[Stefano Lattarini]

On 01/20/2012 09:10 PM, Paul Smith wrote:
> On Fri, 2012-01-20 at 20:35 +0100, Stefano Lattarini wrote:
 What automake does for source files it knows about is just "include
 $(DEPDIR)/srcfile.Po" (apparently "include" is considered portable
 make?).
>>>
>>> It's not considered portable make.
>>>
>> Still, it's worth noting that it works with every make implementation
>> I usually test automake with:
> 
> FYI, I believe that POSIX has approved adding an "include" command to
> the make standard.  I can't remember for sure.
>
It seems this is the case:

  

An excerpt:

  ``If the word include appears at the beginning of a line and is
followed by one or more  characters, the string formed
by the remainder of the line shall be processed as follows to
produce a pathname ... [SNIP] ... The contents of the file
specified by the pathname shall be read and processed as if they
appeared in the makefile in place of the include line.''

> I know that POSIX standard != portable, necessarily, but it's a start.
> 

Regards,
  Stefano

Re: allowing users to add source files without rerunning the autotools?

[NightStrike]

On Wed, Jan 18, 2012 at 8:49 PM, Miles Bader  wrote:
> For cleaning non-automake-handled stuff, you can add a "clean-local:"
> rule (and "maintainer-clean-local:" etc) that does cleaning however
> you want.  The automake-generated clean rule will depend on it, but
> you control what it does.  For packaging, you can use the "dist-hook:"
> rule.

CLEANFILES = ...
EXTRA_DIST = 

I even put wildcards in there:

EXTRA_DIST = dirForUserContrib/*

Re: allowing users to add source files without rerunning the autotools?

[Nick Bowler]

Hi Stefano,

On 2012-01-20 20:36 +0100, Stefano Lattarini wrote:
> On 01/19/2012 09:27 PM, Nick Bowler wrote:
> >
> > [SNIP]
> >
> > Interestingly, if you actually stick a line exactly like the above into
> > your Makefile.am, Automake will actually do "The Right Thing™" and
> > creates the .Po stub as if you had actually specified the source file
> > normally.  Presumably you'd be relying on totally unsupported internal
> > behaviour of Automake in this case, though. :)
> > 
> If you feel like writing a complete example of how to (ab)use this feature
> in practice, so that a user can add source files at make time and have
> automatic dependency tracking work nonetheless, that would make a great
> addition to the automake manual and testsuite.  Otherwise I'll try to cook
> up an example myself in the next days (no promise though).

When I posted that, it was just an interesting side note.  I didn't
think that you could use this to help with the "files not known until
build time" problem.  But now that you mention it, I think I see how it
might be used for this purpose.

I can certainly try to put something together, but I doubt I'll have
time this weekend to do it.

Cheers,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

Re: allowing users to add source files without rerunning the autotools?

[Nick Bowler]

On 2012-01-20 20:35 +0100, Stefano Lattarini wrote:
> Just a minor FYI and a minor question ...
> 
> On 01/19/2012 09:27 PM, Nick Bowler wrote:
> >
> > [SNIP]
> >
> >> What automake does for source files it knows about is just "include
> >> $(DEPDIR)/srcfile.Po" (apparently "include" is considered portable
> >> make?).
> > 
> > It's not considered portable make.
> >
> Still, it's worth noting that it works with every make implementation
> I usually test automake with:
> 
>   - GNU make (obviously)
>   - FreeBSD 8.2 make (I'm almost sure this works with 7.x too, at least)
>   - OpenBSD 4.6
>   - NetBSD 5.1
>   - Solaris 10 XPG 4 make
>   - Solaris 10 CCS make
>   - Sun Distributed Make 7.8 (2007/07/19)

Interesting point.  It works with dmake as well.  Perhaps it can be
someday "included" in the list of effectively portable make features
(like recursive macro expansion).

> Does anyone know whether it works also with the vendor makes for AIX,
> OSF1/Tru64, IRIX, HP-UX?

I'm sure someone knows, but alas that person is not me. :)

Cheers,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

Re: allowing users to add source files without rerunning the autotools?

[Paul Smith]

On Fri, 2012-01-20 at 20:35 +0100, Stefano Lattarini wrote:
> >> What automake does for source files it knows about is just "include
> >> $(DEPDIR)/srcfile.Po" (apparently "include" is considered portable
> >> make?).
> > 
> > It's not considered portable make.
> >
> Still, it's worth noting that it works with every make implementation
> I usually test automake with:

FYI, I believe that POSIX has approved adding an "include" command to
the make standard.  I can't remember for sure.

I know that POSIX standard != portable, necessarily, but it's a start.

Re: allowing users to add source files without rerunning the autotools?

[Stefano Lattarini]

Hi Nick.

On 01/19/2012 09:27 PM, Nick Bowler wrote:
>
> [SNIP]
>
> Interestingly, if you actually stick a line exactly like the above into
> your Makefile.am, Automake will actually do "The Right Thing™" and
> creates the .Po stub as if you had actually specified the source file
> normally.  Presumably you'd be relying on totally unsupported internal
> behaviour of Automake in this case, though. :)
> 
If you feel like writing a complete example of how to (ab)use this feature
in practice, so that a user can add source files at make time and have
automatic dependency tracking work nonetheless, that would make a great
addition to the automake manual and testsuite.  Otherwise I'll try to cook
up an example myself in the next days (no promise though).

Best regards,
  Stefano

Re: allowing users to add source files without rerunning the autotools?

[Stefano Lattarini]

Just a minor FYI and a minor question ...

On 01/19/2012 09:27 PM, Nick Bowler wrote:
>
> [SNIP]
>
>> What automake does for source files it knows about is just "include
>> $(DEPDIR)/srcfile.Po" (apparently "include" is considered portable
>> make?).
> 
> It's not considered portable make.
>
Still, it's worth noting that it works with every make implementation
I usually test automake with:

  - GNU make (obviously)
  - FreeBSD 8.2 make (I'm almost sure this works with 7.x too, at least)
  - OpenBSD 4.6
  - NetBSD 5.1
  - Solaris 10 XPG 4 make
  - Solaris 10 CCS make
  - Sun Distributed Make 7.8 (2007/07/19)

So maybe it is "almost portable" today.  Does anyone know whether it works also
with the vendor makes for AIX, OSF1/Tru64, IRIX, HP-UX?

Regards,
  Stefano

Re: allowing users to add source files without rerunning the autotools?

[Bob Friesenhahn]

On Fri, 20 Jan 2012, Miles Bader wrote:

This sort of thing is already well supported by Automake via Makefile
includes.  But it does require that automake be executed again.


Right, my intent is something that _doesn't_ require that, which can
be handled entirely by the configure script.

[That's more or less the whole point of this thread, right?]


Yes, exactly.

Since there is now a GNU-make dependent fork of Automake, the ability 
to adjust the file list without re-running Automake is likely best 
addressed in that fork.  GNU-make is quite good at this sort of thing.


It remains to be decided if Automake should just pick up '*.c' like 
some people ask for.  I am against that flimsy approach, but a 
hard-coded list in a GNU-make include file seems fine to me.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/

Re: allowing users to add source files without rerunning the autotools?

[Bill Sacks]

Thank you all for your very helpful thoughts over the last day, especially to 
Miles for the example -- which looks quite promising! -- and Stefano for all 
your thoughts. I haven't had time to parse everything in detail yet, but I 
wanted to reply to let you know how much I appreciate your help!

Just to clarify / confirm a couple of points that have been raised:

>> This sort of thing is already well supported by Automake via Makefile
>> includes.  But it does require that automake be executed again.
> 
> Right, my intent is something that _doesn't_ require that, which can
> be handled entirely by the configure script.
> 
> [That's more or less the whole point of this thread, right?]

Right: we're trying to find a solution that does not require users to run 
automake / autoconf / etc.

> with this rule:
> 
>  $(CC) -c $(CFLAGS) -DIM_STUUPD=1 $<
> 
> you are losing some important features offered by automake -- most
> notably, the automatic dependency tracking and the configurable verbosity
> specification (silent-rules).

The loss of dependency tracking is not a big deal for us: most of our code is 
Fortran 90, for which automake doesn't generate dependencies anyway, and we can 
continue to use our own tool for building dependency lists.

Finally, to confirm: yes, we are assuming gnu make already, so introducing more 
gnu make-specific features isn't too bad.

This all makes me think that Miles's solution might work for us, but I'll have 
to look at it more closely in the next couple of days.

Thanks again,
Bill


On Jan 19, 2012, at 7:41 PM, Miles Bader wrote:

> 2012/1/20 Bob Friesenhahn :
>>> One interesting thing would be to add some simple automake feature to
>>> allow specifying this stuff explicitly via some interface
>>> (e.g. AM_EXTRA_SOURCE_FILES([blah.c barf.h]) in configure, or something
>>> like that).  That might make it possible to really do this stuff
>>> correctly without too much work for the developer, by letting him take
>>> advantage of some of the lower-level automake machinery.
>> 
>> This sort of thing is already well supported by Automake via Makefile
>> includes.  But it does require that automake be executed again.
> 
> Right, my intent is something that _doesn't_ require that, which can
> be handled entirely by the configure script.
> 
> [That's more or less the whole point of this thread, right?]
> 
> -miles
> 
> -- 
> Cat is power.  Cat is peace.
> 

Re: allowing users to add source files without rerunning the autotools?

[Miles Bader]

2012/1/20 Bob Friesenhahn :
>> One interesting thing would be to add some simple automake feature to
>> allow specifying this stuff explicitly via some interface
>> (e.g. AM_EXTRA_SOURCE_FILES([blah.c barf.h]) in configure, or something
>> like that).  That might make it possible to really do this stuff
>> correctly without too much work for the developer, by letting him take
>> advantage of some of the lower-level automake machinery.
>
> This sort of thing is already well supported by Automake via Makefile
> includes.  But it does require that automake be executed again.

Right, my intent is something that _doesn't_ require that, which can
be handled entirely by the configure script.

[That's more or less the whole point of this thread, right?]

-miles

-- 
Cat is power.  Cat is peace.

Re: allowing users to add source files without rerunning the autotools?

[Bob Friesenhahn]

On Fri, 20 Jan 2012, Miles Bader wrote:


One interesting thing would be to add some simple automake feature to
allow specifying this stuff explicitly via some interface
(e.g. AM_EXTRA_SOURCE_FILES([blah.c barf.h]) in configure, or something
like that).  That might make it possible to really do this stuff
correctly without too much work for the developer, by letting him take
advantage of some of the lower-level automake machinery.


This sort of thing is already well supported by Automake via Makefile 
includes.  But it does require that automake be executed again.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/

Re: allowing users to add source files without rerunning the autotools?

[Miles Bader]

Nick Bowler  writes:
> Interestingly, if you actually stick a line exactly like the above into
> your Makefile.am, Automake will actually do "The Right Thing™" and
> creates the .Po stub as if you had actually specified the source file
> normally.  Presumably you'd be relying on totally unsupported internal
> behaviour of Automake in this case, though. :)

Heh. :)

Another thing is that if the project is happy with discovering out the
"extra" source files at configure-time (as opposed to make-time like my
example did), things probably become even easier, as the configure file
can do more stuff.

One interesting thing would be to add some simple automake feature to
allow specifying this stuff explicitly via some interface
(e.g. AM_EXTRA_SOURCE_FILES([blah.c barf.h]) in configure, or something
like that).  That might make it possible to really do this stuff
correctly without too much work for the developer, by letting him take
advantage of some of the lower-level automake machinery.

Thanks,

-Miles

-- 
"1971 pickup truck; will trade for guns"

Re: allowing users to add source files without rerunning the autotools?

[Nick Bowler]

On 2012-01-20 05:12 +0900, Miles Bader wrote:
> Stefano Lattarini  writes:
> > Still, things are not as easy as it would appear from your sample project.
> > For example, with this rule:
> >
> >   $(CC) -c $(CFLAGS) -DIM_STUUPD=1 $<
> >
> > you are losing some important features offered by automake -- most
> > notably, the automatic dependency tracking and the configurable verbosity
> > specification (silent-rules).  Which might be OK in some circumstances,
> > but unacceptable in others.
> 
> Actually, although I provided my own compilation rule, the default rule
> works too, if the user considers that acceptable (compiler flags the
> same etc).  You need at least one "automake-controlled" source-file of
> the same type (.c etc) for automake to generate it, but that's probably
> a safe bet for most projects. :)
> 
> Since the default rule generates the dependency information into
> $(DEPDIR)/*.Po as Makefile fragments, one need only figure out some
> appropriate way to include fragments into the Makefile for the "extra"
> source files.
> 
> What automake does for source files it knows about is just "include
> $(DEPDIR)/srcfile.Po" (apparently "include" is considered portable
> make?).

It's not considered portable make.  Automake's dependency tracking
requires a mechanism to include makefile fragments to work, but this is
not required to build the package.  The lines generated by Automake look
something like this:

  @AMDEP_TRUE@@am__include@ @am__quote@$(DEPDIR)/foo.Po@am__quote@

Note all the configure-substituted variables there.  In particular, if
configure cannot figure out how to make dependency tracking work, then
it will disable it.  In that case, @AMDEP_TRUE@ will be substituted with
a # and the line will be ignored by make.

> The include directive apparently understands wildcards, but is
> a little tricky to use in the same way, because it fails for
> non-existant files (automake arranges to make sure the .Po files always
> exist by just sticking in code to explicitly create them all over the
> place, which is probably too annoying for this sort of user case).

Interestingly, if you actually stick a line exactly like the above into
your Makefile.am, Automake will actually do "The Right Thing™" and
creates the .Po stub as if you had actually specified the source file
normally.  Presumably you'd be relying on totally unsupported internal
behaviour of Automake in this case, though. :)

Cheers,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

Re: allowing users to add source files without rerunning the autotools?

[Miles Bader]

Stefano Lattarini  writes:
> Still, things are not as easy as it would appear from your sample project.
> For example, with this rule:
>
>   $(CC) -c $(CFLAGS) -DIM_STUUPD=1 $<
>
> you are losing some important features offered by automake -- most
> notably, the automatic dependency tracking and the configurable verbosity
> specification (silent-rules).  Which might be OK in some circumstances,
> but unacceptable in others.

Actually, although I provided my own compilation rule, the default rule
works too, if the user considers that acceptable (compiler flags the
same etc).  You need at least one "automake-controlled" source-file of
the same type (.c etc) for automake to generate it, but that's probably
a safe bet for most projects. :)

Since the default rule generates the dependency information into
$(DEPDIR)/*.Po as Makefile fragments, one need only figure out some
appropriate way to include fragments into the Makefile for the "extra"
source files.

What automake does for source files it knows about is just "include
$(DEPDIR)/srcfile.Po" (apparently "include" is considered portable
make?).  The include directive apparently understands wildcards, but is
a little tricky to use in the same way, because it fails for
non-existant files (automake arranges to make sure the .Po files always
exist by just sticking in code to explicitly create them all over the
place, which is probably too annoying for this sort of user case).

If GNU-make-specific features are OK, then one could just use
"-include", e.g. in my example "-include $(DEPDIR)/stuupd-*.Po".
Alternatively, a "dummy" .Po file could be created by configure that
matches the above include pattern (and so prevents an error until some
real .Po files exist to match it).

Thanks,

-Miles

-- 
Carefully crafted initial estimates reward you not only with
reduced computational effort, but also with understanding and
increased self-esteem. -- Numerical methods in C,
  Chapter 9. "Root Finding and Nonlinear Sets of Equations"

Re: allowing users to add source files without rerunning the autotools?

[Stefano Lattarini]

Hi Miles, thanks for chiming in.

On 01/19/2012 02:49 AM, Miles Bader wrote:
> 2012/1/19 Stefano Lattarini :
>>> I assume you're referring to the section describing why automake doesn't
>>> support wildcards, which I have seen. I was hoping this meant that, "to
>>> the extent that you list source files, they need to be listed explicitly".
>>>
>> No, it means "you have to list all your source files, period." :-(  Sorry.
> 
> It's not true though
>
>> Here is an excerpt from the manual that should make this clean (from the
>> paragraph 8.3.1 "Conditional compilation of sources"):
>>
>>  Automake must know all the source files that could possibly go into a
>>  program, even if not all the files are built in every circumstance.
>>
>>> But your statement supports the alternative, and more likely, 
>>> interpretation,
>>> which is that automake has to know about all source files in order to work
>>> correctly.
>>>
>> Basically, this is the correct interpretation.
> 
> It's not really.
> 
> Automake needs to know about everything that _it interacts with
> directly_ (for obvious reasons).
>
Well put; this was what I meant, but I failed to communicate it correctly.
Thanks for filling the gap.

>  Automake does a lot of stuff --
> building, cleaning, packaging, etc -- so in most cases that means it's
> very desirable to just tell it about as much as possible, but if you
> have other stuff, you can add your own rules to deal with them, as
> long as there's an automake-known intermediary, e.g. a library: the
> name of the library is known to automake but the sources used to build
> it are not.  It works fine, but of course you don't get any of the
> automake conveniences for those things (probably the most important
> being dependency generation, but I guess fortran doesn't have that
> even for automake-handled stuff!).
> 
> For cleaning non-automake-handled stuff, you can add a "clean-local:"
> rule (and "maintainer-clean-local:" etc) that does cleaning however
> you want.  The automake-generated clean rule will depend on it, but
> you control what it does.  For packaging, you can use the "dist-hook:"
> rule.
> 
> I do this in my own project to embed a Lua distribution:  my program
> is 95% "automake handled" but depends on "liblua.a", and I provide
> olde-style make rules in Makefile.am to build liblua.a, and use
> clean-local etc to clean up after this, and dist-hook to stick the
> library sources into my tarballs.  It works like a champ, and indeed
> is quite convenient:  automake takes care of 99% of the disgusting
> details, and I only have to add a little make-style grot to handle
> this one case.
> 
> I've attached a sample project (the tarball generated using its own
> "make dist") that has _no_ automake-known source files (..._SOURCES is
> empty), and the real source ("stuupd-main.c") is discovered at
> build-time using $(wild stuupd-*.c); obviously in a real program
> ..._SOURCES would probably have much more stuff in it.  $(wild) is of
> course a GNU make feature,
>
In this particular case, the OP is already assuming GNU make, so
using its specific features in our example is OK I think.

(BTW, note that also $(notdir ...) and '%' pattern-rules are GNU make
specific BTW, or at least not portable to all make implementations).

> but the same thing could be accomplished
> for portable make using a make-time-generated sub-makefile ("echo ...
>> Makefile.tmp; $(MAKE) -f Makefile.tmp").
> 
> -Miles
> 
Still, things are not as easy as it would appear from your sample project.
For example, with this rule:

  $(CC) -c $(CFLAGS) -DIM_STUUPD=1 $<

you are losing some important features offered by automake -- most
notably, the automatic dependency tracking and the configurable verbosity
specification (silent-rules).  Which might be OK in some circumstances,
but unacceptable in others.

Anyway, I think a modified version of your example would make a nice addition
to the automake test suite and manual.  I'll write a patch in the next days.

Thanks!

Stefano

Re: allowing users to add source files without rerunning the autotools?

[Stefano Lattarini]

On 01/18/2012 08:09 PM, Bill Sacks wrote:
> Hi Stefano,
> 
> Thank you very much for your fast and very helpful reply. This
> looks like a clever solution.  However, I should have been more
> clear: In my prototype, Srcfiles is a static file, but in our
> actual build system, Srcfiles will be generated by a rule in the
> makefile.
>
This would complicate the implementation, but it would still be
possible to make things work anyway (you'll have to be more
careful to avoid potential circular dependencies, tough).  But
see below.

> (In particular: we use a script to find all user-created
> source files in a number of pre-defined locations.) Please correct
> me if I'm wrong, but my understanding is that your proposed solution
> can't be extended to that case. 
>
It might be extended; the problem is that your users will be required
to have the autmake and autoconf installed themselves, which is quite
against the philosophy of the autotools :-(

So scrap my suggestion.  We should try to find a better way to proceed.

> Actually, the fundamental problem that I'm trying to address here
> is just that: Can we use an autotools-based build and still allow
> for user-created source files that are unknown until build time,
> and if so, what is the most robust way to do that?
>
Ah, now that you put it this way, the problem is clearer.

First part of the answer: since autoconf doesn't require the use of
automake, you could start your conversion by introducing the use of
autoconf in the "configuration" part of your build system, keeping the
modifications to the rest of it as small as possible (in particular,
no automake at first).  This would be a useful step by itself, even if
you end up not using automake after all.

Second part of the answer: I don't really know whether it's possible to
implement what ask in a sane way (I've never thought about the problem
before).  Maybe we could cook up some hack that does what you want and
require only current automake features; maybe not.  If we can do so, that
would make a lovely addition to the automake manual.  I'll think about
the problem in the next days.

>>> And are there fundamental problems with automake not knowing all
>>> of the source
>>> files?
>>>
>> Yes; definitely yes.  See the automake manual for more details (I'd give you
>> a more precise link, but today it seems that gnu.org has "gone dark to raise
>> awareness agains SOPA and PIPA bills").
> 
> I assume you're referring to the section describing why automake doesn't
> support wildcards, which I have seen. I was hoping this meant that, "to
> the extent that you list source files, they need to be listed explicitly".
>
No, it means "you have to list all your source files, period." :-(  Sorry.

Here is an excerpt from the manual that should make this clean (from the
paragraph 8.3.1 "Conditional compilation of sources"):

  Automake must know all the source files that could possibly go into a
  program, even if not all the files are built in every circumstance.

> But your statement supports the alternative, and more likely, interpretation,
> which is that automake has to know about all source files in order to work
> correctly.
>
Basically, this is the correct interpretation.

> Thanks again,
> Bill
>

Regards, and HTH,
  Stefano

Re: allowing users to add source files without rerunning the autotools?

[Bill Sacks]

Hi Stefano,

Thank you very much for your fast and very helpful reply. This looks like a 
clever solution. However, I should have been more clear: In my prototype, 
Srcfiles is a static file, but in our actual build system, Srcfiles will be 
generated by a rule in the makefile. (In particular: we use a script to find 
all user-created source files in a number of pre-defined locations.) Please 
correct me if I'm wrong, but my understanding is that your proposed solution 
can't be extended to that case. 

Actually, the fundamental problem that I'm trying to address here is just that: 
Can we use an autotools-based build and still allow for user-created source 
files that are unknown until build time, and if so, what is the most robust way 
to do that?

>> And are there fundamental problems with automake not knowing all of the 
>> source
>> files?
>> 
> Yes; definitely yes.  See the automake manual for more details (I'd give you
> a more precise link, but today it seems that gnu.org has "gone dark to raise
> awareness agains SOPA and PIPA bills").

I assume you're referring to the section describing why automake doesn't 
support wildcards, which I have seen. I was hoping this meant that, "to the 
extent that you list source files, they need to be listed explicitly". But your 
statement supports the alternative, and more likely, interpretation, which is 
that automake has to know about all source files in order to work correctly.

Thanks again,
Bill

On Jan 18, 2012, at 11:03 AM, Stefano Lattarini wrote:

> Hi Bill.
> 
> On 01/18/2012 06:42 PM, Bill Sacks wrote:
>> 
>> [SNIP]
>> 
>> This would be the typical way to set up Makefile.am:
>> 
>> noinst_LIBRARIES = libjupcommon.a
>> libjupcommon_a_SOURCES = jupcommon.h print.c jupcommon2.h print2.c
>> 
>> Instead I did the following:
>> 
>> noinst_LIBRARIES = libjupcommon.a
>> libjupcommon_a_SOURCES_standard = print.c
>> libjupcommon_a_SOURCES := $(libjupcommon_a_SOURCES_standard) $(shell cat 
>> Srcfiles)
>> am_libjupcommon_a_OBJECTS := $(addsuffix .o, $(basename 
>> $(libjupcommon_a_SOURCES)))
>> 
> This is probably doomed to failure, since Automake needs to process the
> contents of libjupcommon_a_SOURCES at automake runtime.  In particular,
> any GNU-make specific syntax used in the value of libjupcommon_a_SOURCES
> won't be recognized by automake.
> 
>> where Srcfiles contains:
>> 
>> print2.c
>> 
> I assume this 'Srcfiles' file is static, and not generated by rules in your
> Makefile, right?  In this case, you could generate, from the 'Srcfiles'
> file itself, a Makefile.am fragment, that will add the extra files in there
> to $(libjupcommon_a_SOURCES):
> 
>  libjupcommon-sources.am: Srcfiles
>   echo "libjupcommon_a_SOURCES += `cat Srcfiles`" >$@
> 
> Then you can include it in your Makefile.am to extend the "default" value
> of $(libjupcommon_a_SOURCES):
> 
>  libjupcommon_a_SOURCES = print.c
>  include $(srcdir)/libjupcommon-sources.am
> 
> This should work; but has the downside that you'll also a script to
> bootstrap your initial Makefile.in, by creating an initial version of
> 'libjupcommon-sources.am'.
> 
>> However, the manual specification of am_libjupcommon_a_OBJECTS feels like
>> something I shouldn't be doing -- I'm worried that will make the build system
>> less robust. But some sort of manual creation of the OBJECTS list seemed
>> necessary since automake didn't know all of the source files.
>> 
> This was to be expected unfortunately, given the rationale above about
> Automake not grasping GNU make constructs.
> 
>> (I also left out the header files so that the OBJECTS list would be created
>> correctly, but I'm less worried about that.)
>> 
>> Is there a more robust way to achieve the same thing?
>> 
> You might give a try to my advice.  I can't assure it will work, but IMHO
> it's worth a shot.  And if something is unclear or confusing, just ask.
> 
>> And are there fundamental problems with automake not knowing all of the 
>> source
>> files?
>> 
> Yes; definitely yes.  See the automake manual for more details (I'd give you
> a more precise link, but today it seems that gnu.org has "gone dark to raise
> awareness agains SOPA and PIPA bills").
> 
> HTH,
>  Stefano

Re: allowing users to add source files without rerunning the autotools?

[Stefano Lattarini]

Hi Bill.

On 01/18/2012 06:42 PM, Bill Sacks wrote:
>
> [SNIP]
>
> This would be the typical way to set up Makefile.am:
> 
> noinst_LIBRARIES = libjupcommon.a
> libjupcommon_a_SOURCES = jupcommon.h print.c jupcommon2.h print2.c
> 
> Instead I did the following:
> 
> noinst_LIBRARIES = libjupcommon.a
> libjupcommon_a_SOURCES_standard = print.c
> libjupcommon_a_SOURCES := $(libjupcommon_a_SOURCES_standard) $(shell cat 
> Srcfiles)
> am_libjupcommon_a_OBJECTS := $(addsuffix .o, $(basename 
> $(libjupcommon_a_SOURCES)))
>
This is probably doomed to failure, since Automake needs to process the
contents of libjupcommon_a_SOURCES at automake runtime.  In particular,
any GNU-make specific syntax used in the value of libjupcommon_a_SOURCES
won't be recognized by automake.

> where Srcfiles contains:
> 
> print2.c
> 
I assume this 'Srcfiles' file is static, and not generated by rules in your
Makefile, right?  In this case, you could generate, from the 'Srcfiles'
file itself, a Makefile.am fragment, that will add the extra files in there
to $(libjupcommon_a_SOURCES):

  libjupcommon-sources.am: Srcfiles
echo "libjupcommon_a_SOURCES += `cat Srcfiles`" >$@

Then you can include it in your Makefile.am to extend the "default" value
of $(libjupcommon_a_SOURCES):

  libjupcommon_a_SOURCES = print.c
  include $(srcdir)/libjupcommon-sources.am

This should work; but has the downside that you'll also a script to
bootstrap your initial Makefile.in, by creating an initial version of
'libjupcommon-sources.am'.

> However, the manual specification of am_libjupcommon_a_OBJECTS feels like
> something I shouldn't be doing -- I'm worried that will make the build system
> less robust. But some sort of manual creation of the OBJECTS list seemed
> necessary since automake didn't know all of the source files.
>
This was to be expected unfortunately, given the rationale above about
Automake not grasping GNU make constructs.

> (I also left out the header files so that the OBJECTS list would be created
> correctly, but I'm less worried about that.)
> 
> Is there a more robust way to achieve the same thing?
>
You might give a try to my advice.  I can't assure it will work, but IMHO
it's worth a shot.  And if something is unclear or confusing, just ask.

> And are there fundamental problems with automake not knowing all of the source
> files?
>
Yes; definitely yes.  See the automake manual for more details (I'd give you
a more precise link, but today it seems that gnu.org has "gone dark to raise
awareness agains SOPA and PIPA bills").

HTH,
  Stefano

Re: allowing users to add source files without rerunning the autotools?

[Bill Sacks]

Miles: Thanks for these additional thoughts.

I have been trying out a couple of methods for achieving what I want, but am 
running into problems. I'm probably going to put this migration to the 
autotools on hold for now, unless anyone has insight into a good way to set 
this up.

I first tried a prototype where user-added source code is built in its own 
library, to keep it isolated from the standard stuff. However, I ran into 
problems getting the dependencies to be worked out correctly, because it's 
likely that some user-added source will depend on existing code, and at the 
same time, some existing code will depend on user-added source.

So then I tried a prototype where user-added source is added in the same 
library as the existing code (this is most similar to our current system). I 
got this working with a simple example (the jupiter example from Ch. 5 of John 
Calcote's autotools book), by doing the following:

This would be the typical way to set up Makefile.am:

noinst_LIBRARIES = libjupcommon.a
libjupcommon_a_SOURCES = jupcommon.h print.c jupcommon2.h print2.c

Instead I did the following:

noinst_LIBRARIES = libjupcommon.a
libjupcommon_a_SOURCES_standard = print.c
libjupcommon_a_SOURCES := $(libjupcommon_a_SOURCES_standard) $(shell cat 
Srcfiles)
am_libjupcommon_a_OBJECTS := $(addsuffix .o, $(basename 
$(libjupcommon_a_SOURCES)))

where Srcfiles contains:

print2.c

However, the manual specification of am_libjupcommon_a_OBJECTS feels like 
something I shouldn't be doing -- I'm worried that will make the build system 
less robust. But some sort of manual creation of the OBJECTS list seemed 
necessary since automake didn't know all of the source files. (I also left out 
the header files so that the OBJECTS list would be created correctly, but I'm 
less worried about that.)

Is there a more robust way to achieve the same thing? And are there fundamental 
problems with automake not knowing all of the source files?

Thanks,
Bill

On Jan 14, 2012, at 4:09 AM, Miles Bader wrote:

> Bill Sacks  writes:
>> The dependency issue that Jeff raised is not a problem for us, since
>> we have a script to determine Fortran 90 dependencies.
>> 
>> I'm not sure that it will work to have a separate library of the
>> user-added code, since we don't know ahead of time what dependencies
>> there will be between existing code and the user-added code.
>> 
>> Your other suggestions are helpful. We'll give this some more thought.
> 
> I think the main thing to remember is that most traditional make
> facilities are still available, should automake's higher-level
> constructs not suffice, and the two can usually co-exist reasonably
> well.
> 
> So whatever you did before for user-contributed stuff may well work
> perfectly fine in Makefile.am (maybe with a few tweaks).
> 
> -miles
> 
> -- 
> Cabbage, n. A familiar kitchen-garden vegetable about as large and wise as a
> man's head.

Re: allowing users to add source files without rerunning the autotools?

[Miles Bader]

Bill Sacks  writes:
> The dependency issue that Jeff raised is not a problem for us, since
> we have a script to determine Fortran 90 dependencies.
>
> I'm not sure that it will work to have a separate library of the
> user-added code, since we don't know ahead of time what dependencies
> there will be between existing code and the user-added code.
>
> Your other suggestions are helpful. We'll give this some more thought.

I think the main thing to remember is that most traditional make
facilities are still available, should automake's higher-level
constructs not suffice, and the two can usually co-exist reasonably
well.

So whatever you did before for user-contributed stuff may well work
perfectly fine in Makefile.am (maybe with a few tweaks).

-miles

-- 
Cabbage, n. A familiar kitchen-garden vegetable about as large and wise as a
man's head.

Re: allowing users to add source files without rerunning the autotools?

[Bill Sacks]

Hi Jeff & Miles,

Thank you both for your helpful replies. 

The dependency issue that Jeff raised is not a problem for us, since we have a 
script to determine Fortran 90 dependencies.

I'm not sure that it will work to have a separate library of the user-added 
code, since we don't know ahead of time what dependencies there will be between 
existing code and the user-added code.

Your other suggestions are helpful. We'll give this some more thought.

Thanks again,
Bill

On Jan 13, 2012, at 1:32 AM, Miles Bader wrote:

> "Daily, Jeff A"  writes:
>> Yes, all sources must be listed, AFAIK.  You could write your own build
>> rules and targets for these files, which would be copied by automake into
>> your eventual Makefile, but that somewhat defeats the purpose of using a
>> build tool if you're writing many of the rules yourself.
> 
> Er, what?  I think automake actually does a pretty good job of making
> custom rules painless and useful...
> 
> Even if you use some custom rules for a few files in a Makefile.am,
> automake can still do about 99% of the heavy-lifting for a project.
> 
> Typically my approach is to put all the "funny" stuff in its own
> library, and have the "normal" automake targets (99% of them) depend on
> that, with a few custom rules to dispatch the building of "libfunny.a"
> to its own Makefile or something (which can be generated separately by
> configure if necessary).
> 
> -Miles
> 
> -- 
> /\ /\
> (^.^)
> (")")
> *This is the cute kitty virus, please copy this into your sig so it can 
> spread.

Re: allowing users to add source files without rerunning the autotools?

[Miles Bader]

"Daily, Jeff A"  writes:
> Yes, all sources must be listed, AFAIK.  You could write your own build
> rules and targets for these files, which would be copied by automake into
> your eventual Makefile, but that somewhat defeats the purpose of using a
> build tool if you're writing many of the rules yourself.

Er, what?  I think automake actually does a pretty good job of making
custom rules painless and useful...

Even if you use some custom rules for a few files in a Makefile.am,
automake can still do about 99% of the heavy-lifting for a project.

Typically my approach is to put all the "funny" stuff in its own
library, and have the "normal" automake targets (99% of them) depend on
that, with a few custom rules to dispatch the building of "libfunny.a"
to its own Makefile or something (which can be generated separately by
configure if necessary).

-Miles

-- 
/\ /\
(^.^)
(")")
*This is the cute kitty virus, please copy this into your sig so it can spread.

Re: allowing users to add source files without rerunning the autotools?

[Daily, Jeff A]

On 1/12/12 2:16 PM, "Bill Sacks"  wrote:

> I work on a climate model (CESM); this is mostly written in Fortran 90, with a
> bit of C, but we are starting to add some C++ code, too. Until now, we have
> used our own, custom build system, but as the build becomes more complex, we
> are thinking of migrating towards greater use of the autotools.

Note that automake does not (yet?) support Fortran 90 dependency tracking.
I have raised this issue previously but those of us interested in developing
it are lacking time to work on it.  I am a fan of the autotools, but be
advised that CMake has its own Fortran parser for generating dependencies.
Now, if dependency tracking is not desired then this is no longer an issue.

> However, one challenge we face is the blurred line between developers and
> users. Many users need to modify the source code before building the model,
> and this can include adding entirely new source files. Existing code can
> depend on the new, user-created source files, and vice versa.
> 
> We implement this ability in our current build system (which does not use any
> of the autotools) through the use of two scripts, both of which are executed
> according to rules in the makefile: The first script builds a list of all
> source files residing in a set of pre-defined directories, and the second
> script parses these source files to generate dependency lists. This system
> isn't bulletproof, but it works well for us. In particular, note that this
> means that we do not need to explicitly list the source or object files in our
> makefile, since these lists are generated at build time.
> 
> My (admittedly limited) understanding of automake is that it requires all
> source files to be listed explicitly in Makefile.am, and thus any addition to
> this list would require rerunning automake, etc.

Yes, all sources must be listed, AFAIK.  You could write your own build
rules and targets for these files, which would be copied by automake into
your eventual Makefile, but that somewhat defeats the purpose of using a
build tool if you're writing many of the rules yourself.  This issue would
be similar with other build tools, having to rerun the tools if the lists of
source files change.

You could potentially generate a makefile fragment containing your list of
sources and "include" it from your automake-generated Makefile, but you'd
need a hacky work-around since "include" is also an automake directive.  See
the following discussion:
http://old.nabble.com/ifdef-...-endif-issue-td12330117.html

> We are wondering if it will be possible for us to support the functionality of
> user-added source files in an autotools-based build, without requiring users
> to rerun the full autotools chain. Is this possible? We can assume that
> user-created source files do not change any of the options required for the
> build (e.g., there would be no new library dependencies).

What if you built a separate library representing the user code?  The
interdependencies between user code and original code would be tricky.
There's nothing wrong with having automake build the rest of the code and
have a recursive make call to a hand-rolled Makefile.

> I have read the FAQ entry on why automake doesn't support wildcards, so I'm
> wondering if there is some other workaround to achieve the functionality we
> want.
> 
> Thank you,
> Bill

I tried not to ramble here but rather offer hopefully a few useful ideas...