Re: Extending Automake to build non-file objects?

[Zé]

On 06/14/2014 01:00 PM, Conrad Dean wrote:

Hey!

I have a project where I need to generate datasets that are not stored as
flat files.  I can use  a CLI interface to generate them off of others and
the same CLI to check when datasets were last modified.  I think I want to
use automake to describe the relationships between datasets, and detect
when an upstream dataset has been generated to update the downstream ones.

Would make be a good fit for that? or is it too-closely tied to just
dealing with files?

Thanks!
Conrad


As no one replied yet, here are my 0,02€.

Automake lets you specify makefile rules, as the contents of a 
Makefile.am are copied to Makefile.in verbatim[¹].


Therefore, if you can write a makefile that generates your datasets then 
you can extend automake to build them too.


And make certainly can generate your datasets.


Hope this helps,

--
Zé

[¹] http://www.gnu.org/software/automake/manual/automake.html#Extending

Extending Automake to build non-file objects?

[Conrad Dean]

Hey!

I have a project where I need to generate datasets that are not stored as
flat files.  I can use  a CLI interface to generate them off of others and
the same CLI to check when datasets were last modified.  I think I want to
use automake to describe the relationships between datasets, and detect
when an upstream dataset has been generated to update the downstream ones.

Would make be a good fit for that? or is it too-closely tied to just
dealing with files?

Thanks!
Conrad

Re: Extending automake rules

[Miguel Guedes]

Hi Stefano.

On 11/11/12 09:19, Stefano Lattarini wrote:

Unfortunately, it is basically impossible to accomplish what you
want with the current Automake.  That is, most rules can only be
overridden, not extended (and those which can, can be only in a
limited, unflexible way).  You might manage to hack something
together by (ab)using Automake internals, but then of course
you'll have no guarantee such an hack will continue to work with
future Automake release.

Lack of extensibility is one of the main drawbacks of Automake,
and improving the situation consistently would require a lot of
work I fear, so I don't see anything happening in that area
anytime soon.


Thank you for your reply!

It's not a problem that Automake lacks the extensibility I was after. 
After some reading and thinking about it I've decided to use a 
combination of Autoconf, Autoheader and handcrafted Makefile.in files. 
It works beautifully and allows for effortless creation/updating of 
project builds.


The wonderful flexibility of Autotools saved the day! :)

Thanks again,

Miguel

Re: Extending automake rules

[Stefano Lattarini]

HI Miguel.

On 11/10/2012 12:46 PM, Miguel Guedes wrote:
> There's one thing I would simply love to accomplish with Automake:
> the ability to extend rules without overriding them. For instance,
> how would one print a custom message in a specific format (i.e.
> color) each time a source file is compiled and a binary linked?
> 
> For instance, in a traditional Makefile I would simply add the
> following rule:
> 
> %.o: %.c
> @echo -e "[C] `basename $<`
> @$(CC) $(CFLAGS) -c $< -o $@
> 
> $(bin): $(bin_OBJECTS)
> @echo "\033[1;32m[L] `basename $@`\033[0m"
> @LINK_LOG="/tmp/`basename $@`.link.log"; \
> [ -f $$LINK_LOG ] && rm $$LINK_LOG; \
> $(CXX) $(CXX_FLAGS) -o $@ $(EXE_OBJECTS) \
> $(LIBRARIES) 2>$$LINK_LOG ; \
> EXIT_CODE=$$?; \
> if [ ! $$? -eq 0 ]; then \
> cat "$$LINK_LOG"; \
> exit $$EXIT_CODE; \
> fi;
> 
> The above would result in something like:
> 
> [C] foo.c
> [C] bar.c
> [L] bin# in yellow
> 
> But how does one setup a Makefile.am such that compilation and
> linking rules are extended in Makefile.in and the generated
> Makefile.am still contains the normally generated compile/link
> rules logic - so dependencies, compilation and linking logic is
> kept and *not* overriden. I've had a look at the Automake hooks
> but they seem to be executed after the main rule.
> 
Unfortunately, it is basically impossible to accomplish what you
want with the current Automake.  That is, most rules can only be
overridden, not extended (and those which can, can be only in a
limited, unflexible way).  You might manage to hack something
together by (ab)using Automake internals, but then of course
you'll have no guarantee such an hack will continue to work with
future Automake release.

Lack of extensibility is one of the main drawbacks of Automake,
and improving the situation consistently would require a lot of
work I fear, so I don't see anything happening in that area
anytime soon.

Sorry,
  Stefano

Extending automake rules

[Miguel Guedes]

There's one thing I would simply love to accomplish with Automake: the 
ability to extend rules without overriding them. For instance, how would 
one print a custom message in a specific format (i.e. color) each time a 
source file is compiled and a binary linked?


For instance, in a traditional Makefile I would simply add the following 
rule:


%.o: %.c
@echo -e "[C] `basename $<`
@$(CC) $(CFLAGS) -c $< -o $@

$(bin): $(bin_OBJECTS)
@echo "\033[1;32m[L] `basename $@`\033[0m"
@LINK_LOG="/tmp/`basename $@`.link.log"; \
[ -f $$LINK_LOG ] && rm $$LINK_LOG; \
$(CXX) $(CXX_FLAGS) -o $@ $(EXE_OBJECTS) \
$(LIBRARIES) 2>$$LINK_LOG ; \
EXIT_CODE=$$?; \
if [ ! $$? -eq 0 ]; then \
cat "$$LINK_LOG"; \
exit $$EXIT_CODE; \
fi;

The above would result in something like:

[C] foo.c
[C] bar.c
[L] bin# in yellow

But how does one setup a Makefile.am such that compilation and linking 
rules are extended in Makefile.in and the generated Makefile.am still 
contains the normally generated compile/link rules logic - so 
dependencies, compilation and linking logic is kept and *not* overriden. 
I've had a look at the Automake hooks but they seem to be executed after 
the main rule.

Re: extending automake targets

[Stefano Lattarini]

On Monday 11 October 2010, Ralf Wildenhues wrote:
> Hello Stefano,
> 
> * Stefano Lattarini wrote on Fri, Oct 08, 2010 at 11:56:55AM CEST:
> > On Friday 08 October 2010, Ralf Wildenhues wrote:
> > > Can we also keep its discussion and its eventual addition separate, for
> > > both user-defined and automake-defined recursive targets, please?
> 
> > Not really, because,  *for user targets only*, I've already implemented
> > that feature (and added tests for it) in my patch (it was easier than
> > not implementing it, and its implementation requires no extra forks,
> > relying exclusively on the make dependency graph).
> 
> While that may be a good reason for discussing both features at the same
> time, I don't think it is a good reason for or against the feature.
> Rather, it's a good reason for doing design first, then doing tests
> and implementation.
> 
> > > I don't yet see much chance for regression yet, but the feature does
> > > violate the "given Makefile.am, let Makefile.in be as simple as
> > > possible" development guide line.  It would not be easy to draw a
> > > consistent line:
> 
> > True; but as the pythonistas say, "practicality beats purity".
> 
> Logical semantics are practical, too.  See how users complain about the
> complexity of Automake due to many special-cases and such.
IMHO the current implementation of the `*-local' targets is more complex
and special-cased than the one proposed by me.

> Being able to remember semantics is very practical.  So no, I will
> reject the special-case feature
>
>   "with recursive targets @var{foo} provided by the user,
>@command{automake} does not require the @code{foo-local} target
>to be visible to @command{automake}, but with most or all other
>Automake features it does".
> 
> for now.
OK, and even if I disagree with you on this, I will amend my patch along
these lines (if it's not too difficult).

> I also think that it is at odds with conditionals.
Why?
 
> > > what about *-hook extensions defined only in GNUmakefile?
> 
> > I think we should support them too in the future, if it's not too
> > much work.
> 
> See below.
> 
> > > What about things like BUILT_SOURCES?
> 
> > Well, assignments of special variables in Makefile.am files have always
> > caused automake to trigger injections of precanned Makefile fragments
> > into the generated Makefile.in (plus other tweakings); this is the very
> > basic tenet and working principle of Automake.  So it's fair to expect
> > that such definitions *must* be in Makefile.am (and not in a wrapping
> > GNUmakefile) for the automake-generated rules to behave correctly.
> > This is where I'd draw the line (theoretically, at least; some things
> > that should be theoretically possible might not be pratically possible
> > for implementation or historical reasons, but that's a wholly different
> > matter).
> 
> This is a good argument.
Well, it's basically taken from the automake manual ;-)

> > > Either of those would trigger extra forks,
> 
> > Anyway, many automake-generated rules are already so fork-ridden that
> > the extra forks we are speaking about here wouldn't make any noticeable
> > difference -- but would, on the other hand, provide us with more
> > flexibility.
> 
> Sorry, but I reject this notion.  We strive to avoid extra make
> recursion if possible, because that induces a number of problems
> (with parallelism, with portability of overridden macros, with
> performance).
OK; but this might be a problem for hooks, not for `*-local' targets
(and maybe we could even rethink how hooks are implemented, to avoid
make recursion as much as possible; but this is out of the scope of
the current discussion, so let's drop it for now).
 
> > > maybe even extra make recursions with associated undesirable
> > > slowdown effects.
> 
> > This should not be the case with the `*-local' and `*-hook' targets
> 
> This is wrong.  The hook targets cause extra recursion in at least some
> cases, if not always.
OK, sorry for the ignorance; but see above.

> > (and in general, I agree that we should strive to avoid extra make
> > recursion).
> 
> > IMVHO the small increase in flexibility would be worth the very tiny
> > decrease in consistency.
> 
> I'm afraid we have to agree to disagree on this point then.  I don't
> think the primary user base Automake should cater to is users hiding
> rules from automake in GNUmakefile files, and giving up consistency
> in order to allow them to save writing three lines of Makefile.am code.
> I also don't think this use base makes up a very large part of the
> overall Automake use base.
> 
> I am two thirds through with the review of your posted 3-part patch
> series; you might want to wait with updating it until I've finished it,
> in order to avoid extra work.
OK, thanks for the tip.

As an aside, I must warn you that I'm experiencing harware problems,
so I might not be able to reply right away (or at all) to your mails
in the next days.  Sorry about the inconvenient.

Re

Re: extending automake targets

[Ralf Wildenhues]

Hello Stefano,

* Stefano Lattarini wrote on Fri, Oct 08, 2010 at 11:56:55AM CEST:
> On Friday 08 October 2010, Ralf Wildenhues wrote:
> > Can we also keep its discussion and its eventual addition separate, for
> > both user-defined and automake-defined recursive targets, please?

> Not really, because,  *for user targets only*, I've already implemented
> that feature (and added tests for it) in my patch (it was easier than
> not implementing it, and its implementation requires no extra forks,
> relying exclusively on the make dependency graph).

While that may be a good reason for discussing both features at the same
time, I don't think it is a good reason for or against the feature.
Rather, it's a good reason for doing design first, then doing tests
and implementation.

> > I don't yet see much chance for regression yet, but the feature does
> > violate the "given Makefile.am, let Makefile.in be as simple as
> > possible" development guide line.  It would not be easy to draw a
> > consistent line:

> True; but as the pythonistas say, "practicality beats purity".

Logical semantics are practical, too.  See how users complain about the
complexity of Automake due to many special-cases and such.  Being able
to remember semantics is very practical.  So no, I will reject the
special-case feature
  "with recursive targets @var{foo} provided by the user,
   @command{automake} does not require the @code{foo-local} target
   to be visible to @command{automake}, but with most or all other
   Automake features it does".

for now.  I also think that it is at odds with conditionals.

> > what about *-hook extensions defined only in GNUmakefile?

> I think we should support them too in the future, if it's not too
> much work.

See below.

> > What about things like BUILT_SOURCES?

> Well, assignments of special variables in Makefile.am files have always
> caused automake to trigger injections of precanned Makefile fragments
> into the generated Makefile.in (plus other tweakings); this is the very
> basic tenet and working principle of Automake.  So it's fair to expect
> that such definitions *must* be in Makefile.am (and not in a wrapping
> GNUmakefile) for the automake-generated rules to behave correctly.
> This is where I'd draw the line (theoretically, at least; some things
> that should be theoretically possible might not be pratically possible
> for implementation or historical reasons, but that's a wholly different
> matter).

This is a good argument.

> > Either of those would trigger extra forks,

> Anyway, many automake-generated rules are already so fork-ridden that
> the extra forks we are speaking about here wouldn't make any noticeable
> difference -- but would, on the other hand, provide us with more
> flexibility.

Sorry, but I reject this notion.  We strive to avoid extra make
recursion if possible, because that induces a number of problems
(with parallelism, with portability of overridden macros, with
performance).

> > maybe even extra make recursions with associated undesirable
> > slowdown effects.

> This should not be the case with the `*-local' and `*-hook' targets

This is wrong.  The hook targets cause extra recursion in at least some
cases, if not always.

> (and in general, I agree that we should strive to avoid extra make
> recursion).

> IMVHO the small increase in flexibility would be worth the very tiny
> decrease in consistency.

I'm afraid we have to agree to disagree on this point then.  I don't
think the primary user base Automake should cater to is users hiding
rules from automake in GNUmakefile files, and giving up consistency
in order to allow them to save writing three lines of Makefile.am code.
I also don't think this use base makes up a very large part of the
overall Automake use base.

I am two thirds through with the review of your posted 3-part patch
series; you might want to wait with updating it until I've finished it,
in order to avoid extra work.

Thanks,
Ralf

extending automake targets (was: Re: Recursive targets for the user)

[Stefano Lattarini]

On Friday 08 October 2010, Ralf Wildenhues wrote:
> Hello Stefano,
> 
> * Stefano Lattarini wrote on Fri, Oct 08, 2010 at 02:46:58AM CEST:
> > What I'd like to do is to allow the developers to extend the nonrecursive
> > part of any recursive `foo' rule (be it user-defined or automake-defined)
> > with a simple:
> > 
> >   foo-local: foo-extra-deps
> > 
> > *without causing automake to tweak the generated Makefile.in or to take
> > different codepaths*.
> 
> This is a valid feature request, but orthogonal to the feature request
> we have discussed so far.
Yes, but I hadn't understand that at first :-(
Now I do (in fact I stated that it should be done in a follow-up patch).

> Can we also keep its discussion and its eventual addition separate, for
> both user-defined and automake-defined recursive targets, please?
Not really, because,  *for user targets only*, I've already implemented
that feature (and added tests for it) in my patch (it was easier than
not implementing it, and its implementation requires no extra forks,
relying exclusively on the make dependency graph).

> I don't yet see much chance for regression yet, but the feature does
> violate the "given Makefile.am, let Makefile.in be as simple as
> possible" development guide line.  It would not be easy to draw a
> consistent line:
True; but as the pythonistas say, "practicality beats purity".
> what about *-hook extensions defined only in GNUmakefile?
I think we should support them too in the future, if it's not too
much work.

> What about things like BUILT_SOURCES?
Well, assignments of special variables in Makefile.am files have always
caused automake to trigger injections of precanned Makefile fragments
into the generated Makefile.in (plus other tweakings); this is the very
basic tenet and working principle of Automake.  So it's fair to expect
that such definitions *must* be in Makefile.am (and not in a wrapping
GNUmakefile) for the automake-generated rules to behave correctly.
This is where I'd draw the line (theoretically, at least; some things
that should be theoretically possible might not be pratically possible
for implementation or historical reasons, but that's a wholly different
matter).

> Either of those would trigger extra forks,
Anyway, many automake-generated rules are already so fork-ridden that
the extra forks we are speaking about here wouldn't make any noticeable
difference -- but would, on the other hand, provide us with more
flexibility.
> maybe even extra make recursions with associated undesirable
> slowdown effects.
This should not be the case with the `*-local' and `*-hook' targets
(and in general, I agree that we should strive to avoid extra make
recursion).
> Then what about special variables like bin_PROGRAMS?
See above.
> If we don't have any _PROGRAMS in
> Makefile.am (or included files), then we completely avoid the
> compilation machinery; OTOH, it only needs a
>   noinst_PROGRAMS =
> 
> line to enable it, just as it only needs a
>   all-local:
> 
> line to enable the -local hook for all (and such a line can easily be
> script-added to Makefile.am files in need).  IOW, the current behavior
> is very consistent, while your proposed extension is less consistent.
IMVHO the small increase in flexibility would be worth the very tiny
decrease in consistency.

Regards,
  Stefano

extending automake: implicit rules and sufixes is not enought

[Piotr Skólski]

I would like have Makefile.am like this :
> #include $(top_srcdir)/m4/autotroll.mk
>
> noinst_LIBRARIES = libcutitdata.a
>
> libcutitdata_a_SOURCES = data.qrc.cpp
> libcutitdata_a_QT_RESOURCE_TEMPLATE = data.qrc
>
> libcutitdata_a_CXXFLAGS = $(QT_CXXFLAGS) $(AM_CXXFLAGS)
> libcutitdata_a_CPPFLAGS = $(QT_CPPFLAGS) $(AM_CPPFLAGS)
>
> data_qrc_FILES =exit.png\
> unknown.png

I read documentation and i apend something like that :
> SUFIXES = .qrc.cpp
>
> .qrc.qrc.cpp: #$(data_qrc_FILES)
> $(RCC) -name `echo "$<" | sed 's/\.qrc$$//'` $< -o $@

but dont work. data.qrc.cpp can't be done because don't rule.

I copy-paste from Makefile and append:
> #copy paste and hand writed
> data.qrc.cpp: data.qrc $(data_qrc_FILES)
> $(RCC) -name `echo "$<" | sed 's/\.qrc$$//'` $< -o $@
>
> #and this if above is not enought
> libcutitdata_a-data.qrc.cpp: data.qrc $(data_qrc_FILES)
> $(RCC) -name `echo "$<" | sed 's/\.qrc$$//'` $< -o $@

Its work but:
- make clean make istclean don't work
- if i change some files resource is not rebuild
- make dist have only data.qrc.cpp

I have the same problem with generating png from svg, in uic in few my 
packages.
I have parser generator and arg parser generator i i would like to 
distribute it with *.m4 and *.mk.

How do job in proper way and dond re-invent circle (like qmake smake or 
crap-make only for one extension) ???

Re: extending automake

[Ralf Wildenhues]

Hello Bob,

* Bob Rossi wrote on Tue, Jun 03, 2008 at 05:58:59PM CEST:
> Another example I just ran across on the mingw.org site,
>windres -o resfile.o resfile.rc
>gcc -o hello hello.o resfile.o -mwindows
> Often people want to put resources into there program. They would
> greatly benefit from this feature, considering they could teach automake
> how to run these commands.

Wait, the above example is about extending automake, but not about rules
which produce multiple output files, right?

For the above, I'd just use (assuming SUFFIXES has already been set)

SUFFIXES += .rc
WINDRES = windres
.rc.$(OBJEXT):
$(WINDRES) $(WINDRESFLAGS) -o $@ $<

bin_PROGRAMS = hello
hello_SOURCES = hello.c
# probably the following needs to go in a conditional
hello_LDADD = resfile.$(OBJEXT)


Hope that helps.

Cheers,
Ralf

Re: extending automake

[Bob Rossi]

On Thu, May 01, 2008 at 03:48:25PM -0400, Bob Rossi wrote:
> On Wed, Apr 30, 2008 at 03:21:18PM +0200, Ralf Wildenhues wrote:
> > * Bob Rossi wrote on Wed, Apr 30, 2008 at 02:59:11PM CEST:
> > > You busy or thing the idea is no good?
> > 
> > Busy.  If you want to help, there are still unaddressed questions from 
> > 
> > 
> > (even if the above notation is not adopted, its semantics are still
> > close to what automake should be able to provide; so you could try
> > and report whether those semantics fit your particular problem
> > instance).
> 
> Hi Ralf,
> 
> MULTITARGETS = elc-stamp
> elc_stamp_TARGETS = $(ELCFILES)
> elc_stamp_SOURCES = $(ELFILES)
> elc_stamp_COMMAND = \
> for f in : $(elc_stamp_SOURCES); do \
>   test "$$f" != : || continue; \
>   touch "$${f}c"; \
> done
> 
> Something like this is exactly what would help me. For instance, I'm
> thinking I could do,
> 
>   MULTITARGETS = gen-foo
>   gen_foo_TARGETS = $(srcdir)/foo.xml $(abs_top_srcdir)/gen.py
>   gen_foo_SOURCES = foo.h foo.c
>   gen_foo_COMMAND = python $(abs_top_srcdir)/gen.py $(srcdir)/foo.xml
> 
> is that correct? Also, how would I tell automake that this should be
> done at BUILT_SOURCES time?
> 
> This would also work for flex/bison in the case that Mike Mueller asked
> about.

Another example I just ran across on the mingw.org site,
   windres -o resfile.o resfile.rc
   gcc -o hello hello.o resfile.o -mwindows
Often people want to put resources into there program. They would
greatly benefit from this feature, considering they could teach automake
how to run these commands.

Have you had any time to think about this?

Thanks,
Bob Rossi

Re: extending automake

[Bob Rossi]

On Wed, Apr 30, 2008 at 03:21:18PM +0200, Ralf Wildenhues wrote:
> * Bob Rossi wrote on Wed, Apr 30, 2008 at 02:59:11PM CEST:
> > You busy or thing the idea is no good?
> 
> Busy.  If you want to help, there are still unaddressed questions from 
> 
> 
> (even if the above notation is not adopted, its semantics are still
> close to what automake should be able to provide; so you could try
> and report whether those semantics fit your particular problem
> instance).

Hi Ralf,

MULTITARGETS = elc-stamp
elc_stamp_TARGETS = $(ELCFILES)
elc_stamp_SOURCES = $(ELFILES)
elc_stamp_COMMAND = \
for f in : $(elc_stamp_SOURCES); do \
  test "$$f" != : || continue; \
  touch "$${f}c"; \
done

Something like this is exactly what would help me. For instance, I'm
thinking I could do,

  MULTITARGETS = gen-foo
  gen_foo_TARGETS = $(srcdir)/foo.xml $(abs_top_srcdir)/gen.py
  gen_foo_SOURCES = foo.h foo.c
  gen_foo_COMMAND = python $(abs_top_srcdir)/gen.py $(srcdir)/foo.xml

is that correct? Also, how would I tell automake that this should be
done at BUILT_SOURCES time?

This would also work for flex/bison in the case that Mike Mueller asked
about.

Thanks,
Bob Rossi

Re: extending automake

[Ralf Wildenhues]

* Bob Rossi wrote on Wed, Apr 30, 2008 at 02:59:11PM CEST:
> You busy or thing the idea is no good?

Busy.  If you want to help, there are still unaddressed questions from 


(even if the above notation is not adopted, its semantics are still
close to what automake should be able to provide; so you could try
and report whether those semantics fit your particular problem
instance).

Cheers,
Ralf

Re: extending automake

[Bob Rossi]

On Fri, Apr 25, 2008 at 09:19:58AM -0400, Bob Rossi wrote:
> On Fri, Apr 25, 2008 at 06:54:08AM +0200, Ralf Wildenhues wrote:
> > * Bob Rossi wrote on Fri, Apr 25, 2008 at 03:41:20AM CEST:
> > > On Sat, Apr 19, 2008 at 01:22:29PM -0400, Bob Rossi wrote:
> > > > They generate files during build time, and modify BUILT_SOURCES...
> > > > 
> > > > In fact, think of the bison or flex extension (adding .y or .l files to 
> > > > the _SOURCES variable). That is just another use of this general 
> > > > functionality that I'm talking about. In some sense, it would be like me
> > > > adding foo.xml to the _SOURCES, but telling automake how to turn that
> > > > into a .c file. I want to run foo.py, whereas automake runs bison or
> > > > flex.
> > > > 
> > > > I'm sure that if this was implemented, a LOT of projects would use it.
> > > > So, is there something I can do to help implement it, with my little
> > > > experience writing make file rules?
> > > 
> > > Ping, whatever happened to this idea? You guys think it's stupid?
> > 
> > I don't see a way to formulate it sufficiently general so that it would
> > be useful for more than just a couple of projects.
> > 
> > bison and flex need special-cased handling in automake, how do you
> > propose foo.py would not?
> 
> I think Brian stated it perfectly,
> 
>   That brings up the next logical point, can anyone comment on the
> 
>   feasibility of some kind of generalized "tool X reads A and outputs Y   
> 
>   and Z" construct to help solve the "tools generating multiple outputs"  
> 
>   case without having to emit big ugly stamp rules in Makefile.am or  
> 
> 
...
> I know you are smart, so you must see the pattern. How hard would it be
> to implement something like this? I don't know how automake works under
> the hood, but I think the syntax could be something like,
>   AM_TOOL_GEN([toolname],[input1,input2],[output1,output2])

Hi Ralf,

You busy or thing the idea is no good?

Thanks,
Bob Rossi

Re: extending automake

[Bob Rossi]

On Fri, Apr 25, 2008 at 06:54:08AM +0200, Ralf Wildenhues wrote:
> * Bob Rossi wrote on Fri, Apr 25, 2008 at 03:41:20AM CEST:
> > On Sat, Apr 19, 2008 at 01:22:29PM -0400, Bob Rossi wrote:
> > > They generate files during build time, and modify BUILT_SOURCES...
> > > 
> > > In fact, think of the bison or flex extension (adding .y or .l files to 
> > > the _SOURCES variable). That is just another use of this general 
> > > functionality that I'm talking about. In some sense, it would be like me
> > > adding foo.xml to the _SOURCES, but telling automake how to turn that
> > > into a .c file. I want to run foo.py, whereas automake runs bison or
> > > flex.
> > > 
> > > I'm sure that if this was implemented, a LOT of projects would use it.
> > > So, is there something I can do to help implement it, with my little
> > > experience writing make file rules?
> > 
> > Ping, whatever happened to this idea? You guys think it's stupid?
> 
> I don't see a way to formulate it sufficiently general so that it would
> be useful for more than just a couple of projects.
> 
> bison and flex need special-cased handling in automake, how do you
> propose foo.py would not?

I think Brian stated it perfectly,

  That brings up the next logical point, can anyone comment on the  
  
  feasibility of some kind of generalized "tool X reads A and outputs Y 
  
  and Z" construct to help solve the "tools generating multiple outputs"
  
  case without having to emit big ugly stamp rules in Makefile.am or
  

Bison and flex both fall under that category. If we don't want to change
them for historical reasons, that is fine, I understand.

However, I would guess that a majority of projects generate some source,
and they could all use the above construct. I'm guessing that most of
them have a rule like i have, a wrong one. So, the advantages would be,

  - automake could safely generate the rule to ensure that tool X gets
run when it needs to (if Y and Z change)
  - the rule would be hopefully correct, improving most projects
  - automake could add the clean rules, and whatever other rules it
normally adds for a _SOURCES or _INCLUDES file.

So, in Mike Mueller's situation, he could do,
  tool flex reads foo.l and outputs foo.c and foo.h
He would also include foo.c and foo.h in _SOURCES and _INCLUDES like 
any other file, and automake would natually handle those files like
any other file. This allows people to use the new feature because they
put the .c file in _SOURCES instead of the normal way of putting the .l
file in the _SOURCES.

In my situation, I would do,
  tool foo.py reads foo.xml and generates foo.h and foo.cc

I know you are smart, so you must see the pattern. How hard would it be
to implement something like this? I don't know how automake works under
the hood, but I think the syntax could be something like,
  AM_TOOL_GEN([toolname],[input1,input2],[output1,output2])

What do you think?
Bob Rossi

Re: extending automake

[Ralf Wildenhues]

* Bob Rossi wrote on Fri, Apr 25, 2008 at 03:41:20AM CEST:
> On Sat, Apr 19, 2008 at 01:22:29PM -0400, Bob Rossi wrote:
> > They generate files during build time, and modify BUILT_SOURCES...
> > 
> > In fact, think of the bison or flex extension (adding .y or .l files to 
> > the _SOURCES variable). That is just another use of this general 
> > functionality that I'm talking about. In some sense, it would be like me
> > adding foo.xml to the _SOURCES, but telling automake how to turn that
> > into a .c file. I want to run foo.py, whereas automake runs bison or
> > flex.
> > 
> > I'm sure that if this was implemented, a LOT of projects would use it.
> > So, is there something I can do to help implement it, with my little
> > experience writing make file rules?
> 
> Ping, whatever happened to this idea? You guys think it's stupid?

I don't see a way to formulate it sufficiently general so that it would
be useful for more than just a couple of projects.

bison and flex need special-cased handling in automake, how do you
propose foo.py would not?

Cheers,
Ralf

Re: extending automake

[Bob Rossi]

On Sat, Apr 19, 2008 at 01:22:29PM -0400, Bob Rossi wrote:
> On Sat, Apr 19, 2008 at 05:51:35PM +0200, Ralf Wildenhues wrote:
> > * Brian Dessent wrote on Sat, Apr 19, 2008 at 04:45:54PM CEST:
> > > It would be equally difficult as in the case with MULTITARGETS and
> > > foo_{TARGETS,SOURCES,COMMAND}, no?
> > 
> > Well, the first step in exploring this further would be somebody writing
> > out how suitable generated rules should look like: if you can then
> > factor it from the input that you're getting, that's already half of the
> > work done.
> > 
> > In any case, I won't be working on this right now due to time
> > constraints, sorry.
> 
> Unfortunately, both of you are talking over my head. I don't have all
> that much experience with make. However, I've worked on a lot of open
> source projects, and all of them do this common task.
> 
> They generate files during build time, and modify BUILT_SOURCES...
> 
> In fact, think of the bison or flex extension (adding .y or .l files to 
> the _SOURCES variable). That is just another use of this general 
> functionality that I'm talking about. In some sense, it would be like me
> adding foo.xml to the _SOURCES, but telling automake how to turn that
> into a .c file. I want to run foo.py, whereas automake runs bison or
> flex.
> 
> I'm sure that if this was implemented, a LOT of projects would use it.
> So, is there something I can do to help implement it, with my little
> experience writing make file rules?

Ping, whatever happened to this idea? You guys think it's stupid?

Bob Rossi

Re: extending automake

[Bob Friesenhahn]

On Sat, 19 Apr 2008, Bob Rossi wrote:


Unfortunately, both of you are talking over my head. I don't have all
that much experience with make. However, I've worked on a lot of open
source projects, and all of them do this common task.

They generate files during build time, and modify BUILT_SOURCES...


BUILT_SOURCES is a very ugly approach which requires an extra forced 
recusion in the build.  It is a sort of crutch which is best avoided.


Bob
==
Bob Friesenhahn
[EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/

Re: extending automake

[Bob Rossi]

On Sat, Apr 19, 2008 at 05:51:35PM +0200, Ralf Wildenhues wrote:
> * Brian Dessent wrote on Sat, Apr 19, 2008 at 04:45:54PM CEST:
> > It would be equally difficult as in the case with MULTITARGETS and
> > foo_{TARGETS,SOURCES,COMMAND}, no?
> 
> Well, the first step in exploring this further would be somebody writing
> out how suitable generated rules should look like: if you can then
> factor it from the input that you're getting, that's already half of the
> work done.
> 
> In any case, I won't be working on this right now due to time
> constraints, sorry.

Unfortunately, both of you are talking over my head. I don't have all
that much experience with make. However, I've worked on a lot of open
source projects, and all of them do this common task.

They generate files during build time, and modify BUILT_SOURCES...

In fact, think of the bison or flex extension (adding .y or .l files to 
the _SOURCES variable). That is just another use of this general 
functionality that I'm talking about. In some sense, it would be like me
adding foo.xml to the _SOURCES, but telling automake how to turn that
into a .c file. I want to run foo.py, whereas automake runs bison or
flex.

I'm sure that if this was implemented, a LOT of projects would use it.
So, is there something I can do to help implement it, with my little
experience writing make file rules?

Thanks,
Bob Rossi

Re: extending automake

[Ralf Wildenhues]

* Brian Dessent wrote on Sat, Apr 19, 2008 at 04:45:54PM CEST:
> Ralf Wildenhues wrote:
> 
> > Do you mean that, given that keyword, all rules of the form
> >   target1 target2 : prereq ...
> > command ...
> > 
> > should be rewritten to be a multiple-target rule?
> 
> Yeah.
> 
> > Ugh.  That would
> > violate the "input appears in output" quite heavily.
> 
> Sure, but that's already violated in minor ways like adding $(EXEEXT). 

I really hate to tread this path any further than I have to.  $(EXEEXT)
has already been a painful experience.  Also, this approach violates the
principle that, while it's good to make things easy for newbies, it
shouldn't make things harder for experienced users.  So if we rewrite
rules in this case, then it should be a syntax no current 'make'
understands (and we could even ask the GNU make developer whether it's
safe to assume so for the future).

> The idea here is to leverage what people would naively write as a first
> attempt at a multitarget rule.  Inventing a new syntax has all the same
> downsides but none of the upsides (if you can even call this an upside.)

No, as I said above: for users who know how 'make' works, not being able
to exploit it (in a Makefile.am where the "new" syntax is used) is a
downside.

> > - can automake extract all needed information if, say, the targets are
> >   not given literally but as either $(macro) or $(substituted_macro) or
> >   @[EMAIL PROTECTED]
> 
> It would be equally difficult as in the case with MULTITARGETS and
> foo_{TARGETS,SOURCES,COMMAND}, no?

Well, the first step in exploring this further would be somebody writing
out how suitable generated rules should look like: if you can then
factor it from the input that you're getting, that's already half of the
work done.

In any case, I won't be working on this right now due to time
constraints, sorry.

Cheers,
Ralf

Re: extending automake

[Brian Dessent]

Ralf Wildenhues wrote:

> Do you mean that, given that keyword, all rules of the form
>   target1 target2 : prereq ...
> command ...
> 
> should be rewritten to be a multiple-target rule?

Yeah.

> Ugh.  That would
> violate the "input appears in output" quite heavily.

Sure, but that's already violated in minor ways like adding $(EXEEXT). 
And in this case you have to ask for it with an option, so presumably
you are prepared for it.

> And be rather
> inflexible in that you cannot at the same time have in the same
> Makefile.am a rule that is meant to be one separate for each target.

True.

> In this case I would prefer inventing a new syntax (like ::: as
> suggested by Olly in the other thread), at least that avoids
> ambiguities.

The idea here is to leverage what people would naively write as a first
attempt at a multitarget rule.  Inventing a new syntax has all the same
downsides but none of the upsides (if you can even call this an upside.)

> More questions, giving the whole thing only a couple minutes thought:
> - does this scale?  It's not all that useful if Bob has to write one
>   such rule for each of his set of files: that's exactly what he wanted
>   to avoid.

That's another thing to consider.  In an ideal world I'd like to be able
to write "%.cc %.h: %.xml ; command $<" and have it be rewritten in a
way that isn't actually a GNUmake-requiring pattern rule but still is
parallel-make safe.  But I have no idea how you'd do that in portable
make, so that's probably shooting for the moon.

> - can automake extract all needed information if, say, the targets are
>   not given literally but as either $(macro) or $(substituted_macro) or
>   @[EMAIL PROTECTED]

It would be equally difficult as in the case with MULTITARGETS and
foo_{TARGETS,SOURCES,COMMAND}, no?

Brian

Re: extending automake

[Ralf Wildenhues]

* Brian Dessent wrote on Sat, Apr 19, 2008 at 03:59:19PM CEST:
> Ralf Wildenhues wrote:
> > 
> 
> The MULTITARGETS and foo_{TARGETS,SOURCES,COMMAND} syntax that you came
> up with is certainly in line with the Automake way of doing things, but
> the observation that writing a MULTITARGET rule looks nothing like a
> normal rule is valid I think.  And ideally you shouldn't have to worry
> about the stamp or lock names at all, they are implementation details.

Agreed.

> I know this will sound a little crazy, but what about simply an
> AUTOMAKE_OPTIONS/AM_INIT_AUTOMAKE keyword that says "whenever I write a
> rule with more than one target, assume I want the 'one program, multiple
> outputs' semantics and not the traditional make semantics."  Automake
> would transparently handle coming up with lock and stamp names and
> adding them to a 'clean' target, as well as emitting all the boring lock
> stuff around the recipe.  This is a DWIM kind of idea, since I have the
> sense that people do in fact write such rules with the expectation of
> "one program, multiple outputs" semantics.  And it would of course
> default to off so that there's no worry of it regressing anything
> existing; and it can be enabled per-file via AUTOMAKE_OPTIONS.

Do you mean that, given that keyword, all rules of the form
  target1 target2 : prereq ...
command ...

should be rewritten to be a multiple-target rule?  Ugh.  That would
violate the "input appears in output" quite heavily.  And be rather
inflexible in that you cannot at the same time have in the same
Makefile.am a rule that is meant to be one separate for each target.
In this case I would prefer inventing a new syntax (like ::: as
suggested by Olly in the other thread), at least that avoids
ambiguities.

More questions, giving the whole thing only a couple minutes thought:
- does this scale?  It's not all that useful if Bob has to write one
  such rule for each of his set of files: that's exactly what he wanted
  to avoid.
- can automake extract all needed information if, say, the targets are
  not given literally but as either $(macro) or $(substituted_macro) or
  @[EMAIL PROTECTED]
- are we safe on the stamp namespace?  Probably yes, we can just use a
  counter and an automake-reserved prefix or so.

Cheers,
Ralf

Re: extending automake

[Brian Dessent]

Ralf Wildenhues wrote:

> This thread has some ideas:
> 
> 
> If you (or anybody) finds the above useful, and think the setup is
> general enough for several use cases, we can think about how to
> integrate it into Automake proper.  Of course, if anyone volunteers
> to work on this, be my guest!  ;-)

The MULTITARGETS and foo_{TARGETS,SOURCES,COMMAND} syntax that you came
up with is certainly in line with the Automake way of doing things, but
the observation that writing a MULTITARGET rule looks nothing like a
normal rule is valid I think.  And ideally you shouldn't have to worry
about the stamp or lock names at all, they are implementation details.

I know this will sound a little crazy, but what about simply an
AUTOMAKE_OPTIONS/AM_INIT_AUTOMAKE keyword that says "whenever I write a
rule with more than one target, assume I want the 'one program, multiple
outputs' semantics and not the traditional make semantics."  Automake
would transparently handle coming up with lock and stamp names and
adding them to a 'clean' target, as well as emitting all the boring lock
stuff around the recipe.  This is a DWIM kind of idea, since I have the
sense that people do in fact write such rules with the expectation of
"one program, multiple outputs" semantics.  And it would of course
default to off so that there's no worry of it regressing anything
existing; and it can be enabled per-file via AUTOMAKE_OPTIONS.

Brian

Re: extending automake

[Ralf Wildenhues]

* Brian Dessent wrote on Sat, Apr 19, 2008 at 02:42:57PM CEST:
> 
> That brings up the next logical point, can anyone comment on the
> feasibility of some kind of generalized "tool X reads A and outputs Y
> and Z" construct to help solve the "tools generating multiple outputs"
> case without having to emit big ugly stamp rules in Makefile.am or
> resorting the GNUmake-isms?

This thread has some ideas:


If you (or anybody) finds the above useful, and think the setup is
general enough for several use cases, we can think about how to
integrate it into Automake proper.  Of course, if anyone volunteers
to work on this, be my guest!  ;-)

Cheers,
Ralf

Re: extending automake

[Brian Dessent]

Ralf Wildenhues wrote:

> Well, this scheme can easily be generalized to one stamp file per set of
> output files, no?  And then it parallelizes well, too.

Yes, I suppose if you named the stamp with a filename derivable from
that of the .xml filename you could generalize this without having to
write 'n' copies of the rule.  That would be a better way.

That brings up the next logical point, can anyone comment on the
feasibility of some kind of generalized "tool X reads A and outputs Y
and Z" construct to help solve the "tools generating multiple outputs"
case without having to emit big ugly stamp rules in Makefile.am or
resorting the GNUmake-isms?

Brian

Re: extending automake

[Brian Dessent]

Bob Rossi wrote:

> verbose). Why are you not sure this is a good idea?

I sort of feel like there are "workspaces", the configure area and the
make area.  This parallels the idea that there are configure-substituted
variables and make-substituted variables, the latter can be
changed/overridden when invoking make, the former can only be changed by
running "./config.status --recheck" or re-running configure.  Whenever
you see a "FOO = @FOO@" in Makefile.in, that's transferring a value from
configure-space to make-space.

Anyway, I tend to see configure-space as being about m4 macros and shell
variables, and the make-space as being about targets, prerequisites, and
recipies.  Since we're talking about the latter it just seemed a little
odd to want to use m4 macros to achieve a goal.

It of course would be possible to put the makefile fragments in a macro
in a .m4 file, and then just leave a slug in Makefile.am to be
substituted, but I'm not sure it seems right to my eyes to see e.g.

# typical Makefile.am stuff
foo_SOURCES = 
bar_LDFLAGS = ...

@SOMETHING@

baz_CPPFLAGS = 

This also has the disadvantage that it hides the expansion of
@SOMETHING@ from automake's eyes, since the @foo@ expansion happens by
config.status when the Makefile is created at the end of running
configure, whereas automake runs much earlier at bootstrap/autoreconf
time.  In this case it might not matter since what you'd be passing
through would likely not contain anything interesting to automake.  But
it could, for example both BUILT_SOURCES and EXTRA_DIST are automake
variables whose assignments it would probably have to see, so you
wouldn't be able to use any of them which could cramp your style a lot.

Besides, my impression was that there wouldn't really be any need for
any repitition in the first place, you'd just write one copy and then
list all your .xml files in PYXMLFILES and that's that.  But now I see
that's not the case:

> Yes, I do know that, and that's how I got this far. Basically, I write
> these rules in several Makefile.am's, since I have multiple directory
> setup. I would love to not duplicate the make code, and hide it all
> behind a single function call that can be expanded.

You could put the fragments in a common file and then include that file
in each Makefile.am that requires the functionality.  This gets around
the above problem of the fragments not being visible at automake time
since AFAIK automake can "see through" make-level includes (correct me
if I'm wrong anyone), which means you would be free to make use of
automake variables/features instead of having to tippy-toe around them
carefully.

Brian

Re: extending automake

[Bob Rossi]

On Fri, Apr 18, 2008 at 09:23:59PM -0700, Brian Dessent wrote:
> Bob Rossi wrote:
> > I would like to automake this by writing a wrapper like,
> >   AM_GEN(foo.xml)
> 
> I'm not sure that configure-time is where you want to deal with this.

Wow, your answer is great, and shows how little I actually know about 
make. However, it proves to me even slightly more that I want to
generate these rules somehow (because the generated rules are reasonably
verbose). Why are you not sure this is a good idea?

> > Is this possible? If so, is there some documentation on how to do it?
> > I looked through the manual and didn't see it, or just missed it.
> 
> Remember that you can write arbitrary Makefile syntax in Makefile.am and
> it is simply copied through to the output file, so in that sense you
> don't have to use any special syntax to extend automake, just pretend
> it's not there.

Yes, I do know that, and that's how I got this far. Basically, I write
these rules in several Makefile.am's, since I have multiple directory
setup. I would love to not duplicate the make code, and hide it all
behind a single function call that can be expanded.

Is this possible?

Thanks again!
Bob Rossi

Re: extending automake

[Ralf Wildenhues]

Hi Bob, Brian,

* Brian Dessent wrote on Sat, Apr 19, 2008 at 06:23:59AM CEST:
> Method B, not GNU make specific, using a single witness stamp along the
> lines of what is suggested in the manual:
[...]
> pyxml-stamp: $(PYXMLFILES)
>   @rm -f pyxml-tmp
>   @touch pyxml-tmp
>   (set -e; for F in $<; do \
> python $(abs_top_srcdir)/gen.py $$F; \
> done)
>   @mv -f pyxml-tmp $@
[...]
> Notes:
>  - This should be parallel-make safe due to the single stamp file,
> except for the contrived situation mentioned in the manual where one or
> more generated file is removed without also removing the stamp.
>  - But also due to a single stamp file representing all .xml files,
> parallel make won't be able to run multiple python invocations in
> parallel, and if one .xml file is modified they will all be regenerated.

Well, this scheme can easily be generalized to one stamp file per set of
output files, no?  And then it parallelizes well, too.

Cheers,
Ralf

Re: extending automake

[Brian Dessent]

Bob Rossi wrote:

> EXTRA_DIST = foo.xml
> 
> BUILT_SOURCES = foo.h foo.cc
> 
> foo.h foo.cc: $(srcdir)/foo.xml $(top_srcdir)/gen.py
>   python $(abs_top_srcdir)/gen.py $(srcdir)/foo.xml

This is not good.  When you write:

target1 target2: prereq
recipe

It is really shorthand for:

target1: prereq
recipe

target2: prereq
recipe

You are not actually expressing to make that running 'recipe' generates
both target1 and target2, so there is a chance that make will needlessly
run 'recipe' twice.  Additionally, this is not parallel-make safe since
make will spawn two simultaneous instances of 'recipe' which will
compete to write to the same two output files.  There is an entire
section in the Automake manual (27.9, "Handling Tools that Produce Many
Outputs") dealing with this situation that's worth reading.

> I would like to automake this by writing a wrapper like,
>   AM_GEN(foo.xml)

I'm not sure that configure-time is where you want to deal with this.

> Is this possible? If so, is there some documentation on how to do it?
> I looked through the manual and didn't see it, or just missed it.

Remember that you can write arbitrary Makefile syntax in Makefile.am and
it is simply copied through to the output file, so in that sense you
don't have to use any special syntax to extend automake, just pretend
it's not there.

I see several potential methods of tackling this (both untested):

Method A, which would require the use of GNU make:

PYXMLFILES = foo.xml bar.xml baz.xml
EXTRA_DIST += $(PYXMLFILES)
BUILT_SOURCES += $(PYXMLFILES:.xml=.h) $(PYXMLFILES:.xml=.cc)

%.cc %.h: %.xml $(top_srcdir)/gen.py
python $(word 2,$^) $<

Notes:
 - In general you should not use $(srcdir)/filename when referring to
targets or prerequisites in a Makefile.  That is the whole point of
VPATH -- make will automatically look in $srcdir to find things, whether
that means the current directory or somewhere else entirely.  The only
thing to remember is that when referring to the corresponding file in
the recipe, you should use automatic variables instead of specifying the
filename a second time.  That way, make substitutes the path from where
it was found from VPATH.  Or stated differently: don't duplicate
filenames from the target or prerequisite list in the recipe, refer to
them through $@, $<, $^, etc.

 - Writing a pattern rule with multiple targets (%.a %.b: %.c) is
different than writing a standard rule with multiple targets, in that
this form *does* tell make that both %.a and %.b are generated by
running 'recipe', so it should be parallel-safe.  But again this is GNU
make specific.

Method B, not GNU make specific, using a single witness stamp along the
lines of what is suggested in the manual:

PYXMLFILES = foo.xml bar.xml baz.xml
EXTRA_DIST += $(PYXMLFILES)
BUILT_SOURCES += $(PYXMLFILES:.xml=.h) $(PYXMLFILES:.xml=.cc)

pyxml-stamp: $(PYXMLFILES)
@rm -f pyxml-tmp
@touch pyxml-tmp
(set -e; for F in $<; do \
  python $(abs_top_srcdir)/gen.py $$F; \
done)
@mv -f pyxml-tmp $@

$(PYXMLFILES:.xml=.h) $(PYXMLFILES:.xml=.cc): pyxml-stamp
@if test -f $@; then :; else \
  rm -f $<; \
  $(MAKE) $(AM_MAKEFLAGS) $<; \
fi

Notes:
 - This should be parallel-make safe due to the single stamp file,
except for the contrived situation mentioned in the manual where one or
more generated file is removed without also removing the stamp.
 - But also due to a single stamp file representing all .xml files,
parallel make won't be able to run multiple python invocations in
parallel, and if one .xml file is modified they will all be regenerated.

Brian

extending automake

[Bob Rossi]

Hi,

Up until now, I've just used the canned macros that automake provides.
However, I'm wondering if it has a way that I can extend it.

I basically have an xml file, and a python script, that generates a
header and a source file. I write this in multiple spots in my
Makefile.am

EXTRA_DIST = foo.xml

BUILT_SOURCES = foo.h foo.cc

foo.h foo.cc: $(srcdir)/foo.xml $(top_srcdir)/gen.py
  python $(abs_top_srcdir)/gen.py $(srcdir)/foo.xml

I would like to automake this by writing a wrapper like,
  AM_GEN(foo.xml)

Is this possible? If so, is there some documentation on how to do it?
I looked through the manual and didn't see it, or just missed it.

Thanks,
Bob Rossi

Re: extending automake support for exotic languages

[Alexandre Duret-Lutz]

>>> "Guillaume" == Guillaume Rousse <[EMAIL PROTECTED]> writes:

 Guillaume> Hello.

 Guillaume> A colleague of mine developped a custom language for
 Guillaume> natural language processing, called DyALog
 Guillaume> (http://atoll.inria.fr/~clerger/work.html#DyALog). In
 Guillaume> order to ease module development, he also extended
 Guillaume> automake to support this language. However, he is
 Guillaume> currently maintaining it as a patch against current
 Guillaume> automake version, which is quite a nightmare.

 Guillaume> Is there a more convenient way for doing this ? 

Right now I believe the only maintainable way is to distribute a
Makefile fragment that contains the suffix rules for your
language, and let users include this fragment in their
Makefile.am.  This won't cope with per-target flags, or subdir
compilation without subdir-objects, but that should be pretty
usable nonetheless.

There is a long-term plan for modularizing Automake to ease the
addition of new features (using Perl modules).  However such
overhauling of the Automake internal is not easy, and given the
current development pace you should not expect this to happen
any time soon.  Unless somebody comes and helps.
(http://sources.redhat.com/automake/contribute.html)
Unfortunately I'm maintaining Automake almost alone, with
limited time, and my priorities are to correct bugs and to
improve the documentation.

 Guillaume> I suppose you don't intend to accept a patch for
 Guillaume> every minor research language in the world.

Indeed.
-- 
Alexandre Duret-Lutz

extending automake support for exotic languages

[Guillaume Rousse]

Hello.
A colleague of mine developped a custom language for natural language 
processing, called DyALog 
(http://atoll.inria.fr/~clerger/work.html#DyALog). In order to ease 
module development, he also extended automake to support this language. 
However, he is currently maintaining it as a patch against current 
automake version, which is quite a nightmare. Is there a more convenient 
way for doing this ? I suppose you don't intend to accept a patch for 
every minor research language in the world.
--
The tool you need is back at the shop.
		-- Mechanics' Maxims n°3

Re: Extending automake

[ML]

If somebody can take me off this list or tell me how I can do it myself I'd
be rather happy.

-mario

- Original Message -
From: "Faschingbauer" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Cc: <[EMAIL PROTECTED]>
Sent: Tuesday, August 03, 2004 3:30 PM
Subject: Re: Extending automake


> >>>>> "Chris" == Chris Lee <[EMAIL PROTECTED]> writes:
>
> Chris> Hi, all.
> Chris> In KDE, currently, we have a custom perl script called am_edit
which
> Chris> does a bunch of voodoo to support some custom keywords that we use
in
> Chris> KDE-ized Makefile.am files.
>
> Chris> I personally am on a vendetta against our existing crufty build
system,
> Chris> and as part of that I'm trying to figure out how to kill am_edit.
>
> Chris> Now, one of the things[0] that our build system supports which I
really
> Chris> want to continue to support is the definition:
> Chris> foo_METASOURCES = AUTO
>
> Chris> Which automatically scans all of the source files and generates
'.moc'
> Chris> files for the ones that require it by running the moc command. It's
> Chris> slightly more complicated than just running moc, as you need to run
moc
> Chris> against the header file and not the cpp file.
>
> Chris> For example, foo.cpp may have the line '#include "foo.moc"' in it
> Chris> somewhere. The command that needs to be run before foo.cpp gets
compiled is:
>
> Chris> moc -o foo.moc foo.h
>
> Chris> How do I go about doing this The Right Way?
>
> I can imagine that this could be done with confix (.sf.net), which
> generates automake input basically, and which you can extend with a
> plugin scheme for arbitrary code generators.
>
> Joerg
>

Re: Extending automake

[Faschingbauer]

> "Chris" == Chris Lee <[EMAIL PROTECTED]> writes:

Chris> Hi, all.
Chris> In KDE, currently, we have a custom perl script called am_edit which 
Chris> does a bunch of voodoo to support some custom keywords that we use in 
Chris> KDE-ized Makefile.am files.

Chris> I personally am on a vendetta against our existing crufty build system, 
Chris> and as part of that I'm trying to figure out how to kill am_edit.

Chris> Now, one of the things[0] that our build system supports which I really 
Chris> want to continue to support is the definition:
Chris> foo_METASOURCES = AUTO

Chris> Which automatically scans all of the source files and generates '.moc' 
Chris> files for the ones that require it by running the moc command. It's 
Chris> slightly more complicated than just running moc, as you need to run moc 
Chris> against the header file and not the cpp file.

Chris> For example, foo.cpp may have the line '#include "foo.moc"' in it 
Chris> somewhere. The command that needs to be run before foo.cpp gets compiled is:

Chris> moc -o foo.moc foo.h

Chris> How do I go about doing this The Right Way?

I can imagine that this could be done with confix (.sf.net), which
generates automake input basically, and which you can extend with a
plugin scheme for arbitrary code generators.

Joerg

Re: Extending automake

[Jan Kratochvil]

Hi,

On Sun, 01 Aug 2004 21:47:36 +0200, Chris Lee wrote:
...
> Even removing the ability to provide the foo_METASOURCES = AUTO would be
> fine with me if I could at least do 'foo_METASOURCES = foo.moc bar.moc'
> but at the moment I still don't quite grok how to do that.

In such case you can use:

bin_PROGRAMS=foo
foo_SOURCES= \
foo.c \
foo.h

.c.moc:
moc $< -o $@

.h.moc:
moc $< -o $@

BUILT_SOURCES = $(shell cat $(foo_SOURCES) | sed -n 
's/^\#.*include.*"\(.*[.]moc\)".*$$/\1/p')


Although there is a GNU make dependency on $(shell ...). I hope the
cross-make(1) compatible way is:


bin_PROGRAMS=foo
foo_SOURCES= \
foo.c \
foo.h

.c.moc:
moc $< -o $@

.h.moc:
moc $< -o $@

BUILT_SOURCES = built_sources
.PHONY: built_sources

./$(DEPDIR)/.moc_deps: $(foo_SOURCES)
cat $(foo_SOURCES) | sed -n 's/^\#.*include.*"\(.*[.]moc\)".*$$/built_sources: 
\1/p' >$@

include ./$(DEPDIR)/.moc_deps


There is a GNU make(1) warning message
Makefile:1170: .deps/.moc_deps: No such file or directory

, I hope some non-GNU make(1)s do not bail out on this; AFAIK automake(1) also
uses this approach.


Regards,
Lace

P.S.: It is not fair to reply offlist mails not intended for the list to the
  list.

-- 
Jan Kratochvil; Captive: free r/w NTFS Filesystem; http://www.jankratochvil.net/

Re: Extending automake

[Andrew Suffield]

On Sun, Aug 01, 2004 at 03:47:36PM -0400, Chris Lee wrote:
> Even removing the ability to provide the foo_METASOURCES = AUTO would be
> fine with me if I could at least do 'foo_METASOURCES = foo.moc bar.moc'
> but at the moment I still don't quite grok how to do that.

I generally do this sort of thing (three times now) by writing a perl
script that generates makefile chunks, which I have include directives
for. This script handles any appropriate scanning or higher-level
input files and rule generation. With a little imagination and
*proper* development work (not the ham-handed effort that many people,
especially projects like KDE, tend to put into their build systems) it
can be done neatly, maintainably, and with a minimum of fuss.

I don't believe it's possible without adding an extra pass. Nor does
it seem particularly useful to do so.

The trick is to not do it stupidly, like the current KDE build system
does. There's no reason why the higher-level description should
pretend to be a Makefile.am. Don't "customize" the automake input
format - make up your own in a way that's appropriate to the problem
(possibly you can do it with no input file at all, and just hardcode
the behaviour into the generating script), and generate classic
makefile chunks, then feed these back to automake with include
directives.

As a matter of form I try to generate makefile chunks that just
contain variable definitions (and sometimes dependencies), and put the
rules and automake variables into the Makefile.am proper - even if I
end up with "foo_SOURCES = $(foo_sources)". If I have rules that are
common to several makefiles, I use something like
"include $(top_srcdir)/build-misc/docbook.mk".

-- 
  .''`.  ** Debian GNU/Linux ** | Andrew Suffield
 : :' :  http://www.debian.org/ |
 `. `'  |
   `- -><-  |


signature.asc
Description: Digital signature

Re: Extending automake

[Chris Lee]

I would really appreciate keeping holy wars out of the discussion; I
fail to see how your view of the dichotomy between GNOME and KDE has
anything to do with what I asked.
Whether or not the GNOME people like it isn't my business; I would like
to get rid of the crufty implementation we have accumulated as our build
system, and I'd like to do it in a way that uses the standard mechanisms
provided for extending automake, while still providing backwards
compatibility if at all possible. I'm just terribly confused at the
moment by the .am files in my /usr/share/automake directory, so I
thought I would ask if anyone here could point me to something that
might enlighten me.
Even removing the ability to provide the foo_METASOURCES = AUTO would be
fine with me if I could at least do 'foo_METASOURCES = foo.moc bar.moc'
but at the moment I still don't quite grok how to do that.
-clee
Jan Kratochvil wrote:
Hi,
This is the reason there is Gnome and there is KDE. Each has its own processes,
each goes in its own style.
KDE uses the quick&dirty way of coding to get the job done.
Gnome uses your "The Right Way" although it takes more effort to code it
(at least in the short-term).
I doubt any GNU and/or Gnome people are going to accept such horrible hacks
like some autoscanning of wildcarded sources.
I think it is right these two environments live separated to each other.
Possible replies smell with flaming so I welcome no replies at all.
Regards,
Lace - Gnome guy

Extending automake

[Chris Lee]

Hi, all.
In KDE, currently, we have a custom perl script called am_edit which 
does a bunch of voodoo to support some custom keywords that we use in 
KDE-ized Makefile.am files.

I personally am on a vendetta against our existing crufty build system, 
and as part of that I'm trying to figure out how to kill am_edit.

Now, one of the things[0] that our build system supports which I really 
want to continue to support is the definition:
foo_METASOURCES = AUTO

Which automatically scans all of the source files and generates '.moc' 
files for the ones that require it by running the moc command. It's 
slightly more complicated than just running moc, as you need to run moc 
against the header file and not the cpp file.

For example, foo.cpp may have the line '#include "foo.moc"' in it 
somewhere. The command that needs to be run before foo.cpp gets compiled is:

moc -o foo.moc foo.h
How do I go about doing this The Right Way?
-clee
[0] Our build system supports some other crack too, which would also be 
nice to support; there's documentation about it at 
http://developer.kde.org/documentation/other/developer-faq.html if 
people are interested in looking at it.

extending Automake with a custom primary

[Ireneusz SZCZESNIAK]

Hi,

I want to extend Automake with my own primary.  I am writing to ask if
Automake can be customized this way.  I would like to write in
Automake.am:

VPSRC = @VPSRC@
noinst_VPBUILD = vorpal_ser vorpal_par
vorpal_ser_CONFFLAGS = --disable-mpi
vorpal_par_CONFFLAGS = --enable-mpi

My primary would be "VPBUILD", and I would pass some parameters with
"CONFFLAGS" to build a particular target (vorpal_ser or vorpal_par).
The example above would instruct Automake (and my custom macros) to
create two directories: "vorpal_ser" and "vorpal_par".  In each of
these two directories our software would be configured with different
flags (the flags would be given with "CONFFLAGS") and then built.  The
variable "VPSRC" would be used in the process of configuring and
building.


Thanks & best,
Irek

extending Automake with a custom primary

[Ireneusz SZCZESNIAK]

Hi,

I want to extend Automake with my own primary.  I am writing to ask if
Automake can be customized this way.  I would like to write in
Automake.am:

VPSRC = @VPSRC@
noinst_VPBUILD = vorpal_ser vorpal_par
vorpal_ser_CONFFLAGS = --disable-mpi
vorpal_par_CONFFLAGS = --enable-mpi

My primary would be "VPBUILD", and I would pass some parameters with
"CONFFLAGS" to build a particular target (vorpal_ser or vorpal_par).
The example above would instruct Automake (and my custom macros) to
create two directories: "vorpal_ser" and "vorpal_par".  In each of
these two directories our software would be configured with different
flags (the flags would be given with "CONFFLAGS") and then built.  The
variable "VPSRC" would be used in the process of configuring and
building.


Thanks & best,
Irek

Re: Extending automake syntax

[Erik Walthinsen]

On 1 Jan 2002, Tom Tromey wrote:

> Is SpeciaLib GNU software?
> One of automake's goals is to interoperate well with GNU software.
SpeciaLib doesn't quite exist yet except as pieces pulled from other
projects of mine, mostly libcodec (see codecs.org for all this stuff
anyway).  I intend to make other libcodec-style libs (like libcolorspace,
etc.), and don't want to do the work over and over.  It will be GPL/LGPL,
but won't be "GNU software" in the strictest sense.

> It depends on demand, design, and patches.  Usually I try to work on
> addressing the problems reported by the most people (as subjectively
> interpreted by me :-).


  Erik Walthinsen <[EMAIL PROTECTED]> - System Administrator
__
   /  \GStreamer - The only way to stream!
  || M E G A* http://gstreamer.net/ *
  _\  /_

Re: Extending automake syntax

[Tom Tromey]

> "Erik" == Erik Walthinsen <[EMAIL PROTECTED]> writes:

>> You can't accomplish what you want without hacking automake itself.
>> It seems like something specific to your project.  Is that the case?

Erik> Yup.  Well, sorta.  "SpeciaLib" is a library and a set of tools
Erik> for creating specializable libraries.

Is SpeciaLib GNU software?
One of automake's goals is to interoperate well with GNU software.

Erik> Any chance automake will move towards being more externally
Erik> customizable?

It depends on demand, design, and patches.  Usually I try to work on
addressing the problems reported by the most people (as subjectively
interpreted by me :-).

Tom

Re: Extending automake syntax

[Erik Walthinsen]

On 31 Dec 2001, Tom Tromey wrote:

> Erik> lib_SLLIBRARIES = libaverage.sl
>
> Erik> libaverage_sl_SOURCES = \
> Erik> average_c.c \
> Erik> average_mmx.c   \
> Erik> average_sse.c   \
> Erik> average_altivec.c
>
> You can't accomplish what you want without hacking automake itself.
> It seems like something specific to your project.  Is that the case?

Yup.  Well, sorta.  "SpeciaLib" is a library and a set of tools for
creating specializable libraries.  As such, these modified Makefiles would
exist in any library that uses SpeciaLib to construct itself (the tools do
things like generate headers and structures used in run-time
specialization).  I've thought of writing a tool that would conveniently
be called 'specialize' that would be a cross between libtoolize and
automake, creating Makefile.am's from Makefile.sl's (or whatever I call
them), but that means a significant amount of work in properly
interpreting the Makefile.am-isms...

Any chance automake will move towards being more externally customizable?

  Erik Walthinsen <[EMAIL PROTECTED]> - System Administrator
__
   /  \GStreamer - The only way to stream!
  || M E G A* http://gstreamer.net/ *
  _\  /_

Re: Extending automake syntax

[Tom Tromey]

> "Erik" == Erik Walthinsen <[EMAIL PROTECTED]> writes:

Sorry for the delay in my reply.

Erik> I'm working on a project where I want to create a new class of
Erik> vars for automake, which will include rules which will turn into
Erik> both generated source targets, and normal libtool libs.  For
Erik> example:

Erik> lib_SLLIBRARIES = libaverage.sl

Erik> libaverage_sl_SOURCES = \
Erik> average_c.c \
Erik> average_mmx.c   \
Erik> average_sse.c   \
Erik> average_altivec.c

You can't accomplish what you want without hacking automake itself.
It seems like something specific to your project.  Is that the case?

Tom

Extending automake syntax

[Erik Walthinsen]

I'm working on a project where I want to create a new class of vars for
automake, which will include rules which will turn into both generated
source targets, and normal libtool libs.  For example:

lib_SLLIBRARIES = libaverage.sl

libaverage_sl_SOURCES = \
average_c.c \
average_mmx.c   \
average_sse.c   \
average_altivec.c

would turn into something like:

lib_LTLIBRARIES = libaverage.la

libaverage_la_SOURCES = \
libaverage_stub.c   \
average.h

average.h: $(libaverage_la_SOURCES) average.h.proto
cp average.h.proto average.h# replace with a script

libaverage_stub.c:
echo '#define __SL_IN_SPECIALIB' > $@
echo '#include "config.h"' >> $@
echo '#include "average.h"' >> $@

clean:
rm -f libaverage_stub.c average.h

Is it possible to hook into automake itself to accomplish this, or am I
going to have to create a script to be run before automake to parse the
file and rewrite it as appropriate?  I'd rather not have to do that

TIA,
   Omega

  Erik Walthinsen <[EMAIL PROTECTED]> - System Administrator
__
   /  \GStreamer - The only way to stream!
  || M E G A* http://gstreamer.net/ *
  _\  /_