bug#13928: [PATCH 1/4] deps: 'subdir-object' option now works when foo_SOURCES contains $(var)

[Stefano Lattarini]

Following a suggestions of Johan Kristensen, we have config.status use
'make' invocations rather than Makefile-parsing 'sed' hacks to bootstrap
the dependency-tracking '.Po' and '.Plo' makefile fragments. To handle
the inclusion of such files that are still missing when make is first
we basically generate a temporary Makefile without these includes, and
call 'make' on that Makefile.

This fixes the serious bug bug#13928, which was an hard blocker to make the
behavior mandated by the 'subdir-object' active by default (which we want
to do in Automake 2.0).

The issue has also been reported in bug#15919.

* NEWS, THANKS: Update.
* bin/automake.in (handle_languages): Add a trailing "marking" comment
("# am--include-marker") to the generated Makefile lines issuing 'include'
directives for the dependency-tracking '.Po' and '.Plo' makefile fragments.
Also rename the generated Makefile variable 'am__depfiles_maybe' to the
clearer 'am__maybe_remake_depfiles'.
Minor unrelated refactoring.
* lib/am/configure.am: Adjust to account for the 'am__depfiles_maybe' ->
'am__maybe_remake_depfiles' renaming.
* lib/am/depend.am: Add rules to generate a dummy version of all the
dependency-tracking '.Po' and '.Plo' makefile fragments.
* m4/depout.m4: Use make invocations rather than Makefile-parsing sed hacks
to bootstrap the dependency-tracking '.Po' and '.Plo' makefile fragments.
We still use some sed trickery in order to remove the inclusion of the
still non existing .Po and .Plo files from the Makefile we invoke make
upon; this is done stripping lines that contain the magic string
"# am--include-marker".
* m4/make.m4 (AM_MAKE_INCLUDE): Given that now automake generates Makefiles
containing include statements with trailing comment, adjust the checks done
here to make sure $MAKE support that; e.g., "include foo.mk # comment"
rather than just "include foo.mk".
Also refactor and adjust to leave better debugging info in config.log.
* t/postproc.sh: Rename ...
* t/depend-postproc.sh: ... to this, and adjust and enhance.
* t/list-of-tests.mk (handwritten_TESTS): Adjust.
(XFAIL_TESTS): Remove 't/subobj-indir-pr13928.sh', which is now succeeding.
* t/subobj-indir-pr13928.sh: Simplify slightly, now that we expect it to
pass.
* t/depcomp8a.sh: Adjust grepping check to account for the changes in
the generated Makefile, and tp be somewhat more robust in light of possible
future modifications.
* t/depcomp8b.sh: Likewise.
* t/subobj11b.sh: Likewise.
* t/subobj11c.sh: Likewise.
* t/extra-sources.sh: Likewise.
* t/lex-depend-grep.sh: Likewise.
* t/lex-depend-cxx.sh: Add a command to help debugging in case of test
failure.

Helped-by: Johan Kristensen 
Signed-off-by: Stefano Lattarini 
---
 NEWS  | 28 
 THANKS|  1 +
 bin/automake.in   | 84 ++-
 lib/am/configure.am   |  6 +--
 lib/am/depend.am  |  7 +++
 m4/depout.m4  | 78 ++--
 m4/make.m4| 59 +++-
 t/depcomp8a.sh| 16 +++
 t/depcomp8b.sh| 16 +++
 t/{postproc.sh => depend-postproc.sh} | 29 ++--
 t/extra-sources.sh|  5 ++-
 t/lex-depend-cxx.sh   |  5 +++
 t/lex-depend-grep.sh  |  2 +-
 t/list-of-tests.mk|  3 +-
 t/subobj-indir-pr13928.sh |  4 +-
 t/subobj11b.sh| 10 ++---
 t/subobj11c.sh|  2 +-
 17 files changed, 203 insertions(+), 152 deletions(-)
 rename t/{postproc.sh => depend-postproc.sh} (68%)

diff --git a/NEWS b/NEWS
index 84a5e0a..a930049 100644
--- a/NEWS
+++ b/NEWS
@@ -62,6 +62,34 @@
 
 
 
+New in 1.16:
+
+* Bugs fixed:
+
+  - Automatic dependency tracking has been fixed to work also when the
+subdir-object option is used and some 'foo_SOURCES' definition contains
+unexpanded references to make variables, as in, e.g.:
+
+a_src = sources/libs/aaa
+b_src = sources/bbb
+foo_SOURCES = $(a_src)/bar.c $(b_src)/baz.c
+
+With such a setup, the created makefile fragment containing dependency
+tracking information will be correctly placed under the directories
+named 'sources/libs/aaa/.deps' and 'sources/bbb/.deps', rather than
+mistakenly under directories named (literally!) '$(src_a)/.deps' and
+'$(src_b)/.deps' (this was automake bug#13928).
+
+Notice that in order to fix this bug we had to slightly change the
+semantics of how config.status bootstraps the makefile fragments
+required for the dependency tracking to work: rather than attempting
+to parse the Makefiles via grep and sed trickeries only, we actually
+invoke 'make' on a slightly preprocessed version of those Makefiles,
+using a pri

bug#13928: [PATCH 3/4] compile: don't place built object files in $(srcdir), ever ...

[Stefano Lattarini]

... even when a source file is specified as '$(srdir)/foo.c' or
'$(top_srcdir)/bar.c'. And ditto for dependency-tracking makefile
fragments (those under '.deps' directories).

Such issues used to occur when the 'subdir-objects' option was given.

This change should fix the second and last part of automake bug#13928.
See also bug#16375 and bug#15293.

* NEWS: Update.
* bin/automake.in (handle_single_transform): Make sure object files
and dependency-tracking makefile fragments coming from source like
'$(srcdir)/foo.c' and '$(top_srcdir)/bar.c' are placed respectively
under $(builddir) and $(top_builddir).
* t/subobj-vpath-pr13928.sh: Enhance to expose even more aspects
of the bug we've just fixed.
* t/subobj-pr13928-more-langs.sh: New test, similar to the one above,
but with non-C languages as well.
* t/list-of-tests.mk (XFAIL_TESTS): Remove 'subobj-vpath-pr13928.sh',
it's now supposed to pass.
(handwritten_TESTS): Add 'subobj-pr13928-more-langs.sh'.

Signed-off-by: Stefano Lattarini 
---
 NEWS   |  21 ++-
 bin/automake.in|  50 ++-
 t/list-of-tests.mk |   2 +-
 t/subobj-pr13928-more-langs.sh | 138 +
 t/subobj-vpath-pr13928.sh  |  89 ++
 5 files changed, 269 insertions(+), 31 deletions(-)
 create mode 100644 t/subobj-pr13928-more-langs.sh

diff --git a/NEWS b/NEWS
index a930049..eb3bc6f 100644
--- a/NEWS
+++ b/NEWS
@@ -67,8 +67,8 @@ New in 1.16:
 * Bugs fixed:
 
   - Automatic dependency tracking has been fixed to work also when the
-subdir-object option is used and some 'foo_SOURCES' definition contains
-unexpanded references to make variables, as in, e.g.:
+'subdir-object' option is used and some 'foo_SOURCES' definition
+contains unexpanded references to make variables, as in, e.g.:
 
 a_src = sources/libs/aaa
 b_src = sources/bbb
@@ -78,7 +78,7 @@ New in 1.16:
 tracking information will be correctly placed under the directories
 named 'sources/libs/aaa/.deps' and 'sources/bbb/.deps', rather than
 mistakenly under directories named (literally!) '$(src_a)/.deps' and
-'$(src_b)/.deps' (this was automake bug#13928).
+'$(src_b)/.deps' (this was the first part of automake bug#13928).
 
 Notice that in order to fix this bug we had to slightly change the
 semantics of how config.status bootstraps the makefile fragments
@@ -88,6 +88,21 @@ New in 1.16:
 using a private target that is only meant to bootstrap the required
 makefile fragments.
 
+  - The 'subdir-object' option no longer causes object files corresponding
+to source files specified with an explicit '$(srcdir)' component to be
+placed in the source tree rather than in the build tree.
+
+For example, if Makefile.am contains:
+
+AUTOMAKE_OPTIONS = subdir-objects
+foo_SOURCES = $(srcdir)/foo.c $(srcdir)/s/bar.c $(top_srcdir)/baz.c
+
+then "make all" will create 'foo.o' and 's/bar.o' in $(builddir) rather
+than in $(srcdir), and will create 'baz.o' in $(top_builddir) rather
+than in $(top_srcdir).
+
+This was the second part of automake bug#13928.
+
 
 
 New in 1.15:
diff --git a/bin/automake.in b/bin/automake.in
index f19be92..d8ecca5 100644
--- a/bin/automake.in
+++ b/bin/automake.in
@@ -1617,9 +1617,9 @@ sub handle_single_transform
my $renamed = 0;
my ($linker, $object);
 
-   # This records whether we've seen a derived source file (e.g.
-   # yacc output).
-   my $derived_source = 0;
+# This records whether we've seen a derived source file (e.g., yacc
+# or lex output).
+my $derived_source;
 
# This holds the 'aggregate context' of the file we are
# currently examining.  If the file is compiled with
@@ -1667,17 +1667,36 @@ sub handle_single_transform
# Now extract linker and other info.
$linker = $lang->linker;
 
-   my $this_obj_ext;
-   if (defined $source_extension)
-   {
-   $this_obj_ext = $source_extension;
-   $derived_source = 1;
-   }
-   else
-   {
-   $this_obj_ext = $obj;
-   }
-   $object = $base . $this_obj_ext;
+my $this_obj_ext;
+if (defined $source_extension)
+  {
+$this_obj_ext = $source_extension;
+$derived_source = 1;
+  }
+else
+  {
+$this_obj_ext = $obj;
+$derived_source = 0;
+# Don't ever place built object files in $(srcdir),
+# even when sources are specified explicitly as (say)
+# '$(srcdir)/foo.c' or '$(top_srcdir)/foo.c'.
+# See automake bug#13928.
+my @d = split '/', $directory;
+if (@d > 0 &

bug#13928: [PATCH 0/4] Fix automake bug#13928

[Stefano Lattarini]

Stefano Lattarini (4):
  deps: 'subdir-object' option now works when foo_SOURCES contains $(var)
  tests: fix some bugs in an XFAILing test
  compile: don't place built object files in $(srcdir), ever ...
  deps: fix corner-case "make distclean" bug

 NEWS  |  43 +++
 PLANS/subdir-objects.txt  |  10 +--
 THANKS|   1 +
 bin/automake.in   | 134 +++--
 lib/am/configure.am   |   6 +-
 lib/am/depend.am  |  14 +++-
 m4/depout.m4  |  78 +--
 m4/make.m4|  59 +++
 t/depcomp8a.sh|  16 ++--
 t/depcomp8b.sh|  16 ++--
 t/{postproc.sh => depend-postproc.sh} |  29 ++-
 t/extra-sources.sh|   5 +-
 t/lex-depend-cxx.sh   |   5 ++
 t/lex-depend-grep.sh  |   2 +-
 t/list-of-tests.mk|   5 +-
 t/pr224.sh|   4 +-
 t/subobj-indir-pr13928.sh |   4 +-
 t/subobj-pr13928-more-langs.sh| 138 ++
 t/subobj-vpath-pr13928.sh |  89 +++---
 t/subobj11b.sh|  10 +--
 t/subobj11c.sh|   2 +-
 21 files changed, 478 insertions(+), 192 deletions(-)
 rename t/{postproc.sh => depend-postproc.sh} (68%)
 create mode 100644 t/subobj-pr13928-more-langs.sh

-- 
2.1.3

bug#13928: [PATCH 4/4] deps: fix corner-case "make distclean" bug

[Stefano Lattarini]

Assume we have package satisfying the following conditions:
  (1) automatic dependency tracking is enabled;
  (2) the 'subdir-objects' Automake option is enabled;
  (3) the package uses a recursive make setup.

Also assume that:
  (a) a subdir Makefile declares a foo_SOURCES variable containing
  a source file in the parent directory;
  (b) that parent Makefile declare a compiled program itself.

Then BSD and Solaris make used to fail when running "make distclean",
because the 'distclean' target of the subdir Makefile removed the
whole '.deps' directory before the parent Makefile was done with the
included '.Po' makefile fragments in that directory. This issue was
revealed by failures in the 'subobj-vpath-pr13928.sh' test when those
make implementations were used.

We fix the issue by ensuring the 'distclean' target of any Makefile
only removed the '.Po' makefile fragments included by it, rather than
the whole '.deps' directory where such files resides.

This change should be the last step in fixing automake bug#13928
for good.

* bin/automake.in (handle_languages), lib/am/depend.am: Adjust
to implement the new 'distclean' logic.
* t/pr224.sh: Adjust to avoid a spurious failure.
* PLANS/subdir-objects.txt: Update.

Signed-off-by: Stefano Lattarini 
---
 PLANS/subdir-objects.txt | 10 +++---
 bin/automake.in  | 20 ++--
 lib/am/depend.am |  7 ---
 t/pr224.sh   |  4 ++--
 4 files changed, 19 insertions(+), 22 deletions(-)

diff --git a/PLANS/subdir-objects.txt b/PLANS/subdir-objects.txt
index 3cf6101..c849e33 100644
--- a/PLANS/subdir-objects.txt
+++ b/PLANS/subdir-objects.txt
@@ -5,9 +5,6 @@ We want to make the behaviour currently enabled by the 
'subdir-objects'
 the default one, and in fact the *only* one, in Automake 2.0.
 See automake bug#13378: .
 
-Sadly, **THIS IS IMPOSSIBLE** until automake bug#13928 is resolved:
-http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13928
-
 Details
 ---
 
@@ -53,11 +50,10 @@ We also make sure to avoid the warning when it would be 
irrelevant, i.e.,
 if all source files sit in "current" directory (thanks to Peter Johansson
 for suggesting this).
 
-For some automake 1.x (*before* 2.0 can be released)
-
+For automake 1.16 (*before* 2.0 can be released)
+
 
-Find a proper way to fix the blocking automake bug#13928:
-http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13928
+Submit the pending patch series that fixes http://debbugs.gnu.org/13928
 
 For automake 2.0
 
diff --git a/bin/automake.in b/bin/automake.in
index d8ecca5..21cccbe 100644
--- a/bin/automake.in
+++ b/bin/automake.in
@@ -1242,14 +1242,16 @@ sub handle_languages ()
  INTERNAL);
 define_variable ('am__maybe_remake_depfiles', 'depfiles', 
INTERNAL);
 define_variable ('am__depfiles_remade', "@dep_files", INTERNAL);
-# Generate each 'include' directive individually.  Several make
-# implementations (IRIX 6, Solaris 10, FreeBSD 8) will fail to
-# properly include several files resulting from a variable
-# expansion. Just Generating many separate includes seems thus
-# safest.
 $output_rules .= "\n";
+my @dist_rms;
 foreach my $depfile (@dep_files)
   {
+push @dist_rms, "\t-rm -f $depfile";
+# Generate each 'include' directive individually.  Several
+# make implementations (IRIX 6, Solaris 10, FreeBSD 8) will
+# fail to properly include several files resulting from a
+# variable expansion. Just Generating many separate includes
+# seems thus safest.
 $output_rules .= subst ('AMDEP_TRUE') .
  subst ('am__include') .
  " " .
@@ -1262,11 +1264,9 @@ sub handle_languages ()
 
 require_conf_file ("$am_file.am", FOREIGN, 'depcomp');
 
-# Compute the set of directories to remove in distclean-depend.
-my @dep_dirs = uniq (map { dirname ($_) } @dep_files);
-$output_rules .= file_contents ('depend',
-new Automake::Location,
-DEPDIRS => "@dep_dirs");
+$output_rules .= file_contents (
+'depend', new Automake::Location,
+'DISTRMS' => join ("\n", @dist_rms));
   }
   }
 else
diff --git a/lib/am/depend.am b/lib/am/depend.am
index 0b226b6..3813996 100644
--- a/lib/am/depend.am
+++ b/lib/am/depend.am
@@ -27,8 +27,9 @@ am--depfiles: $(am__depfiles_remade)
 ## erase them in -am or -recursive rules; that would prevent any other
 ## rules from being recursive (for instance multilib clean rules are
 ## recursive).
+if %?DI

bug#13928: [PATCH 2/4] tests: fix some bugs in an XFAILing test

[Stefano Lattarini]

* t/subobj-vpath-pr13928.sh: This one. The test would have failed (or
hung!) even if the bug it was testing against were fixed.

Signed-off-by: Stefano Lattarini 
---
 t/subobj-vpath-pr13928.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/t/subobj-vpath-pr13928.sh b/t/subobj-vpath-pr13928.sh
index 9c78d2e..8c3a6c5 100644
--- a/t/subobj-vpath-pr13928.sh
+++ b/t/subobj-vpath-pr13928.sh
@@ -33,7 +33,7 @@ AUTOMAKE_OPTIONS = subdir-objects
 noinst_PROGRAMS = test
 test_SOURCES = $(srcdir)/test.c
 test-objs:
-   test ! -f $(srcdir)/test.$(OBJEXT)
+   test ! -f '@srcdir@/test.$(OBJEXT)'
test -f test.$(OBJEXT)
 END
 
@@ -42,7 +42,7 @@ $ACLOCAL && $AUTOCONF && $AUTOMAKE -a || fatal_ "autotools 
failed"
 $EGREP 'test\.|DEPDIR|dirstamp|srcdir' Makefile.in || : # For debugging.
 $EGREP '\$.srcdir./test\.[o$]' Makefile.in && exit 1
 $FGREP '$(srcdir)/$(am__dirstamp)' Makefile.in && exit 1
-$FGREP '$(srcdir)/$(DEPDIR)' && exit 1
+$FGREP '$(srcdir)/$(DEPDIR)' Makefile.in && exit 1
 
 cat > test.c << 'END'
 int main (void)
-- 
2.1.3