Re: [PATCH] docs: other issues with parallel BSD make
On Thursday 18 August 2011, Eric Blake wrote: > On 08/18/2011 12:51 PM, Stefano Lattarini wrote: > > OK, given your considerations, I've updated my patch with the attached > > squash-in. The amended patch is attached too. Let me know if it is > > good to apply now. > > I think it's good enough now; go ahead and push, and if we come up with > any further tweaks, it can be separate patches. > > Thanks again for chasing this down. > Pushed now. Thanks, Stefano
Re: [PATCH] docs: other issues with parallel BSD make (was: Re: bug#9245: FreeBSD make in concurrent mode report spurious success in automake-generated tests harness)
On Thu, 18 Aug 2011, Stefano Lattarini wrote: OK, given your considerations, I've updated my patch with the attached squash-in. The amended patch is attached too. Let me know if it is good to apply now. This seems much better to me. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: [PATCH] docs: other issues with parallel BSD make
On 08/18/2011 12:51 PM, Stefano Lattarini wrote: Perhaps specifying the OS release version the issue was noticed on (not all-inclusive) or the date that the issue was noticed would be sufficient? For example "FreeBSD 8.2" rather than "FreeBSD"? or "FreeBSD (2011)" rather than "FreeBSD"? The drawback of this might be that someone may assume that the issues are particular to that release only. Hardly anyone reads the raw Texinfo so comments in the Texinfo do not seem very useful for the common reader. If the reader knows that the current version of FreeBSD is 12.8, then he might want to check to see if claims made about 8.2 (totally defunct by then) are still applicable. Comments in the raw Texinfo from the authors regarding versions when the issue was checked (or rechecked) would be useful. Bob OK, given your considerations, I've updated my patch with the attached squash-in. The amended patch is attached too. Let me know if it is good to apply now. I think it's good enough now; go ahead and push, and if we come up with any further tweaks, it can be separate patches. Thanks again for chasing this down. -- Eric Blake ebl...@redhat.com+1-801-349-2682 Libvirt virtualization library http://libvirt.org
Re: [PATCH] docs: other issues with parallel BSD make (was: Re: bug#9245: FreeBSD make in concurrent mode report spurious success in automake-generated tests harness)
On Tuesday 16 August 2011, Bob Friesenhahn wrote:
> On Tue, 16 Aug 2011, Stefano Lattarini wrote:
> >>
> >> What is useful information today may become 'lore' in a few years so
> >> it would be good to add additional data so that the reader (and
> >> documentation maintainer) knows the vintage of the information.
> >>
> > That's a good point. Do you think it would be OK to put such information
> > only in Texinfo comments for the moment, and then, as a second and later
> > step, devise a way to report it in the final manual too? This second step
> > wouldn't be trivial IMHO, since we would need to present such version
> > information in a way that is at the same time clear, non-obtrusive and
> > complete (mabe we could take a look at how Gnulib does this?). Finally,
> > as a third step, we might try to revisit the existing examples of bugs
> > and portability pitfalls, and try to pin-point them to a precise version
> > of the system/tools used (in case this version is not already reported).
>
> Perhaps specifying the OS release version the issue was noticed on
> (not all-inclusive) or the date that the issue was noticed would be
> sufficient? For example "FreeBSD 8.2" rather than "FreeBSD"? or
> "FreeBSD (2011)" rather than "FreeBSD"? The drawback of this might be
> that someone may assume that the issues are particular to that release
> only.
>
> Hardly anyone reads the raw Texinfo so comments in the Texinfo do not
> seem very useful for the common reader. If the reader knows that the
> current version of FreeBSD is 12.8, then he might want to check to see
> if claims made about 8.2 (totally defunct by then) are still
> applicable. Comments in the raw Texinfo from the authors regarding
> versions when the issue was checked (or rechecked) would be useful.
>
> Bob
>
OK, given your considerations, I've updated my patch with the attached
squash-in. The amended patch is attached too. Let me know if it is
good to apply now.
Regards,
Stefano
diff --git a/doc/autoconf.texi b/doc/autoconf.texi
index b2b0a59..a0f7a55 100644
--- a/doc/autoconf.texi
+++ b/doc/autoconf.texi
@@ -19838,7 +19838,8 @@ all:
@end example
@noindent
-may output the following with @code{make -j1}:
+may output the following with @code{make -j1}, at least on NetBSD up to
+5.1 and FreeBSD up to 8.2:
@example
/
@@ -19877,17 +19878,18 @@ read:
@c $$ @c restore font-lock
$ @kbd{echo foo | make read}
LINE: foo
-$ @kbd{echo foo | make -j1 read}
+$ @kbd{echo foo | make -j1 read} # NetBSD 5.1 and FreeBSD 8.2
LINE:
@end example
@noindent
-Moreover, when FreeBSD @command{make} is run in parallel mode, it
-implements the @code{@@} and @code{-} ``recipe modifiers'' by dynamically
-modifying the active shell flags. This behaviour has the effects of
-potentially clobbering the exit status of recipes silenced with the
-@code{@@} modifier if they also unset the @option{errexit} shell flag,
-and of mangling the output in unexpected ways:
+Moreover, when FreeBSD @command{make} (up at least to 8.2) is run in
+parallel mode, it implements the @code{@@} and @code{-} ``recipe
+modifiers'' by dynamically modifying the active shell flags. This
+behaviour has the effects of potentially clobbering the exit status
+of recipes silenced with the @code{@@} modifier if they also unset
+the @option{errexit} shell flag, and of mangling the output in
+unexpected ways:
@example
$ @kbd{cat Makefile}
@@ -19928,12 +19930,12 @@ standard output is connected to a tty, the recipe command will not be:
$ @kbd{cat Makefile}
all:
@@test -t 1 && echo "Is a tty" || echo "Is not a tty"
-$ @kbd{make -j 2} # FreeBSD make
+$ @kbd{make -j 2} # FreeBSD 8.2 make
Is not a tty
-$ @kbd{make -j 2} # NetBSD make
+$ @kbd{make -j 2} # NetBSD 5.1 make
--- all ---
Is not a tty
-$ @kbd{dmake -j 2} # Solaris dmake
+$ @kbd{dmake -j 2} # Solaris 10 dmake
@var{hostname} --> 1 job
@var{hostname} --> Job output
Is not a tty
From a621f6fb03ee2ce45bafd882b17dafc9a8d0bdeb Mon Sep 17 00:00:00 2001
Message-Id:
From: Stefano Lattarini
Date: Tue, 16 Aug 2011 18:31:42 +0200
Subject: [PATCH] docs: other issues with parallel BSD make
Motivated by automake bug#9245:
<http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9245>
and FreeBSD PR bin/159730:
<http://www.freebsd.org/cgi/query-pr.cgi?pr=159730>
* doc/autoconf.texi (Parallel Make): Document other BSD make
incompatibilities. Reorganize the existing related documentation
accordingly.
---
ChangeLog | 11
doc/autoconf.texi | 141 +++-
2 files changed, 106 insertions(+), 46 deletions(-)
diff --git a/ChangeLog b/ChangeLog
index 6ca51fd..6e36455 100644
--- a/ChangeLog
Re: [PATCH] docs: other issues with parallel BSD make (was: Re: bug#9245: FreeBSD make in concurrent mode report spurious success in automake-generated tests harness)
On Tue, 16 Aug 2011, Stefano Lattarini wrote: What is useful information today may become 'lore' in a few years so it would be good to add additional data so that the reader (and documentation maintainer) knows the vintage of the information. That's a good point. Do you think it would be OK to put such information only in Texinfo comments for the moment, and then, as a second and later step, devise a way to report it in the final manual too? This second step wouldn't be trivial IMHO, since we would need to present such version information in a way that is at the same time clear, non-obtrusive and complete (mabe we could take a look at how Gnulib does this?). Finally, as a third step, we might try to revisit the existing examples of bugs and portability pitfalls, and try to pin-point them to a precise version of the system/tools used (in case this version is not already reported). Perhaps specifying the OS release version the issue was noticed on (not all-inclusive) or the date that the issue was noticed would be sufficient? For example "FreeBSD 8.2" rather than "FreeBSD"? or "FreeBSD (2011)" rather than "FreeBSD"? The drawback of this might be that someone may assume that the issues are particular to that release only. Hardly anyone reads the raw Texinfo so comments in the Texinfo do not seem very useful for the common reader. If the reader knows that the current version of FreeBSD is 12.8, then he might want to check to see if claims made about 8.2 (totally defunct by then) are still applicable. Comments in the raw Texinfo from the authors regarding versions when the issue was checked (or rechecked) would be useful. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: [PATCH] docs: other issues with parallel BSD make (was: Re: bug#9245: FreeBSD make in concurrent mode report spurious success in automake-generated tests harness)
Hi Bob. On Tuesday 16 August 2011, Bob Friesenhahn wrote: > On Tue, 16 Aug 2011, Stefano Lattarini wrote: > >>> > >> I'll have a "draft patch" read soonish. There is ample room for > >> improvements, > >> but I'll post it here anyway since it can benefit from early feedback. > >> > > Here it is. As usual, comments and suggestions welcome. > > The proposed documentation seems quite useful. It does have a flaw in > that it identifies 'make' programs based on the operating system where > they were currently found (e.g. "FreeBSD make"). The issues may > pertain to only certain versions of such make programs, or the 'make' > associated with an OS may be entirely supplanted (or optionally > replaced) with a 'make' which offers completely different behavior. > > What is useful information today may become 'lore' in a few years so > it would be good to add additional data so that the reader (and > documentation maintainer) knows the vintage of the information. > That's a good point. Do you think it would be OK to put such information only in Texinfo comments for the moment, and then, as a second and later step, devise a way to report it in the final manual too? This second step wouldn't be trivial IMHO, since we would need to present such version information in a way that is at the same time clear, non-obtrusive and complete (mabe we could take a look at how Gnulib does this?). Finally, as a third step, we might try to revisit the existing examples of bugs and portability pitfalls, and try to pin-point them to a precise version of the system/tools used (in case this version is not already reported). Regards, Stefano
Re: [PATCH] docs: other issues with parallel BSD make (was: Re: bug#9245: FreeBSD make in concurrent mode report spurious success in automake-generated tests harness)
On Tue, 16 Aug 2011, Stefano Lattarini wrote: I'll have a "draft patch" read soonish. There is ample room for improvements, but I'll post it here anyway since it can benefit from early feedback. Here it is. As usual, comments and suggestions welcome. The proposed documentation seems quite useful. It does have a flaw in that it identifies 'make' programs based on the operating system where they were currently found (e.g. "FreeBSD make"). The issues may pertain to only certain versions of such make programs, or the 'make' associated with an OS may be entirely supplanted (or optionally replaced) with a 'make' which offers completely different behavior. What is useful information today may become 'lore' in a few years so it would be good to add additional data so that the reader (and documentation maintainer) knows the vintage of the information. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
[PATCH] docs: other issues with parallel BSD make (was: Re: bug#9245: FreeBSD make in concurrent mode report spurious success in automake-generated tests harness)
[dropping bug-automake, adding autoconf-patches]
References:
<http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9245>
<http://www.freebsd.org/cgi/query-pr.cgi?pr=159730>
On Tuesday 16 August 2011, Stefano Lattarini wrote:
> Hi Eric.
>
> On Tuesday 16 August 2011, Eric Blake wrote:
> > On 08/16/2011 10:04 AM, Stefano Lattarini wrote:
> > > The "proper fix" is even easier BTW: just use "$$exit || exit 1" instead.
> > >
> > > The updated patch should now work (and I've tested it properly this time).
> > > This new patch hasn't been derived by "cargo-cult guessing", but from the
> > > explanation of the relevant FreeBSD make bug provided by Jilles Tjoelker:
> > > <http://www.freebsd.org/cgi/query-pr.cgi?pr=159730>
> > >
> > > Now that's something to be documented in the Autoconf manual I'd say ...
> >
> > Indeed. So it all boils down to BSD make rewriting rules (but only for
> > make -j) to be roughly:
> >
> > sh -c 'set -v; your rules; set +v'
> >
> > and losing your exit status (not to mention sharing shell status between
> > commands that are supposed to be in independent shells), when it really
> > should have been rewriting as something more complex:
> >
> > sh -c '(your rule 1); (your rule 2); st=$?; set +v; exit $st'
> >
> > And your workaround of an explicit exit rather than relying on $? at the
> > end of your 'set +e' invocation is indeed the proper way to force the
> > parallel make bug to be avoided, by exiting prior to any post-commands
> > injected by the BSD make rewriting going on that corrupt $?.
> >
> I'll have a "draft patch" read soonish. There is ample room for improvements,
> but I'll post it here anyway since it can benefit from early feedback.
>
Here it is. As usual, comments and suggestions welcome.
Regards,
Stefano
From a60b30ea47b2cc36d0b85bbed94a106643fba745 Mon Sep 17 00:00:00 2001
Message-Id:
From: Stefano Lattarini
Date: Tue, 16 Aug 2011 18:31:42 +0200
Subject: [PATCH] docs: other issues with parallel BSD make
Motivated by automake bug#9245:
<http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9245>
and FreeBSD PR bin/159730:
<http://www.freebsd.org/cgi/query-pr.cgi?pr=159730>
* doc/autoconf.texi (Parallel Make): Document other BSD make
incompatibilities. Reorganize the existing related documentation
accordingly.
---
ChangeLog | 11
doc/autoconf.texi | 137 +++-
2 files changed, 103 insertions(+), 45 deletions(-)
diff --git a/ChangeLog b/ChangeLog
index 6ca51fd..6e36455 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,14 @@
+2011-08-16 Stefano Lattarini
+
+ docs: other issues with parallel BSD make
+ Motivated by automake bug#9245:
+ <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9245>
+ and FreeBSD PR bin/159730:
+ <http://www.freebsd.org/cgi/query-pr.cgi?pr=159730>
+ * doc/autoconf.texi (Parallel Make): Document other BSD make
+ incompatibilities. Reorganize the existing related documentation
+ accordingly.
+
2011-08-08 Stefano Lattarini
docs: fix minor typos
diff --git a/doc/autoconf.texi b/doc/autoconf.texi
index 7c1818d..b2b0a59 100644
--- a/doc/autoconf.texi
+++ b/doc/autoconf.texi
@@ -19826,45 +19826,10 @@ sh
Support for parallel execution in @command{make} implementation varies.
Generally, using GNU make is your best bet.
-Some make implementations (among them, FreeBSD @command{make}, NetBSD
-@command{make}, and Solaris @command{dmake}), when invoked with a
-@option{-j@var{N}} option, connect the standard output and standard
-error of all their child processes to pipes or temporary regular
-files. This can lead to subtly different semantics in the behaviour
-of the spawned processes. For example, even if the @command{make}
-standard output is connected to a tty, the recipe command will not be:
-
-@example
-$ @kbd{cat Makefile}
-all:
-@@test -t 1 && echo "Is a tty" || echo "Is not a tty"
-$ @kbd{make -j 2} # FreeBSD make
-Is not a tty
-$ @kbd{make -j 2} # NetBSD make
all ---
-Is not a tty
-$ @kbd{dmake -j 2} # Solaris dmake
-@var{hostname} --> 1 job
-@var{hostname} --> Job output
-Is not a tty
-@end example
-
-@noindent
-On the other hand:
-
-@example
-$ @kbd{make -j 2} # GNU make, Heirloom make
-Is a tty
-@end example
-
-Furthermore, parallel runs of those @command{make} implementations will
-route standard error from commands that they spawn into their own
-standard output, and may remove leading whitespace from output lines.
-
When NetBSD or FreeBSD @command{make} are run in parallel mode, they will
reuse the same shell for multiple commands within one recipe. This can
-have
