Re: Parallel building of Linux Kernel is broken

[Mike Shal]

On Mon, Sep 10, 2018 at 4:18 AM Masahiro Yamada <
yamada.masah...@socionext.com> wrote:

> Hello.
>
>
> Seems no more feedback for this regression report.
>
> OK, the Linux kernel build system is too complicated.
> So, I have come back with a much simpler test-case.
>
> Here, the test-case is only 2 makefiles, less than 50 lines.
>
>
> Please take a look this problem.
>
> As I already reported, the git-bisect points to
>
> commit 2b8e3bb23f96c2458818f011593557d3353dade3
> Author: Paul Smith 
> Date:   Mon Jan 2 14:08:54 2017 -0500
>
> Clean up close-on-exec, particularly with jobserver pipes.
>
>
>
>
> I attached the test case below.
>
> For convenience, this test-case is available from my GitHub repository as
> well:
> https://github.com/masahir0y/make-testcase
>
>
>
> [Test Case]
>
> --(Makefile)---
>
> # If MAKECMDGOALS contains two or more targets, handle them one by one.
> ifneq ($(word 2,$(MAKECMDGOALS)),)
> PHONY += $(MAKECMDGOALS) __build_one_by_one
>
> $(filter-out __build_one_by_one, $(MAKECMDGOALS)): __build_one_by_one
> @:
>
> __build_one_by_one:
> set -e; \
> for i in $(MAKECMDGOALS); do \
> $(MAKE) -f Makefile $$i; \
> done
>
> else
>
> ifeq ($(MAKECMDGOALS),config)
>
> config: FORCE
> touch .config
>
> else
>
> include auto.conf
>
> PHONY += all
> all:
> echo all
>
> auto.conf: .config
> $(MAKE) -f Makefile.config syncconfig
>
> endif
> endif
>
> PHONY += FORCE
> FORCE:
>
> .PHONY: $(PHONY)
> --(Makefile END)---
>
> --(Makefile.config)---
> syncconfig:
> touch auto.conf
> --(Makefile.config END)---
>

It looks like the patch in question changed the default state of the
jobserver tokens to be not inherited. Since this Makefile generates an
included file (auto.conf), the 'make all' invocation creates that file and
then re-invokes itself. However, the re-invoking of make happens in main.c
instead of posixos.c, and is not wrapped in jobserver_pre_child/post_child
which are responsible for updating the jobserver fd inheritance. I'd guess
main.c also needs to call fd_inherit on the jobserver tokens (or use
jobserver_pre_child?) before the re-invocation call. The following hack
seems to fix the issue:

diff --git a/main.c b/main.c
index 5dd539b..83f30f8 100644
--- a/main.c
+++ b/main.c
@@ -2446,7 +2446,9 @@ main (int argc, char **argv, char **envp)
   if (stack_limit.rlim_cur)
 setrlimit (RLIMIT_STACK, _limit);
 #endif
+  jobserver_pre_child(1);
   exec_command ((char **)nargv, environ);
+  jobserver_post_child(1);
 #endif
   free (aargv);
   break;

-Mike
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: [bug #46242] Race condition when input file is updated while compiling

[Mike Shal]

On Tue, Oct 20, 2015 at 3:57 PM, David Boyce 
wrote:

> My view is that this is really a variant of the longstanding
> discussion about getting make to not rely on timestamps at all and use
> MD5 or similar digital signatures instead. There are a number of
> reasons why timestamps can be insufficient beside this race condition,
> so I think it might be better to put effort into getting that working
> than doing more work on the timestamp implementation.
>
> Note that there are published mechanisms for using checksums with
> existing make. For instance
> http://www.cmcrossroads.com/article/rebuilding-when-files-checksum-changes
> .
>
>
I would clarify that the benefit from using the technique in that article
is really because the rebuild algorithm is essentially changed from:

if timestamp(foo.o) < timestamp(foo.c) {rebuild foo.o}

to:

if checksum(foo.c) != last_checksum(foo.c) {rebuild things that depend on
foo.c}

But you can get pretty much all the same benefits without having to hash
everything just by doing:

if timestamp(foo.c) != last_timestamp(foo.c) {rebuild things that depend on
foo.c}

In this last case, the only way things would rebuild incorrectly is if you
managed to change the contents of foo.c but somehow managed to keep its
timestamp identical between runs (which you can force with 'touch -t' or
some such, but doesn't come up in normal usage). Using checksums instead of
timestamps only addresses this additional minor usecase. Changing the
algorithm from a comparison between two files to a comparison between a
file and that same file's last-seen timestamp/checksum/whatever is the
primary improvement.

-Mike
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Make: Unexpected Behavior with Dependencies and include Statement

[Mike Shal]

On Wed, Sep 23, 2015 at 12:57 PM, John Westing  wrote:

> I made a typo in my previous email: The following make file duplicates the
> problem, the one in the previous email does not:
>
> a.d:
> gcc -m32 -MM -o $@ a.c
> sed 's!a.o!$@ a.o!' --in-place $@
>
> a.o:
> gcc -c -m32 -o $@ a.c
>
> all: a.d a.o
>
> -include a.d
>
>
>
See:
https://www.gnu.org/software/make/manual/html_node/Remaking-Makefiles.html#Remaking-Makefiles

Specifically, after reading all the Makefiles (like "a.d", since it's
included), if any of them need to be rebuilt, it will be updated and then
make re-executes itself. You can get a clue that this is happening by
adding $(info Starting make) or something at the top of your Makefile. Then
you'll see:

Starting make
gcc -m32 -MM -o a.d a.c
sed 's!a.o!a.d a.o!' --in-place a.d
Starting make
make: 'a.d' is up to date.

So make runs, sees that a.d needs to be updated, updates it automatically,
restarts, re-reads all the Makfiles, and now it tries to build 'a.d' since
you specified that as a target.

Though normally you don't want to include .d files as targets in your
Makefile. Just generate them as a side-effect of compilation with -MMD or
whatever in gcc and then include them, and you can avoid the whole issue.

-Mike
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Problem with parallel build

[Mike Shal]

On Thu, Oct 21, 2010 at 1:33 PM, Vardhan, Sundara (GE Energy)
sundara.vard...@ge.com wrote:
 Hi Michael

 During parallel makes if the time of the object files created and the
 time archive command is run is within a second then it will fail to
 include these object files. The reason I believe is due to a bug in make
 rules wherein the time granularity is 1 second. Try to introduce a sleep
 of 1 second before executing the archive command.


Really? I don't think I've ever witnessed 'ar' failing to include
object files that were just created. Do you have a separate test case
that reproduces that behavior?

In any case that doesn't seem to the problem in Michael's post. The
final link command is failing to find app1.o. The archive is already
built successfully. The app1.o in this example doesn't actually use
anything from the library, so even if lib.a was missing some object
files I think the link would still succeed (assuming app1.o was
created before the link command).

I don't see why Michael's example should fail, but I can reproduce the
behavior with 3.81 and 3.82.

Also Michael, just a heads-up that in your example the Compile rule
isn't actually used, since src/%.c doesn't match anything (there's no
src/ directory). Removing src/ from that line doesn't fix the problem,
though.

-Mike

___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

Re: Shorter and less error-prone rule for automatic prerequisite generation in the GNU Make manual

[Mike Shal]

On 4/29/10, Edward Welbourne e...@opera.com wrote:
  If an update to new source code, that would compile just fine in a clean
   checkout, breaks the incremental build, the build system is errornuous.


 I would like to agree with you, but this constraint is, in general,
  incompatible with incremental building, which is too good a benefit to
  throw away.

This is true for make, but not true for build systems in general. It
is entirely possible to have minimal incremental builds and avoid the
use of a manual clean to support cases where files have been moved
or deleted. I agree with Robert's position that the build system is
broken if an incremental build fails where a full build would succeed
after an update.

-Mike


___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

Re: Make with -j8 sometime fail on a single-core machine

[Mike Shal]

On 11/19/09, He Yunlong-B20256 b20...@freescale.com wrote:
 Hi, Mike,

 Thanks very much for your patient clarification, is one task
  only done by one job?

I'm not sure what you mean by task here. Make has a concept of a
job, as you can see by searching around the help documentation. I
think each job will be executing one command script for a particular
rule at any given time, and the command script itself is executed
serially. The parallelization comes in when you have multiple jobs, so
multiple command scripts can be executed simultaneously.


 We are working with many opensource packages, it's really a
  challenge to verify and update all makefiles, it seems such issues are
  very general.

I agree that it is a challenge, and a very general problem.

-Mike


___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

Re: Make with -j8 sometime fail on a single-core machine

[Mike Shal]

On 11/19/09, He Yunlong-B20256 b20...@freescale.com wrote:


 Hi, Experts:

 We met build errors with [gdb] make -j8 for serveral times, but it's
 ok to build with -j1, we wonder whether this is some known issue, and is
 it possible that something wrong in Makefile? We are using gnu make 3.81.


Hello,

Make doesn't guarantee that the build is parallel-safe. Consider this example:

$ cat Makefile
all: a b; echo all done!
a: ; echo hello  $@
b: ; cp a b

clean: ; rm -f a b

It works fine if you run with -j1, since a will be built before
b. With -j2, it is possible that both jobs for a and b will
start at the same time, so the command to create a may not have
finished when the command to create b tries to read it.

The problem of course is that the command for b is reading from the
file a, which is created by another rule. These dependencies need to
always be specified in the Makefile, or otherwise be implicitly
ordered properly. The solution is to add that dependency to the
Makefile, and then it will always run correctly with -jX. However, it
can be tricky to find out what those missing dependencies are or where
they need to go, since you won't find out from make.

-Mike


___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

Re: Enhancement request: Watch source files for changes

[Mike Shal]

On 1/24/09, Noah Slater nsla...@gnu.org wrote:
 Hey,

  I have been using make to regenerate output documents as I write them in a
  source format. This might involve writing in AsciiDoc and having make pick up
  changes, generating a HTML page for me to preview my work.

  To aid the writing process I usually create a shell script like this:

   #!/bin/sh -e

   while true; do
 make 21 | grep -vi nothing to be done || true
 sleep 1
   done

  This means that as I write, and save the file, my HTML is generated for me.

  When I think about it, there are many other situations where I can imagine it
  would be useful to have make watch the input files for changes, updating the
  output files when is appropriate.

Hi Noah,

If you're running on a Linux machine, you might want to give tup a try:

http://gittup.org/tup/

It's a build system based around a file monitor already (using
inotify, hence the Linux-only requirement), so an autoupdate
functionality sorta naturally followed. Assuming it works for you
(which may be a big assumption - it's still pretty err... rough around
the edges), it would be able to do what you're looking for without
continually re-invoking a program every second. It also does a few
other nice things, like delete old output files. For example, you
could rename a webpage, and it would re-generate the output with the
new name while getting rid of the old one. Of course, being fairly
experimental at this point, it's entirely possible it won't do what
you want or cause things to explode.

Anyway, if you want to give it a shot feel free to send me a private
email, as it's not technically related to GNU make, and the
documentation probably isn't complete enough for someone unfamiliar
with it to get started. I just posted the web pages a few minutes ago,
so you would be wading into uncharted territory :).

Hope it helps,
-Mike


___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

Re: automatic prerequisite problems

[Mike Shal]

On 11/11/05, Paul D. Smith [EMAIL PROTECTED] wrote:
 %% Marty Leisner [EMAIL PROTECTED] writes:

   ml ifneq (,$(findstring clean,$(MAKECMDGOALS)))
   ml NO_DEPENDS=1
   ml endif

   ml ifndef NO_DEPENDS
   ml -include $(DEPS)
   ml endif

   ml I've seen this annoying behavior for years...is there a better
   ml wah t handle this?

 No fundamentally better way.  You can be a bit more restrictive of what
 you match on by using filter/filter-out for pattern matching instead of
 findstring, but that's about it.


One other possible solution is to remove the rule to generate the .d
files (but still include them) and just generate the .d's during the
.cc - .o rule. Since make doesn't have a rule to create the .d file,
it won't try to create them if they don't exist when you do the
include. Gcc can generate the .d file in addition to the .o file
pretty easily just by passing in the -MMD flag.

Unfortunately, what gcc actually does with the .d file (as in, which
directory it gets stored, what it lists as the target, etc) has
changed somewhat drastically between gcc versions (IIRC, between 3.0
- 3.3 is pretty wacky). So if you know what gcc version you're
building with it can be pretty simple. But if you want to be nice and
work with a bunch versions (ie: for distribution to others), you'd be
stuck writing a bunch of special cases in order to get correct
dependencies. In that case, what you have is probably simplest, unless
someone else has a better way to get around those flags :)

-Mike


___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make