[bug #27714] expansion of $(shell) in target forces serialization of targets

[Mike Frysinger]

URL:
  

 Summary: expansion of $(shell) in target forces
serialization of targets
 Project: make
Submitted by: vapier
Submitted on: Fri 16 Oct 2009 07:50:48 AM GMT
Severity: 3 - Normal
  Item Group: Bug
  Status: None
 Privacy: Public
 Assigned to: None
 Open/Closed: Open
 Discussion Lock: Any
   Component Version: 3.81
Operating System: POSIX-Based
   Fixed Release: None
   Triage Status: None

___

Details:

i noticed a package recently that would build serially even when i gave it -j
(on a quad machine).  looking into it, the reason was that $(shell) was
appended to a compiler flag leading to it being expanded when the target was
run:
CFLAGS += $(shell .)

forcing the $(shell) to be evaluated immediately allowed things to build in
parallel:
SOME_CHECK := $(shell .)
CFLAGS += $(SOME_CHECK)

behavior is seen with make-3.80 and make-3.81

here is a simple example:
$ cat makefile
t = 0 a b c 1 d e f 2 g h i 3 j k l 4 m n o 5 p q r 6
all: $(t)
@echo
$(t):
@printf $@ $(shell :)

$ while make -f makefile -j ; do : ; done
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6

removing the $(shell :) allows for parallel evaluation:
0abc1de2gi3hjkl4mnoqr56p
0abc1de2ghi3jkl4mn5pqor6
0acde21gbhi3jkl4mno5pqr6
0abc1de2ghi3jkl4mn5qr6po
0abc1d2gehi3jkl4o5mpnqr6
0abc1de2ghi3jkl4mno5pq6r
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jl4n5pkqorm6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mnpqor56




___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

[bug #27714] expansion of $(shell) in target forces serialization of targets

[Paul D. Smith]

Follow-up Comment #1, bug #27714 (project make):

My suspicion is that this is just an artifact of your test case.  In your
scenario, make must evaluate the $(shell ...) function (invoking the shell)
while it's expanding the command to be invoked.  What I think is happening
here is that your actual command is so fast that by the time make is ready to
invoke the next command (after it ran the $(shell :) for that target), the
previous command is already completed.  So, it looks like things are happening
serially even though make is doing its best to run them in parallel.

Please try this test again, but have the command you invoke take longer; for
example add a "sleep 1" or something after your printf (not inside the $(shell
...) function!), then print it again maybe.  You should see all the first
prints show up (running in parallel) then all the second prints.

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

[bug #27714] expansion of $(shell) in target forces serialization of targets

[Mike Frysinger]

Follow-up Comment #2, bug #27714 (project make):

i didnt think it a race condition because the original test case had a fairly
lengthy $(shell) code and the targets always ran in the same order (and
obviously only one cpu at a time -- the other cpus stayed idle)

using sleep shows that it is indeed a make bug.  i see one target show up
every 10 seconds:
$ cat makefile
t = 0 a b c 1 d e f 2 g h i 3 j k l 4 m n o 5 p q r 6
all: $(t) ; @echo
$(t): ; @printf $@ $(shell sleep 10)

$ make -f makefile
<10 seconds>0<10 seconds>a<10 seconds>b

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

[bug #27714] expansion of $(shell) in target forces serialization of targets

[Paul D. Smith]

Follow-up Comment #3, bug #27714 (project make):

Hi Mike; no, sorry, I should have given an explicit example.  Your new test
case is not correct and doesn't prove anything.  I explicitly said do NOT put
the sleep inside the $(shell).  It's important to understand that make itself
is single threaded (it doesn't use pthread() etc.).  Its parallelism is gained
by having the recipes it invokes be run in parallel.  But before it can invoke
the recipe it has to expand all the variables and functions.  That includes
$(shell ...) functions.  Variables and functions (including $(shell ...)
functions) are NOT expanded "in parallel"; they are expanded one at a time as
they are seen in the recipe.

After the entire recipe is expanded, THEN make invokes the command, then it
leaves the command running in the background while it locates the next
possible recipe to start, evaluates it, then runs it in the background, etc.

So, causing the recipe expansion to take a long time (which is what you're
doing by adding the sleep to the $(shell ...) function) actually _decreases_
the amount of parallelism.  Hopefully that's clear.

What you need to do to show the make is working correctly is increase the
time it takes the recipe itself to run: that is, add the sleep OUTSIDE the
$(shell ...).  Something like:


$(t): ; @printf $@ $(shell :); sleep 5; printf $@


You should see the recipes all run in parallel (the first printf's all run,
then a pause, then the second printfs all run).

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

[bug #27714] expansion of $(shell) in target forces serialization of targets

[Mike Frysinger]

Follow-up Comment #4, bug #27714 (project make):

i understand what you're saying, but i think the behavior you describe as
"expected" is actually a bug.  deferred variable expansion for a target
shouldnt affect make's ability to run other targets.

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

[bug #27714] expansion of $(shell) in target forces serialization of targets

[Paul D. Smith]

Follow-up Comment #5, bug #27714 (project make):

Hrm.  The only way this could be "fixed" would be to make the $(shell ...)
function a true make job, so that it takes a job token, can run in the
background asynchronously, and is basically handled as an anonymous recipe
line that is run before the "real" recipe line.

This would be almost impossible to do, because it would mean we'd have to
introduce an entirely new concept in make: the ability to have "deferred
variable expansion", where we say "please expand this string" and instead of
getting the expanded string back, we'd get back a response saying "I started
it, come back later".   Then later, sometime, we'd have to check to see if it
completed, and somehow restart our expansion at that time.  Allowing expansion
to happen asynchronously would mean completely rewriting a very significant
portion of make.

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

[bug #27714] expansion of $(shell) in target forces serialization of targets

[Mike Frysinger]

Follow-up Comment #6, bug #27714 (project make):

i dont really know how GNU make is architected, but i dont see why make cant
fork the process to handle the target and then the subprocess does both
deferred expansion and running the commands listed for that target.

i'd also point out that other make implementations (such as BSD makes) handle
this the way i expect -- in parallel.

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

[bug #27714] expansion of $(shell) in target forces serialization of targets

[Paul D. Smith]

Follow-up Comment #7, bug #27714 (project make):

I'm surprised you say you can do this with BSD make; I'm not aware that BSD
make has any feature like the $(shell ...) function, that can appear inside a
recipe.  What syntax do you use for this in BSD make?

We can't expand the recipe in a forked version of GNU make, because the
expansion of the recipe can have side-effects (for example, if an $(eval ...)
is expanded); if the expansion happens in a forked process then those side
effects are not available in the parent.

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

[bug #27714] expansion of $(shell) in target forces serialization of targets

[anonymous]

Follow-up Comment #8, bug #27714 (project make):

# gmake
FOO=   $(echo bar)

# pmake (BSD)
FOO!=   echo bar

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

[bug #27714] expansion of $(shell) in target forces serialization of targets

[Paul D. Smith]

Follow-up Comment #9, bug #27714 (project make):

No, that's not equivalent.  First, your GNU make syntax is wrong; you mean
"FOO = $(shell echo bar)".  And second, as far as I can see from the pmake
documentation "FOO != echo bar" is actually equivalent to "FOO := $(shell echo
bar)", so it doesn't apply to the issue discussed in this bug (which is that
the expansion of the shell happens when the recipe is evaluated, not when the
makefile is read in).

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

[bug #27714] expansion of $(shell) in target forces serialization of targets

[Paul D. Smith]

Update of bug #27714 (project make):

  Status:None => Wont Fix   
 Open/Closed:Open => Closed 

___

Follow-up Comment #10:

I'm going to close this as "won't fix" because I don't see any reasonable way
to implement it that doesn't involve massive changes to the way make works,
and I don't think the capability to have "parallel function expansion" is
worth the effort.

If you want to invoke shell commands in a recipe you don't have to run
$(shell ...)... the _recipe itself_ is run by the shell so why use the shell
function?  Just put the commands that the shell function would run directly
into the recipe.  Then they will be passed to the subshell and run in parallel
just like the rest of the command script.

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/


___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

Re: [bug #27714] expansion of $(shell) in target forces serialization of targets

[Philip Guenther]

On Fri, Oct 16, 2009 at 4:34 AM, Paul D. Smith  wrote:
...
> But before it can invoke the recipe it has to expand all the variables and 
> functions.

I had to ponder that requirement for a bit before it occurred to me
that doing otherwise would either make $(eval) pointless in recipes or
make it an incredible, royal pain to implement.  Ditto for $(error).


Philip Guenther


___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make