RE: make -j does not work on RedHat7.7 VM

[Zhu, Mason]

Hello Smith,

Thank you for the comments. I will carefully check our make file configurations.
Merry Christmas!

Mason

-Original Message-
From: Paul Smith  
Sent: Friday, December 24, 2021 12:52 AM
To: Zhu, Mason ; bug-make@gnu.org
Cc: Gelle, Sreenivasulu ; Zheng, Fred 
; Bedalov, Matthew ; Foster, Casey 
; Levan, Mark 
Subject: Re: make -j does not work on RedHat7.7 VM

On Thu, 2021-12-23 at 12:49 +, Zhu, Mason wrote:
> In GNU Make 3.82, it seems that -j option will be finally added if
> Make determines my VM has the parallel build capability.

It has never been the case that any version of GNU make has
automatically enabled parallel builds by itself.  It's always been
required that the user request it via the -j option.

If you are seeing parallelism in 3.82 then someone provided -j on the
command line or in MAKEFLAGS before starting make.  I don't believe
that GNU make 3.82 allows -j to be added to MAKEFLAGS inside the
makefile and take effect in the current invocation of make.

> However in GNU Make 4.2.1, there is no parallel build if I does not
> explicitly set -jN option.

Does this mean that you discovered a way to make parallel builds work
as expected using GNU make 4.2.1?  Just by adding this argument?
 Exactly how did you do this?

If so we'll need to fully understand how you are enabling parallel
builds in GNU make 3.82 and what change you made to make it work in GNU
make 4.2.1.

Re: make -j does not work on RedHat7.7 VM

[Henrik Carlqvist]

On Thu, 23 Dec 2021 12:49:51 +
"Zhu, Mason"  wrote:
> I checked our project make file. Yes, we are using recursive Make, but does
> not explicitly set -j options in MAKEFLAGS.
> 
> In GNU Make 3.82, it seems that -j option will be finally added if Make
> determines my VM has the parallel build capability. However in GNU Make
> 4.2.1, there is no parallel build if I does not explicitly set -jN option.
> 
> Does it mean that I have to explicitly set -jN in MAKEFLAGS  for GNU Make
> 4.2.1 now?

My guess is that you in your Makefile are making recursive calls to "make"
instead of recursive calls to $(MAKE)

If you failed to call $(MAKE) and instead called "make" the "make" you called
will be the first one in your path, probably not the "make" you intended to
call in this case.

Having one version of make calling another version of make might work less
good for parallell jobs as those version might not be able to communicate
between each other about tyhe parallell jobs.

regards Henrik

Re: make -j does not work on RedHat7.7 VM

[Paul Smith]

On Thu, 2021-12-23 at 12:49 +, Zhu, Mason wrote:
> In GNU Make 3.82, it seems that -j option will be finally added if
> Make determines my VM has the parallel build capability.

It has never been the case that any version of GNU make has
automatically enabled parallel builds by itself.  It's always been
required that the user request it via the -j option.

If you are seeing parallelism in 3.82 then someone provided -j on the
command line or in MAKEFLAGS before starting make.  I don't believe
that GNU make 3.82 allows -j to be added to MAKEFLAGS inside the
makefile and take effect in the current invocation of make.

> However in GNU Make 4.2.1, there is no parallel build if I does not
> explicitly set -jN option.

Does this mean that you discovered a way to make parallel builds work
as expected using GNU make 4.2.1?  Just by adding this argument?
 Exactly how did you do this?

If so we'll need to fully understand how you are enabling parallel
builds in GNU make 3.82 and what change you made to make it work in GNU
make 4.2.1.

Re: make -j does not work on RedHat7.7 VM

[Martin Dorey]

> Could you help to look into it?

Not much, sorry.

> Is this a misconfiguration in my project build environment or a bug in GNU 
> Make 4.2.1?

Maybe you'd be convinced that it's the former if you run a test like this:

martind@sirius:~/tmp/mason-zhu-2021-12-21$ cat Makefile
TARGETS = a b c
default: $(TARGETS)
$(TARGETS):; sleep 10
martind@sirius:~/tmp/mason-zhu-2021-12-21$ tty
/dev/pts/9
martind@sirius:~/tmp/mason-zhu-2021-12-21$ make -j2 & sleep 1; ps -Hjelf | grep 
pts/9; time -p wait
[1] 14504
sleep 10
sleep 10
5 S martind  14286 14280 14280 14280  0  80   0 - 22903 -  13:42 ?
00:00:00   sshd: martind@pts/9
0 S martind  14287 14286 14287 14287  0  80   0 -  3682 -  13:42 pts/9
00:00:00 -bash
0 S martind  14504 14287 14504 14287  0  80   0 -  2211 -  13:44 pts/9
00:00:00   make -j2
0 S martind  14506 14504 14504 14287  0  80   0 -  1611 -  13:44 pts/9
00:00:00 sleep 10
0 S martind  14507 14504 14504 14287  0  80   0 -  1611 -  13:44 pts/9
00:00:00 sleep 10
0 R martind  14508 14287 14508 14287  0  80   0 -  7773 -  13:44 pts/9
00:00:00   ps -Hjelf
0 S martind  14509 14287 14508 14287  0  80   0 -  3349 -  13:44 pts/9
00:00:00   grep pts/9
sleep 10
[1]+  Donemake -j2
real 18.95
user 0.00
sys 0.00
martind@sirius:~/tmp/mason-zhu-2021-12-21$

Two out of the three targets were visibly built in parallel.  Now I'm not 
running make 4.2.1 there and nor am I using RedHat 7.7 on a VM, but it'd be 
more convincing if you ran the test in the afflicted environment.

> Is there anyway to get it work?

Assuredly, but that's going to have to come largely from you.  We're not privy 
to /usr/local/bin/mr, whatever gmake is, much of the invoke_gmk function and 
your other makefiles.  Nor, I'm afraid, would we want to be.  I can point out, 
though, that Make's NEWS file contains the following snippet regarding change 
in the 4.2 era [

* The interface to GNU make's "jobserver" is stable as documented in the
  manual, for tools which may want to access it.

  WARNING: Backward-incompatibility! The internal-only command line option
  --jobserver-fds has been renamed for publishing, to --jobserver-auth.

* The amount of parallelism can be determined by querying MAKEFLAGS, even when
  the job server is enabled (previously MAKEFLAGS would always contain only
  "-j", with no number, when job server was enabled).

].  The manual section alluded to there is 
https://www.gnu.org/software/make/manual/make.html#Job-Slots.  It looks like 
you have a recursive make setup.  That manual section tells us:

> only command lines that make understands to be recursive invocations of make 
> (see How the MAKE Variable 
> Works) will 
> have access to the jobserver

Here's an example where recursive Make processes cooperate to run the desired 
load in parallel, hinting that they're communicating using an environment 
variable, MAKEFLAGS, that names a couple of file descriptors:

martind@sirius:~/tmp/mason-zhu-2021-12-21$ cat Makefile.1
TARGETS = a b c
default: $(TARGETS)
$(TARGETS):; $(MAKE) -f Makefile.2 $@
martind@sirius:~/tmp/mason-zhu-2021-12-21$ cat Makefile.2
TARGETS = a b c
$(TARGETS):; @echo MAKEFLAGS is $(MAKEFLAGS) && sleep 10
martind@sirius:~/tmp/mason-zhu-2021-12-21$ ~/download/make/make -j2 -f 
Makefile.1 & sleep 1; ps -Hjelf | grep pts/9; time -p wait
[1] 28612
/home/martind/download/make/make -f Makefile.2 a
/home/martind/download/make/make -f Makefile.2 b
make[1]: Entering directory '/home/martind/tmp/mason-zhu-2021-12-21'
make[1]: Entering directory '/home/martind/tmp/mason-zhu-2021-12-21'
MAKEFLAGS is -j2 --jobserver-auth=3,4
MAKEFLAGS is -j2 --jobserver-auth=3,4
5 S martind  14286 14280 14280 14280  0  80   0 - 22927 -  13:42 ?
00:00:02   sshd: martind@pts/9
0 S martind  14287 14286 14287 14287  0  80   0 -  3709 -  13:42 pts/9
00:00:00 -bash
0 S martind  28612 14287 28612 14287  0  80   0 -  2214 core_s 14:27 pts/9
00:00:00   /home/martind/download/make/make -j2 -f Makefile.1
0 S martind  28614 28612 28612 14287  0  80   0 -  2214 -  14:27 pts/9
00:00:00 /home/martind/download/make/make -f Makefile.2 a
0 S martind  28616 28614 28612 14287  0  80   0 -  1071 -  14:27 pts/9
00:00:00   /bin/sh -c echo MAKEFLAGS is  -j2 --jobserver-auth=3,4 
&& sleep 10
0 S martind  28618 28616 28612 14287  0  80   0 -  1611 -  14:27 pts/9
00:00:00 sleep 10
0 S martind  28615 28612 28612 14287  0  80   0 -  2214 -  14:27 pts/9
00:00:00 /home/martind/download/make/make -f Makefile.2 b
0 S martind  28617 28615 28612 14287  0  80   0 -  1071 -  14:27 pts/9
00:00:00   /bin/sh -c echo MAKEFLAGS is  -j2 --jobserver-auth=3,4 
&& sleep 10
0 S martind  28619 28617 28612 14287  0  80   0 -  1611 -  14:2