[PATCH] src/main.c: Add -J to detect number of job slots based on nproc.

[Matt Staveley-Taylor]

Add the -J/--detect-job-slots flag as an shorthand equivalent for
-j$(nproc). The help message is deliberately left ambiguous so that we
could change it to nproc + 1 in future, if desired.

Signed-off-by: Matt Staveley-Taylor 
---
Browsing the mailing list I can see that the behaviour of -j with no
arguments has been discussed a few times. Given it would affect
backwards compatability to change how -j works, introducing a new
flag seems appropriate.

 bootstrap.conf |  3 ++-
 src/main.c | 14 +-
 2 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/bootstrap.conf b/bootstrap.conf
index 98a709a5..9716025e 100644
--- a/bootstrap.conf
+++ b/bootstrap.conf
@@ -62,4 +62,5 @@ getloadavg
 host-cpu-c-abi
 largefile
 make-glob
-make-macros"
+make-macros
+nproc"
diff --git a/src/main.c b/src/main.c
index 15104212..49c329b2 100644
--- a/src/main.c
+++ b/src/main.c
@@ -15,6 +15,7 @@ You should have received a copy of the GNU General Public 
License along with
 this program.  If not, see .  */
 
 #include "makeint.h"
+#include "nproc.h"
 
 #include 
 #if MK_OS_W32
@@ -297,6 +298,9 @@ static int warn_undefined_variables_flag;
 static int always_make_set = 0;
 int always_make_flag = 0;
 
+/* If nonzero, detect the number of job slots based on the number of CPUs.  */
+static int detect_job_slots_flag = 0;
+
 /* If nonzero, we're in the "try to rebuild makefiles" phase.  */
 
 int rebuilding_makefiles = 0;
@@ -360,6 +364,8 @@ static const char *const usage[] =
 N_("\
   --jobserver-style=STYLE Select the style of jobserver to use.\n"),
 N_("\
+  -J, --detect-job-slots  Detect job slots based on number of CPU 
cores.\n"),
+N_("\
   -k, --keep-goingKeep going when some targets can't be made.\n"),
 N_("\
   -l [N], --load-average[=N], --max-load[=N]\n\
@@ -465,6 +471,7 @@ static struct command_switch switches[] =
 { 'E', strlist, &eval_strings, 1, 0, 0, 0, 0, 0, "eval", 0 },
 { 'h', flag, &print_usage_flag, 0, 0, 0, 0, 0, 0, "help", 0 },
 { 'i', flag, &ignore_errors_flag, 1, 1, 0, 0, 0, 0, "ignore-errors", 0 },
+{ 'J', flag, &detect_job_slots_flag, 1, 1, 0, 0, 0, 0, "detect-job-slots", 
0 },
 { 'k', flag, &keep_going_flag, 1, 1, 0, 0, 0, &default_keep_going_flag,
   "keep-going", &keep_going_origin },
 { 'L', flag, &check_symlink_flag, 1, 1, 0, 0, 0, 0, "check-symlink-times", 
0 },
@@ -1616,7 +1623,12 @@ main (int argc, char **argv, char **envp)
 argv_slots = arg_job_slots;
 
 if (arg_job_slots == INVALID_JOB_SLOTS)
-  arg_job_slots = env_slots;
+  {
+if (detect_job_slots_flag)
+  arg_job_slots = num_processors(NPROC_CURRENT_OVERRIDABLE);
+else
+  arg_job_slots = env_slots;
+  }
   }
 
   if (print_usage_flag)
-- 
2.44.0

Re: [PATCH] src/main.c: Add -J to detect number of job slots based on nproc.

[Henrik Carlqvist]

On Fri, 12 Apr 2024 02:13:36 +0100
Matt Staveley-Taylor  wrote:
> Browsing the mailing list I can see that the behaviour of -j with no
> arguments has been discussed a few times. Given it would affect
> backwards compatability to change how -j works, introducing a new
> flag seems appropriate.

Yes, it has been discussed. I would not mind if the default without argument
for -j would be to limit to nproc or nproc+1, instead of like now, creating a
fork bomb when compiling a big project.

Slightly related to this is also a patch contributed by me at
https://savannah.gnu.org/bugs/index.php?51200 which makes it possible to
adjust the number of jobs of an existing make process with SIGUSR2 and SIGUSR1
signals. As it does not seem to have been much interest in those patches I
have instead created a fork of gnu make at
https://github.com/henca/Henriks-make where I have done some continued work.
In the latest version I have limited the default number of processes to 3, but
nproc or nproc+1 would have been a better choice.

I will wait and see how your patch is received. If it gets accepted, I will
simply pull from upstream to my repo at next release. Regardless of if it gets
accepted I might inspired by your code limit the default number of jobs to
nproc instead of now 3 in my fork.

regards Henrik

Re: [PATCH] src/main.c: Add -J to detect number of job slots based on nproc.

[Eli Zaretskii]

> Date: Fri, 12 Apr 2024 07:58:28 +0200
> From: Henrik Carlqvist 
> Cc: bug-make@gnu.org, matt.stav.tay...@gmail.com
> 
> On Fri, 12 Apr 2024 02:13:36 +0100
> Matt Staveley-Taylor  wrote:
> > Browsing the mailing list I can see that the behaviour of -j with no
> > arguments has been discussed a few times. Given it would affect
> > backwards compatability to change how -j works, introducing a new
> > flag seems appropriate.
> 
> Yes, it has been discussed. I would not mind if the default without argument
> for -j would be to limit to nproc or nproc+1, instead of like now, creating a
> fork bomb when compiling a big project.

Isn't nproc or nproc+1 too much?  On systems with hyper-threading,
this will try using too many jobs, and might cause the system be
significantly less responsive.  Maybe nproc/2 is a better default?

Re: [PATCH] src/main.c: Add -J to detect number of job slots based on nproc.

[DSB]

>
> Isn't nproc or nproc+1 too much?  On systems with hyper-threading,
> this will try using too many jobs, and might cause the system be
> significantly less responsive.  Maybe nproc/2 is a better default?
>

This is an interesting question and I suppose the answer depends on many
factors. At my organization we've actually gone the other direction and
default to -j (nproc * 2) on the theory that one of the two jobs is liable
to be sleeping on I/O at any given time. But this is very much a YMMV area
and the builtin default, if there was to be one at all, should be
conservative so nproc * 2 would be a bad choice.

Re: [PATCH] src/main.c: Add -J to detect number of job slots based on nproc.

[Howard Chu via Bug reports and discussion for GNU make]

DSB wrote:
> Isn't nproc or nproc+1 too much?  On systems with hyper-threading,
> this will try using too many jobs, and might cause the system be
> significantly less responsive.  Maybe nproc/2 is a better default?
> 
> 
> This is an interesting question and I suppose the answer depends on many 
> factors. At my organization we've actually gone the other direction and 
> default to -j
> (nproc * 2) on the theory that one of the two jobs is liable to be sleeping 
> on I/O at any given time. But this is very much a YMMV area and the builtin 
> default,
> if there was to be one at all, should be conservative so nproc * 2 would be a 
> bad choice.

I've always used nproc * 1.5. This factor was determined empirically by 
observing idle time on
many builds using nproc * 1 and nproc * 2.

-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: [PATCH] src/main.c: Add -J to detect number of job slots based on nproc.

[Henrik Carlqvist]

> Isn't nproc or nproc+1 too much?  On systems with hyper-threading,
> this will try using too many jobs, and might cause the system be
> significantly less responsive.  Maybe nproc/2 is a better default?

There have been many other good suggestions of choices for make -j and the
choice depends upon your situation.

In your example, when you are considered about responsivness, you are probably
considering a system being used by some user interactively. For such a
situation you might want to consider a low number of parallell jobs and/or
to start make with a nice priority.

The other extreme example would be a system not being used interactively with
only a few low priority, low cpu usage, background processes. If you on such a
system want make to finish as quick as possible you want it to fully utilize
all the resources in a machine. For such situations, I usually use nproc+1,
with the idea that the machine has nproc CPU threads and 1 disk (locally or
remote) for the project. The disk will be fully utilized if 1 or more
processes are waiting for disk, the  CPU threads will be fully utilized if 1
or more processes are waiting for CPU. In either case you will have a
bottleneck. Having more than one process waiting for disk will not increase
IO performance, most likely the IO performance will instead be degraded. The
same applies for processes waiting for CPU which will get slightly less
performance from context switching of unfinished processes.

On a system with a directory like /tmp on a separate disk the performance
might benefit from nproc+2. The same applies if you are reading sources and
writing compiled binaries to different disks.

As allways, your mileage might vary. If the bottleneck during compilation
several times switches between disk and CPU you might benefit from more
processes to fully utilize the resource that currently is not the bottlneck.

Another thing that might be worth to consider is if your CPU(s) are configured
to adjust the frequencies of the cores depending on the load. On one hand, you
might want to keep the cores busy to avoid that they go up and down in
frequency. On the other hand, depending on your CPU, you might be able to
reach some higher turbo frequency on few working cores than max allowed
frequency when all cores are working.

regards Henrik