Re: lang/gambit build parallel
On Sat, Jul 08 2023, Marc Espie wrote: > Actually, I should have documented the current reality ages ago. > (Or replace "jobs" with "direct children" ?) I think "jobs" is fine. ok jca@ > The reference pointing to recursive behavior should be there regardless. > > Index: make.1 > === > RCS file: /cvs/src/usr.bin/make/make.1,v > retrieving revision 1.139 > diff -u -p -r1.139 make.1 > --- make.117 Jan 2023 13:03:22 - 1.139 > +++ make.18 Jul 2023 13:34:35 - > @@ -44,7 +44,7 @@ > .Op Fl d Ar flags > .Op Fl f Ar mk > .Op Fl I Ar directory > -.Op Fl j Ar max_processes > +.Op Fl j Ar max_jobs > .Op Fl m Ar directory > .Op Fl V Ar variable > .Op Ar NAME Ns = Ns Ar value ... > @@ -154,7 +154,7 @@ Extended options are as follows: > Try to be backwards compatible by executing the commands to make > the prerequisites in a target rule in sequence. > This is the default, in the absence of > -.Fl j Ar max_processes . > +.Fl j Ar max_jobs . > .It Fl C Ar directory > Enter > .Ar directory > @@ -241,10 +241,12 @@ Multiple directories can be added to for > Furthermore, the system include path (see the > .Fl m > option) will be used after this search path. > -.It Fl j Ar max_processes > -Specify the maximum number of processes that > +.It Fl j Ar max_jobs > +Specify the maximum number of jobs that > .Nm > may have running at any one time. > +See the discussion about recursive invocations under > +.Sx BUGS . > .It Fl m Ar directory > Specify a directory in which to search for system include files: > .Pa sys.mk > -- jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF DDCC 0DFA 74AE 1524 E7EE
Re: lang/gambit build parallel
In that case, I'll revert the commit. If it ends up passing -j through, we can reconsider enabling it again. -peter On 2023 Jul 08 (Sat) at 14:57:31 +0100 (+0100), Stuart Henderson wrote: :From a very cursory look, it runs child make processes but doesn't pass -j :through to them. : :-- : Sent from a phone, apologies for poor formatting. : :On 8 July 2023 14:15:25 Christian Weisgerber wrote: : :> Theo de Raadt: :> :> > at g2k23 we talked about this issue for a bit. :> :> No, that's not it. The gambit build isn't limited because make :> only starts N jobs with -jN. Rather, there is simply not enough :> work available for it to start N jobs. :> :> -- :> Christian "naddy" Weisgerber na...@mips.inka.de : -- War is menstruation envy.
Re: lang/gambit build parallel
From a very cursory look, it runs child make processes but doesn't pass -j through to them. -- Sent from a phone, apologies for poor formatting. On 8 July 2023 14:15:25 Christian Weisgerber wrote: Theo de Raadt: at g2k23 we talked about this issue for a bit. No, that's not it. The gambit build isn't limited because make only starts N jobs with -jN. Rather, there is simply not enough work available for it to start N jobs. -- Christian "naddy" Weisgerber na...@mips.inka.de
Re: lang/gambit build parallel
Actually, I should have documented the current reality ages ago. (Or replace "jobs" with "direct children" ?) The reference pointing to recursive behavior should be there regardless. Index: make.1 === RCS file: /cvs/src/usr.bin/make/make.1,v retrieving revision 1.139 diff -u -p -r1.139 make.1 --- make.1 17 Jan 2023 13:03:22 - 1.139 +++ make.1 8 Jul 2023 13:34:35 - @@ -44,7 +44,7 @@ .Op Fl d Ar flags .Op Fl f Ar mk .Op Fl I Ar directory -.Op Fl j Ar max_processes +.Op Fl j Ar max_jobs .Op Fl m Ar directory .Op Fl V Ar variable .Op Ar NAME Ns = Ns Ar value ... @@ -154,7 +154,7 @@ Extended options are as follows: Try to be backwards compatible by executing the commands to make the prerequisites in a target rule in sequence. This is the default, in the absence of -.Fl j Ar max_processes . +.Fl j Ar max_jobs . .It Fl C Ar directory Enter .Ar directory @@ -241,10 +241,12 @@ Multiple directories can be added to for Furthermore, the system include path (see the .Fl m option) will be used after this search path. -.It Fl j Ar max_processes -Specify the maximum number of processes that +.It Fl j Ar max_jobs +Specify the maximum number of jobs that .Nm may have running at any one time. +See the discussion about recursive invocations under +.Sx BUGS . .It Fl m Ar directory Specify a directory in which to search for system include files: .Pa sys.mk
Re: lang/gambit build parallel
Theo de Raadt: > at g2k23 we talked about this issue for a bit. No, that's not it. The gambit build isn't limited because make only starts N jobs with -jN. Rather, there is simply not enough work available for it to start N jobs. -- Christian "naddy" Weisgerber na...@mips.inka.de
Re: lang/gambit build parallel
at g2k23 we talked about this issue for a bit. The make(1) manual page can be confusing because it says: -j max_processes Specify the maximum number of processes that make may have running at any one time. Nope, that's wrong. It can create vast number of processes including their sub-processes upon pipelines, so it is very difficult to judge what it will do. We could fix it (with an ugly hack you don't want to hear about), but then -j will probably end up way underutilizing cpus (because it would count even your "sh" processes sitting idle), and then people would probably start running -j 100 on 16 cpu machines and be right back at the same underload vs overload situation. > Peter Hessler: > > > This package takes quite a while to build, so I marked it as parallel to > > use more CPUs. > > This appears inefficient. Right now, I have a gambit build sitting > here, taking up five CPU slots... > > 5*lang/gambit(build) [29274] 26% frozen for 22mn > > ... but running only two cc(1) processes. > > 59060 p1 I+p 0:00.02 | `-- /usr/bin/make build > 97813 p1 I+p 0:00.00 | `-- /bin/sh -ec lock=gambit-4.9.4p0; > export > 74087 p1 I+p 0:00.01 | `-- /usr/bin/make _internal-build > 3321 p1 I+p 0:00.02 | `-- /usr/bin/make do-build > 54023 p1 I+p 0:00.00 | `-- /bin/sh -ec ulimit -s 12288 && > cd > 72554 p1 I+p 0:00.01 | `-- /usr/bin/make mkdir_p=mkdir -p > - > 21475 p1 I+p 0:00.01 | `-- /bin/sh -ec for subdir in > incl > 71313 p1 I+p 0:00.00 | `-- /bin/sh -ec for subdir in > in > 19391 p1 I+p 0:00.05 | `-- /usr/bin/make core > 71280 p1 R+/10 26:59.87 | |-- /usr/bin/cc -B > /usr/obj/ > 4319 p1 R+/926:47.57 | `-- /usr/bin/cc -B > /usr/obj/ > > -- > Christian "naddy" Weisgerber na...@mips.inka.de >
Re: lang/gambit build parallel
Peter Hessler: > This package takes quite a while to build, so I marked it as parallel to > use more CPUs. This appears inefficient. Right now, I have a gambit build sitting here, taking up five CPU slots... 5*lang/gambit(build) [29274] 26% frozen for 22mn ... but running only two cc(1) processes. 59060 p1 I+p 0:00.02 | `-- /usr/bin/make build 97813 p1 I+p 0:00.00 | `-- /bin/sh -ec lock=gambit-4.9.4p0; export 74087 p1 I+p 0:00.01 | `-- /usr/bin/make _internal-build 3321 p1 I+p 0:00.02 | `-- /usr/bin/make do-build 54023 p1 I+p 0:00.00 | `-- /bin/sh -ec ulimit -s 12288 && cd 72554 p1 I+p 0:00.01 | `-- /usr/bin/make mkdir_p=mkdir -p - 21475 p1 I+p 0:00.01 | `-- /bin/sh -ec for subdir in incl 71313 p1 I+p 0:00.00 | `-- /bin/sh -ec for subdir in in 19391 p1 I+p 0:00.05 | `-- /usr/bin/make core 71280 p1 R+/10 26:59.87 | |-- /usr/bin/cc -B /usr/obj/ 4319 p1 R+/926:47.57 | `-- /usr/bin/cc -B /usr/obj/ -- Christian "naddy" Weisgerber na...@mips.inka.de
