Fwd: multi-threaded compiling

uggest? Is this the best bash >>> can do? >>> >>> >>> On Thu, Mar 14, 2024 at 3:47 AM Martin D Kealey >>> wrote: >>> >>>> On Wed, 13 Mar 2024, Mischa Baars wrote: >>>> >>>> > Date: Wed, 13 Mar 2024 22:52:1

Re: multi-threaded compiling

;> >> > Date: Wed, 13 Mar 2024 22:52:16 >> > From: Mischa Baars >> > To: alex xmb sw ratchev >> > Cc: psm...@gnu.org, bug-bash , help-make@gnu.org >> > Subject: Re: multi-threaded compiling >> > >> > I found another mistake of mine:

Re: multi-threaded compiling

On Wed, 13 Mar 2024, Mischa Baars wrote: > > > Date: Wed, 13 Mar 2024 22:52:16 > > From: Mischa Baars > > To: alex xmb sw ratchev > > Cc: psm...@gnu.org, bug-bash , help-make@gnu.org > > Subject: Re: multi-threaded compiling > > > > I found another

Re: multi-threaded compiling

I found another mistake of mine: #ifndef __STRINGIZED__ should be #if __STRINGIZED__ == 0 or it will always go for the second statement in the conditional. Can someone please correct my script??? Things are starting to itch. On Wed, Mar 13, 2024 at 8:44 AM alex xmb sw ratchev wrote: > > > O

Re: multi-threaded compiling

On Wed, Mar 13, 2024, 08:26 Mischa Baars wrote: > On Tue, Mar 12, 2024 at 10:00 PM Paul Smith wrote: > > > On Tue, 2024-03-12 at 13:37 +0100, Mischa Baars wrote: > > > > I'd still like to hear why you aren't simply using "make -j". > > > > > > That's because I don't want to define static compile

Re: multi-threaded compiling

On Wed, Mar 13, 2024 at 4:14 AM Martin D Kealey wrote: > > On Mon, Mar 11, 2024 at 8:20 PM Chet Ramey wrote: >> > On 3/11/24 2:50 PM, Mischa Baars wrote: >> > > Which sort of brings us back to the original question I suppose. Who >> does >> > > that line of code function from a script and why do

Re: multi-threaded compiling

On Tue, Mar 12, 2024 at 10:00 PM Paul Smith wrote: > On Tue, 2024-03-12 at 13:37 +0100, Mischa Baars wrote: > > > I'd still like to hear why you aren't simply using "make -j". > > > > That's because I don't want to define static compile and link targets > > for every new project I start. The Make

Re: multi-threaded compiling

> On Mon, Mar 11, 2024 at 8:20 PM Chet Ramey wrote: > > On 3/11/24 2:50 PM, Mischa Baars wrote: > > > Which sort of brings us back to the original question I suppose. Who > does > > > that line of code function from a script and why does it fail from the > > > command line? > > > > Job control and

Re: multi-threaded compiling

On Tue, 2024-03-12 at 13:37 +0100, Mischa Baars wrote: > > I'd still like to hear why you aren't simply using "make -j". > > That's because I don't want to define static compile and link targets > for every new project I start. The Makefile in question contains only > a few lines of code and some

Re: multi-threaded compiling

On 3/12/24 4:23 AM, Mischa Baars wrote: So... how do I fix it? If you want double quotes in a variable expansion to have some meaning to the parser, you need to run the expanded value through the parser, using something like `eval'. Or you figure out a different way to glue the arguments toget

Re: multi-threaded compiling

In section one, the problem is that "wait -n" does not do what you think it does. (Lots of us think this behaviour is broken, and it may be fixed in an upcoming version of Bash.) You don't need '-n' when you specify a PID; the fix is simply to remove it. In section two, the problem is that quote r

Re: multi-threaded compiling

On Tue, Mar 12, 2024 at 09:42:22AM +0100, Mischa Baars wrote: > Here's the script and the Makefile using "printf '<%s>'": Sadly, your mail user agent chose to attach "Makefile" with content-type application/octet-stream, which my MUA (mutt) refuses to show inline, or to include in a reply as quote

Re: multi-threaded compiling

On Tue, Mar 12, 2024 at 12:49 PM Greg Wooledge wrote: > On Tue, Mar 12, 2024 at 09:42:22AM +0100, Mischa Baars wrote: > > Here's the script and the Makefile using "printf '<%s>'": > > Sadly, your mail user agent chose to attach "Makefile" with content-type > application/octet-stream, which my MUA

Re: multi-threaded compiling

On Tue, Mar 12, 2024, 12:49 Greg Wooledge wrote: > On Tue, Mar 12, 2024 at 09:42:22AM +0100, Mischa Baars wrote: > > Here's the script and the Makefile using "printf '<%s>'": > > Sadly, your mail user agent chose to attach "Makefile" with content-type > application/octet-stream, which my MUA (mut

Re: multi-threaded compiling

Got it!!! bash -c 'set +m; seconds=5; for (( i=0;i<32;i++ )); do exit ${i} & pid[${i}]=${!}; done; sleep ${seconds}; for (( i=0;i<32;i++ )); do wait ${pid[${i}]}; e=${?}; echo "$(printf %3u ${i}) pid ${pid[${i}]} exit ${e}"; done;' One problem solved :) The problem was exactly the -n, and had not

Re: multi-threaded compiling

Oh, sorry. Without the -n that is. bash -c 'set +m; seconds=1; for (( i=0;i<32;i++ )); do exit ${i} & done; sleep ${seconds}; for (( i=0;i<32;i++ )); do wait -p pid; e=${?}; echo "$(printf %3u ${i}) pid ${pid} exit ${e}"; done;' bash -c 'set -m; seconds=1; for (( i=0;i<32;i++ )); do exit ${i} & do

Re: multi-threaded compiling

Hi Chet, The only thing I can find about this behavior is the set +m / set -m option, which disables / enables the notifications, but has no effect otherwise. The line from the script won't execute from the command line with either option, neither enclosed with a 'bash -c' as in: set +m; bash -c

Re: multi-threaded compiling

Here's the script and the Makefile using "printf '<%s>'": On Tue, Mar 12, 2024 at 9:32 AM Martin D Kealey wrote: > In section one, the problem is that "wait -n" does not do what you think > it does. (Lots of us think this behaviour is broken, and it may be fixed in > an upcoming version of Bash.

Re: multi-threaded compiling

On Tue, Mar 12, 2024 at 9:32 AM Martin D Kealey wrote: > In section one, the problem is that "wait -n" does not do what you think > it does. (Lots of us think this behaviour is broken, and it may be fixed in > an upcoming version of Bash.) You don't need '-n' when you specify a PID; > the fix is

Re: multi-threaded compiling

On Tue, Mar 12, 2024, 09:26 Mischa Baars wrote: > On Mon, Mar 11, 2024 at 10:26 PM Chet Ramey wrote: > > > On 3/11/24 3:44 PM, Mischa Baars wrote: > > > On Mon, 11 Mar 2024, 20:20 Chet Ramey, > > > wrote: > > > > > > On 3/11/24 2:50 PM, Mischa Baars wrote: > > >

Re: multi-threaded compiling

On Mon, Mar 11, 2024 at 10:26 PM Chet Ramey wrote: > On 3/11/24 3:44 PM, Mischa Baars wrote: > > On Mon, 11 Mar 2024, 20:20 Chet Ramey, > > wrote: > > > > On 3/11/24 2:50 PM, Mischa Baars wrote: > > > Which sort of brings us back to the original question I su

Re: multi-threaded compiling

a version with success or fail handling ~ $ bash xmb.smallt.2 2/2 1/1 ~ $ cat xmb.smallt.2 #!/bin/bash unset -v success fail cmd a e run() { # uaage , $1 successcase cmd , $2 fail cmd , $3 and further actual cmd to run local IFS=' ' eval "${*:3} &" success[$!]=$1 fail[$!]=$2 cmd[$!]=${*:3} }

Re: multi-threaded compiling

On Mon, Mar 11, 2024, 22:40 alex xmb sw ratchev wrote: > > > On Mon, Mar 11, 2024, 22:36 alex xmb sw ratchev wrote: > >> ~ $ bash xmb.smallt >> pid 14333 cmd t2 returned 3 >> pid 14332 cmd sleep 1 returned 0 >> >> ~ $ cat xmb.smallt >> #!/bin/bash >> >> run() { >> local IFS=' ' run=$* >> eval "

Re: multi-threaded compiling

On Mon, Mar 11, 2024, 22:40 alex xmb sw ratchev wrote: > > > On Mon, Mar 11, 2024, 22:36 alex xmb sw ratchev wrote: > >> ~ $ bash xmb.smallt >> pid 14333 cmd t2 returned 3 >> pid 14332 cmd sleep 1 returned 0 >> >> ~ $ cat xmb.smallt >> #!/bin/bash >> >> run() { >> local IFS=' ' run=$* >> eval "

Re: multi-threaded compiling

On Mon, Mar 11, 2024, 22:36 alex xmb sw ratchev wrote: > ~ $ bash xmb.smallt > pid 14333 cmd t2 returned 3 > pid 14332 cmd sleep 1 returned 0 > > ~ $ cat xmb.smallt > #!/bin/bash > > run() { > local IFS=' ' run=$* > eval "$run &" > me[$!]=$run > } > alternativly to mention aliases usage shopt

Re: multi-threaded compiling

~ $ bash xmb.smallt pid 14333 cmd t2 returned 3 pid 14332 cmd sleep 1 returned 0 ~ $ cat xmb.smallt #!/bin/bash run() { local IFS=' ' run=$* eval "$run &" me[$!]=$run } wa() { local pid wait -n -p pid printf %s\\n "pid $pid cmd ${me[pid]} returned $?" } t2() { sleep .75 return 3 } run sl

Re: multi-threaded compiling

On Mon, Mar 11, 2024 at 10:19:26PM +0100, Mischa Baars wrote: > On Mon, 11 Mar 2024, 21:08 Kerin Millar, wrote: > > The pid can be obtained with the -p option, as of 5.1. Below is a > > synthetic example of how it might be put into practice. I'd forgotten about that one. A recent addition, and o

Re: multi-threaded compiling

On 3/11/24 3:44 PM, Mischa Baars wrote: On Mon, 11 Mar 2024, 20:20 Chet Ramey, > wrote: On 3/11/24 2:50 PM, Mischa Baars wrote: > Which sort of brings us back to the original question I suppose. Who does > that line of code function from a script and

Re: multi-threaded compiling

On Mon, 11 Mar 2024, 21:08 Kerin Millar, wrote: > On Mon, 11 Mar 2024 15:36:48 -0400 > Greg Wooledge wrote: > > > > On Mon, Mar 11, 2024, 20:13 Mischa Baars > > > > wrote: > > > > > > > Also I don't think that gives you an exit status for each 'exit $i' > > > > started. I need that exit status.

Re: multi-threaded compiling

On Mon, 11 Mar 2024 15:36:48 -0400 Greg Wooledge wrote: > > On Mon, Mar 11, 2024, 20:13 Mischa Baars > > wrote: > > > > > Also I don't think that gives you an exit status for each 'exit $i' > > > started. I need that exit status. > > "wait -n" without a PID won't help you, then. You don't get

Re: multi-threaded compiling

Like this: seconds=1; for (( i=0;i<32;i++ )); do exit ${i} & done; sleep ${seconds}; for (( i=0;i<32;i++ )); do wait -np pid; e=${?}; echo "$(printf %3u ${i}) pid ${pid} exit ${e}"; done; which again only works when called from a script. I have been looking for the -p option. Guess I overlooked i

Re: multi-threaded compiling

bash s wait returns the exit code of the proc from the job On Mon, Mar 11, 2024, 20:52 Mischa Baars wrote: > On Mon, 11 Mar 2024, 20:36 Greg Wooledge, wrote: > > > > On Mon, Mar 11, 2024, 20:13 Mischa Baars > > > > wrote: > > > > > > > Also I don't think that gives you an exit status for each

Re: multi-threaded compiling

On Mon, 11 Mar 2024, 20:36 Greg Wooledge, wrote: > > On Mon, Mar 11, 2024, 20:13 Mischa Baars > > wrote: > > > > > Also I don't think that gives you an exit status for each 'exit $i' > > > started. I need that exit status. > > "wait -n" without a PID won't help you, then. You don't get the PID

Re: multi-threaded compiling

On 3/11/24 2:46 PM, Mischa Baars wrote: You mean: for (( i=0; i<32; i++ )); do exit $i; done; for (( i=0; i<32; i++ )); do wait -n; echo $?; done; Because this doesn't and to be honest, I needed the pid and its index to retrieve gcc's output from a log file array afterwards. If you want the

Re: multi-threaded compiling

On Mon, 11 Mar 2024, 20:20 Chet Ramey, wrote: > On 3/11/24 2:50 PM, Mischa Baars wrote: > > Which sort of brings us back to the original question I suppose. Who does > > that line of code function from a script and why does it fail from the > > command line? > > Job control and when the shell not

Re: multi-threaded compiling

On Mon, 2024-03-11 at 15:36 -0400, Greg Wooledge wrote: > You won't be able to take advantage of "wait -n"'s ability to react > to the first job that finishes.  You'll end up reaping each job in > the order they started, not the order they finished. It feels to me like you're trying to reproduce m

Re: multi-threaded compiling

On Mon, Mar 11, 2024 at 07:22:39PM +0100, Mischa Baars wrote: > On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev > wrote: > > > i also completly dont get ur issue > > > > f=( a.c b.c .. ) threads=$( nproc ) i=-1 r= > > > > while [[ -v f[++i] ]] ; do > > (( ++r > threads )) && > > wait -n >

Re: multi-threaded compiling

On Mon, Mar 11, 2024 at 06:51:54PM +0100, Mischa Baars wrote: > SECONDS=5; for (( i=0;i<32;i++ )); do { exit ${i}; } & pid[${i}]=${!}; done; > sleep ${SECONDS}; for (( i=0;i<32;i++ )); do wait -n ${pid[${i}]}; e=${?}; > echo "$(printf %3u ${i}) pid ${pid[${i}]} exit ${e}"; done; > /bin/bash: line

Re: multi-threaded compiling

On 3/11/24 2:50 PM, Mischa Baars wrote: Which sort of brings us back to the original question I suppose. Who does that line of code function from a script and why does it fail from the command line? Job control and when the shell notifies the user about job completion, most likely, two of the

Re: multi-threaded compiling

> On Mon, Mar 11, 2024, 20:13 Mischa Baars > wrote: > > > Also I don't think that gives you an exit status for each 'exit $i' > > started. I need that exit status. "wait -n" without a PID won't help you, then. You don't get the PID or job ID that terminated, and you don't get the exit status.

Re: multi-threaded compiling

afik wait will exit with that $? of returned On Mon, Mar 11, 2024, 20:13 Mischa Baars wrote: > That's not really an answer to the question. > > Also I don't think that gives you an exit status for each 'exit $i' > started. I need that exit status. > > > On Mon, 11 Mar 2024, 20:03 alex xmb sw rat

Re: multi-threaded compiling

That's not really an answer to the question. Also I don't think that gives you an exit status for each 'exit $i' started. I need that exit status. On Mon, 11 Mar 2024, 20:03 alex xmb sw ratchev, wrote: > > > On Mon, Mar 11, 2024, 20:03 alex xmb sw ratchev wrote: > >> the logic between my code

Re: multi-threaded compiling

On Mon, Mar 11, 2024, 20:03 alex xmb sw ratchev wrote: > the logic between my code > > 1 threads_max > 2 loop > 3 inside loop , do if run is > than threads_max then wait -n one > then 4 spawn thread > 3 if run isnt more than max , simply ignore and spawn thread in next cmd i dont get ur points

Re: multi-threaded compiling

the logic between my code 1 threads_max 2 loop 3 inside loop , do if run is > than threads_max then wait -n one then 4 spawn thread i dont get ur points On Mon, Mar 11, 2024, 19:55 Mischa Baars wrote: > Sorry. I mean: > > for (( i=0; i<32; i++ )); do exit $i & done; for (( i=0; i<32; i++ )); d

Re: multi-threaded compiling

Sorry. I mean: for (( i=0; i<32; i++ )); do exit $i & done; for (( i=0; i<32; i++ )); do wait -n; echo $?; done; doesn't function. With an ampersand instead of a semicolon. Why does it function when called from a script and why does it fail when called from the command line? On Mon, Mar 11, 2024

Re: multi-threaded compiling

Which sort of brings us back to the original question I suppose. Who does that line of code function from a script and why does it fail from the command line? My guess was that the same thing makes this line fail from the Makefile. On Mon, Mar 11, 2024 at 7:46 PM Mischa Baars wrote: > You mean:

Re: multi-threaded compiling

You mean: for (( i=0; i<32; i++ )); do exit $i & wait -n; echo $?; done; with one command and one wait in a single loop. And this does execute on the command line. How interesting! for (( i=0; i<32; i++ )); do exit $i; done; for (( i=0; i<32; i++ )); do wait -n; echo $?; done; Because this does

Re: multi-threaded compiling

On Mon, 2024-03-11 at 19:37 +0100, alex xmb sw ratchev wrote: > /data/data/com.termux/files/usr/bin/sh: 1: Syntax error: Bad for loop > variable This is because of the issue I mentioned in my initial reply. This invocation is using /bin/sh which is a POSIX shell but is not bash; probably it's das

Re: multi-threaded compiling

other output ~/2024031100 - gnu questions/one $ make SECONDS=5; for (( i=0;i<32;i++ )); do { exit ${i}; } & pid[${i}]=${!}; done; sleep ${SECONDS}; for (( i=0;i<32;i++ )); do wait -n ${pid[${i}]}; e=${?}; echo "$(printf %3u ${i}) pid ${pid[${i}]} exit ${e}"; done; /data/data/com.termux/files/usr/b

Re: multi-threaded compiling

On Mon, Mar 11, 2024, 19:25 alex xmb sw ratchev wrote: > > > On Mon, Mar 11, 2024, 19:22 Mischa Baars > wrote: > >> On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev >> wrote: >> >>> i also completly dont get ur issue >>> >>> f=( a.c b.c .. ) threads=$( nproc ) i=-1 r= >>> >>> while [[ -v f

Re: multi-threaded compiling

On Mon, Mar 11, 2024 at 7:14 PM Greg Wooledge wrote: > On Mon, Mar 11, 2024 at 06:51:54PM +0100, Mischa Baars wrote: > > SECONDS=5; for (( i=0;i<32;i++ )); do { exit ${i}; } & pid[${i}]=${!}; > done; sleep ${SECONDS}; for (( i=0;i<32;i++ )); do wait -n ${pid[${i}]}; > e=${?}; echo "$(printf %3u $

Re: multi-threaded compiling

On Mon, Mar 11, 2024, 19:22 Mischa Baars wrote: > On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev > wrote: > >> i also completly dont get ur issue >> >> f=( a.c b.c .. ) threads=$( nproc ) i=-1 r= >> >> while [[ -v f[++i] ]] ; do >> (( ++r > threads )) && >> wait -n >> gcc -c "${f[i]}" &

Re: multi-threaded compiling

Hi Greg, Good point. One for you :) Cheerz, Mischa. On Mon, Mar 11, 2024 at 7:14 PM Greg Wooledge wrote: > On Mon, Mar 11, 2024 at 06:51:54PM +0100, Mischa Baars wrote: > > SECONDS=5; for (( i=0;i<32;i++ )); do { exit ${i}; } & pid[${i}]=${!}; > done; sleep ${SECONDS}; for (( i=0;i<32;i++ ));

Re: multi-threaded compiling

On Mon, Mar 11, 2024 at 6:22 PM alex xmb sw ratchev wrote: > i also completly dont get ur issue > > f=( a.c b.c .. ) threads=$( nproc ) i=-1 r= > > while [[ -v f[++i] ]] ; do > (( ++r > threads )) && > wait -n > gcc -c "${f[i]}" & > done > How nice! wait -n exit 1 & echo $? You got me the

Re: multi-threaded compiling

Oh, ok. I have executed the examples for you now. What I expected was also in the first mail: 1) The one.make.sh.log has the expected output. The same commands executed from a Makefile (one.make.log) do not behave as expected. This also holds for this command executed from the command line. 2) Th

Re: multi-threaded compiling

On Mon, 2024-03-11 at 18:14 +0100, Mischa Baars wrote: > The actual command invocations (a Makefile, a make.sh script) can be > found in the attachment, as indicated on the first line of the mail. > In the attachment there are two directories, one and two, belonging > to 1) and 2) respectively. Th

Re: multi-threaded compiling

i also completly dont get ur issue f=( a.c b.c .. ) threads=$( nproc ) i=-1 r= while [[ -v f[++i] ]] ; do (( ++r > threads )) && wait -n gcc -c "${f[i]}" & done On Mon, Mar 11, 2024, 18:16 Mischa Baars wrote: > Hello Paul, > > It seems I'm awake a little longer than you are. > > The secon

Re: multi-threaded compiling

Hello Paul, It seems I'm awake a little longer than you are. The second paragraph as you see it, belongs to 1) The fourth paragraph as you see it, belongs to 2) The actual command invocations (a Makefile, a make.sh script) can be found in the attachment, as indicated on the first line of the mai

Re: multi-threaded compiling

On Mon, 2024-03-11 at 09:56 +0100, Mischa Baars wrote: > I've attached the problem. It consists of two parts: > > 1) multi-threaded bash script and / or multi-threaded Makefile > > Running bash script functions as expected, but executing the same > line of code with make and / or the command line

multi-threaded compiling

Hi, I'd like to compile a project of mine using multiple logical cores. I've attached the problem. It consists of two parts: 1) multi-threaded bash script and / or multi-threaded Makefile Running bash script functions as expected, but executing the same line of code with make and / or the comma