Re: How does this wait -n work to cap parallelism?
On Mon, Jul 29, 2019 at 07:12:42PM +0100, Earnestly wrote:
> #!/usr/bin/env bash
>
> # number of processes to run in parallel
> num_procs=5
>
> # function that processes one item
> my_job() {
> printf 'Processing %s\n' "$1"
> sleep "$(( RANDOM % 5 + 1 ))"
> }
>
> i=0
> while IFS= read -r line; do
> if (( i++ >= num_procs )); then
> wait -n # wait for any job to complete. New in 4.3
> fi
> my_job "$line" &
> done < inputlist
> wait # wait for the remaining processes
>
>
> The question is about how the example works in order to maintain
> parallelism capped at num_proc.
"inputlist" is a file that contains some sort of data, one thing per
line. As each line is read, it is supposed to kick off one job (process)
with that line as the input parameter.
The while loop reads a line at a time, and for every line, it runs
my_job in the background. BUT, if it has already looped 5 or more times,
then before it does that, it calls "wait -n".
wait -n waits for one job to terminate.
Let's suppose the inputlist contains:
1
2
3
4
5
6
7
8
The first time through the loop, we run my_job 1. The second time, my_job 2,
and so on, up to my_job 5. All of those are run without any delay.
So there are 5 instances of my_job running simultaneously in the background.
The sixth time through the loop, i is 5 (before the increment), so
we run "wait -n". This waits for one of the 5 instances of my_job to
finish. Once that happens, we run my_job 6, and loop again.
The same happens for my_job 7, and my_job 8. Each one is preceded by
a wait -n, so it waits for one of the existing jobs to terminate before
the new job is launched.
Re: How does this wait -n work to cap parallelism?
On Mon, Jul 29, 2019 at 02:38:48PM -0400, Greg Wooledge wrote: > The same happens for my_job 7, and my_job 8. Each one is preceded by > a wait -n, so it waits for one of the existing jobs to terminate before > the new job is launched. This aspect of the behaviour isn't in question. Without reiterating too much, the question is about how that cap is maintained after the 'wait -n' loop only ever experiences a single agent while 'i' is incrementing, and only after 'i' has exceeded 'nproc' does the parallelism start.
Re: How does this wait -n work to cap parallelism?
On Mon, Jul 29, 2019 at 07:12:42PM +0100, Earnestly wrote: > The question is about how the example works in order to maintain > parallelism capped at num_proc. Thanks to emg on #bash for explaining how what is essentially going on. Bash essentially maintains a list of completed jobs until wait -n is called and removes one of those completed jobs. Bash then adds another job according to the loop I used. After the "slow" phase of my synthetic code it just so happens that these jobs are removed and new ones are added rapidly. And so the jobs are added and then removed accordingly but is never allowed to exceed "nproc" because wait -n is always called thereafter. > Below I've provided a synthetic scenario which hopefully highlights my > (and others) confusion. It was because of this that I trapped my mind. Because I was watching top I concluded that once a job had ended, bash itself would also do likewise and discard it. I wasn't able to consider that bash might maintain a list of *completed* jobs for which subsequent 'wait' calls would remove. Sorry for the noise.
