On Wednesday, 23 August 2023 at 14:43:33 UTC, Sergey wrote:
On Wednesday, 23 August 2023 at 13:03:36 UTC, Joe wrote:
I use
foreach(s; taskPool.parallel(files, numParallel))
{ L(s); } // L(s) represents the work to be done.
If you make for example that L function return “ok” in case
file successfully downloaded, you can try to use TaskPool.amap.
The other option - use std.concurrency probably.
I think I might know what is going on but not sure:
The tasks are split up in batches and each batch gets a thread.
What happens then is some long task will block it's entire batch
and there will be no re-balancing of the batches.
"A work unit is a set of consecutive elements of range to be
processed by a worker thread between communication with any other
thread. The number of elements processed per work unit is
controlled by the workUnitSize parameter. "
So the question is how to rebalance these work units?
E.g., when a worker thread is done with it's batch it should look
to help finish that batch rather than terminating and leaving all
the work for the last thread.
This seems like a flaw in the design. E.g., if one happens to
have n batches and every batch but one has tasks that finish
instantly then essentially one has no parallelization.
