On Friday, 25 August 2023 at 21:31:37 UTC, Ali Çehreli wrote:
On 8/25/23 14:27, j...@bloow.edu wrote:
> "A work unit is a set of consecutive elements of range to be
processed
> by a worker thread between communication with any other
thread. The
> number of elements processed per work unit is controlled by
the
> workUnitSize parameter. "
>
> So the question is how to rebalance these work units?
Ok, your question brings me back from summer hibernation. :)
This is what I do:
- Sort the tasks in decreasing time order; the ones that will
take the most time should go first.
- Use a work unit size of 1.
The longest running task will start first. You can't get better
than that. When I print some progress reporting, I see that
most of the time N-1 tasks have finished and we are waiting for
that one longest running task.
Ali
"back to sleep"
I do not know the amount of time they will run. They are files
that are being downloaded and I neither know the file size nor
the download rate(in fact, the actual download happens
externally).
While I could use work unit of size 1 then problem then is I
would be downloading N files at once and that will cause other
problems if N is large(and sometimes it is).
There should be a "work unit size" and a "max simultaneous
workers". Then I could set the work unit size to 1 and say the
max simultaneous workers to 8 to get 8 simultaneous downloads
without stalling.
