I use
foreach(s; taskPool.parallel(files, numParallel))
{ L(s); } // L(s) represents the work to be done.
to download files from the internet.
Everything works. The issue is this:
the foreach will download 8 files at once. BUT it will not start
the next batch of 8 *until* ALL of the previous 8 are done. It
seems that taskPool.parallel will not immediately start a new
thread once a task is done
E.g., I get
L(s1);
L(s2);
...
L(s8);
--- // nothing below is executed until all L(s1) through L(s8)
are finished.
L(s9);
L(s10);
...
My expectation is that, say, when the first task is complete, say
L(s4), that L(s9) is then executed.
The reason why this causes me problems is that the downloaded
files, which are cashed to a temporary file, stick around and do
not free up space(think of it just as using memory) and this can
cause some problems some of the time. Also, the point of parallel
tasks is to allow paralleling but the way the code is working is
that it starts the tasks in parallel but then essentially stalls
the paralleling a large portion of the time. E.g.,
If there are a bunch of small downloads but one large one, then
that one large download stalls the everything. E.g., say L(s5) is
a very long download while all others are very quick. Then L(s5)
will prevent downloading anything afterwards until it is
finished(I'll get L(s1) through L(s8) but nothing else until
L(s5) is finished).
What's going on and how to reconcile?
