dsimcha Wrote:
> == Quote from Jerry Quinn (jlqu...@optonline.net)'s article
> > dsimcha Wrote:
> > > == Quote from Jerry Quinn (jlqu...@optonline.net)'s article
> > > > I'm looking at porting an app that maintains a work queue to be
> > > > processed by one
> > > of N engines and written out in order. At first glance, std.parallelism
> > > already
> > > provides the queue, but the Task concept appears to assume that there's
> > > no startup
> > > cost per thread.
> > > > Am I missing something or do I need to roll a shared queue object?
> > >
> > > Not sure if I'm misunderstanding what you're asking, but I'll answer the
> > > question
> > > I think you're asking. std.parallelism's Task can be executed in two
> > > ways.
> > > executeInNewThread() does start a new thread for every Task. However,
> > > you can
> > > also submit a Task to a TaskPool and have it executed by an existing
> > > worker
> > > thread. The whole point of using TaskPool to execute a Task is to avoid
> > > paying
> > > the thread start-up cost for every Task.
> > The question I was asking was how to execute a huge amount of per-thread
> initialization once per thread in the TaskPool framework.
>
> Hmm, currently there isn't an easy/obvious way, though I'm thinking maybe one
> should be added. I've kind of needed it, too, and I don't anticipate it being
> very hard to implement. If you can suggest a good API for this, I'll work on
> it
> for next release.
I'd probably naturally want to do something like:
auto engines = taskPool.init(new Engine(params));
auto lines = File("foo.txt").byLine();
auto results = taskPool.map!(engines.process)(lines)
foreach (auto r; results)
writeln(r);
This would create a set of new Engine objects per object, but initialized
within each thread as thread-local data. Each Engine object has a process()
call taking a line as input and returning a string in this case. Any free
engine can process more work as long as there is still work available.
For the large-scale use case, it could take an extra argument and allow only N
simultaneous init's under the covers.
