Re: David Simcha's std.parallelism

[Russel Winder]

David,

On Thu, 2011-02-03 at 20:56 -0500, dsimcha wrote:
 Ok, I finally got around to tackling most of this.  See individual 
 replies below.

Any chance you can put this package in a separate DVCS repository (I'll
assume Git given the way the rest of D is going) so that updates can be
picked up and tried out?

-- 
Russel.
=
Dr Russel Winder  t: +44 20 7585 2200   voip: sip:russel.win...@ekiga.net
41 Buckmaster Roadm: +44 7770 465 077   xmpp: rus...@russel.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder


signature.asc
Description: This is a digitally signed message part

Re: David Simcha's std.parallelism

[dsimcha]

I could move it over to github, though I'll wait to do that until I get 
a little more comfortable with Git.  I had never used Git before until 
Phobos switched to it.  In the mean time, to remind, the code is at:


http://dsource.org/projects/scrapple/browser/trunk/parallelFuture/std_parallelism.d

The docs are at:

http://cis.jhu.edu/~dsimcha/d/phobos/std_parallelism.html

On 2/4/2011 3:45 AM, Russel Winder wrote:

David,

On Thu, 2011-02-03 at 20:56 -0500, dsimcha wrote:

Ok, I finally got around to tackling most of this.  See individual
replies below.


Any chance you can put this package in a separate DVCS repository (I'll
assume Git given the way the rest of D is going) so that updates can be
picked up and tried out?

Re: David Simcha's std.parallelism

[Russel Winder]

On Fri, 2011-02-04 at 08:09 -0500, dsimcha wrote:
 I could move it over to github, though I'll wait to do that until I get 
 a little more comfortable with Git.  I had never used Git before until 
 Phobos switched to it.  In the mean time, to remind, the code is at:
 
 http://dsource.org/projects/scrapple/browser/trunk/parallelFuture/std_parallelism.d

It seems that DSource Subversion service is even worse (much worse) than
that of GoogleCode.  I appear to be unable to checkout just the
parallelFuture directory with any of Git, Mercurial, Bazaar, or
Subversion (!).  It either gives a 500 Error or a OPTIONS failure :-
This does not bode well for extracting anything from that area of the
Subversion repository.  (Unless I am doing something wrong, which is
entirely possible.)

Given that I am not about to checkout everything in scrapple/browser
just to get one file, I'll wait till you feel able to switch to Git.
Don't think of this as an attempt to put pressure on you, it is really
just an excuse for me not to do something.

 The docs are at:
 
 http://cis.jhu.edu/~dsimcha/d/phobos/std_parallelism.html

Can this not go under version control as well?

Thanks.

PS  I strongly recommend you find ways of learning Git, although I
prefer Mercurial and Bazaar to Git, any of these three make developing a
pleasure compared to using Subversion.  Once you get into DVCS there is
no turning back from being far more productive.

-- 
Russel.
=
Dr Russel Winder  t: +44 20 7585 2200   voip: sip:russel.win...@ekiga.net
41 Buckmaster Roadm: +44 7770 465 077   xmpp: rus...@russel.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder


signature.asc
Description: This is a digitally signed message part

Re: David Simcha's std.parallelism

[Ellery Newcomer]

svn ls http://svn.dsource.org/projects/scrapple/trunk/parallelFuture/

works for me

On 02/04/2011 11:28 AM, Russel Winder wrote:

On Fri, 2011-02-04 at 08:09 -0500, dsimcha wrote:

I could move it over to github, though I'll wait to do that until I get
a little more comfortable with Git.  I had never used Git before until
Phobos switched to it.  In the mean time, to remind, the code is at:

http://dsource.org/projects/scrapple/browser/trunk/parallelFuture/std_parallelism.d


It seems that DSource Subversion service is even worse (much worse) than
that of GoogleCode.  I appear to be unable to checkout just the
parallelFuture directory with any of Git, Mercurial, Bazaar, or
Subversion (!).  It either gives a 500 Error or a OPTIONS failure :-
This does not bode well for extracting anything from that area of the
Subversion repository.  (Unless I am doing something wrong, which is
entirely possible.)

Re: David Simcha's std.parallelism

[Gour]

On Fri, 04 Feb 2011 17:28:25 +
Russel Winder rus...@russel.org.uk wrote:

 PS  I strongly recommend you find ways of learning Git, although I
 prefer Mercurial and Bazaar to Git, any of these three make
 developing a pleasure compared to using Subversion.  Once you get
 into DVCS there is no turning back from being far more productive.

For my own stuff I use and I'm very happy with Fossil SCM, but when I
have to do something with other projects, now I use Bazaar with its
set of plugins bzr-{git,hg,svn} (although 'hg' is not on par with the
rest), so everything can be done with bzr and 'explorer' plugin is
also nice.


Sincerely,
Gour

-- 

Gour  | Hlapicina, Croatia  | GPG key: CDBF17CA



signature.asc
Description: PGP signature

Re: David Simcha's std.parallelism

[Russel Winder]

On Fri, 2011-02-04 at 12:12 -0600, Ellery Newcomer wrote:
 svn ls http://svn.dsource.org/projects/scrapple/trunk/parallelFuture/
 
 works for me
[ . . . ]
  Phobos switched to it.  In the mean time, to remind, the code is at:
 
  http://dsource.org/projects/scrapple/browser/trunk/parallelFuture/std_parallelism.d

Hummm... spot the error.  It works for me now I know ;-)

I should have tried harder I guess.

-- 
Russel.
=
Dr Russel Winder  t: +44 20 7585 2200   voip: sip:russel.win...@ekiga.net
41 Buckmaster Roadm: +44 7770 465 077   xmpp: rus...@russel.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder


signature.asc
Description: This is a digitally signed message part

Re: David Simcha's std.parallelism

[Tomek Sowiński]

dsimcha napisał:

 I could move it over to github, though I'll wait to do that until I get 
 a little more comfortable with Git.  I had never used Git before until 
 Phobos switched to it.  In the mean time, to remind, the code is at:
 
 http://dsource.org/projects/scrapple/browser/trunk/parallelFuture/std_parallelism.d
 
 The docs are at:
 
 http://cis.jhu.edu/~dsimcha/d/phobos/std_parallelism.html

Please run the docs through a spell-checker, there are a few typos:

asyncBuf() - for ecample
stop() - waitied
lazyMap() - Parameters;

But I think it's good overall. These primitives are in demand.

-- 
Tomek

Re: David Simcha's std.parallelism

[Jérôme M. Berger]

Russel Winder wrote:
 On Fri, 2011-02-04 at 12:12 -0600, Ellery Newcomer wrote:
 svn ls http://svn.dsource.org/projects/scrapple/trunk/parallelFuture/

 works for me
 [ . . . ]
 Phobos switched to it.  In the mean time, to remind, the code is at:

 http://dsource.org/projects/scrapple/browser/trunk/parallelFuture/std_parallelism.d
 
 Hummm... spot the error.  It works for me now I know ;-)
 
 I should have tried harder I guess.
 
Not knowing what you did, my bet is that you used the web interface
URL instead of the real svn URL (i.e you forgot the “svn.” at the
beginning).

Jerome
-- 
mailto:jeber...@free.fr
http://jeberger.free.fr
Jabber: jeber...@jabber.fr



signature.asc
Description: OpenPGP digital signature

Re: David Simcha's std.parallelism

[dsimcha]

== Quote from Tomek Sowiński (j...@ask.me)'s article
 Please run the docs through a spell-checker, there are a few typos:
 asyncBuf() - for ecample
 stop() - waitied
 lazyMap() - Parameters;
 But I think it's good overall. These primitives are in demand.
 --
 Tomek

I will certainly tie up all of these kinds of loose ends when all the bigger
picture stuff is taken care of and this thing is ready to be committed.  It's 
just
that I don't see the point in worrying about such minor details before the 
overall
content is finalized.

Re: David Simcha's std.parallelism

[Russel Winder]

On Fri, 2011-02-04 at 23:06 +0100, Jérôme M. Berger wrote:
 Russel Winder wrote:
  On Fri, 2011-02-04 at 12:12 -0600, Ellery Newcomer wrote:
  svn ls http://svn.dsource.org/projects/scrapple/trunk/parallelFuture/
 
  works for me
  [ . . . ]
  Phobos switched to it.  In the mean time, to remind, the code is at:
 
  http://dsource.org/projects/scrapple/browser/trunk/parallelFuture/std_parallelism.d
  
  Hummm... spot the error.  It works for me now I know ;-)
  
  I should have tried harder I guess.
  
   Not knowing what you did, my bet is that you used the web interface
 URL instead of the real svn URL (i.e you forgot the “svn.” at the
 beginning).

That is part of the error, the other part of it is `sed 's/\/browser//'`
put the two together and you transform failure into success :-)

-- 
Russel.
=
Dr Russel Winder  t: +44 20 7585 2200   voip: sip:russel.win...@ekiga.net
41 Buckmaster Roadm: +44 7770 465 077   xmpp: rus...@russel.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder


signature.asc
Description: This is a digitally signed message part

Re: David Simcha's std.parallelism

[dsimcha]

Ok, I finally got around to tackling most of this.  See individual 
replies below.


On 1/1/2011 6:07 PM, Andrei Alexandrescu wrote:

I think David Simcha's library is close to reviewable form. Code:

http://dsource.org/projects/scrapple/browser/trunk/parallelFuture/std_parallelism.d


Documentation:

http://cis.jhu.edu/~dsimcha/d/phobos/std_parallelism.html

Here are a few comments:

* parallel is templated on range, but not on operation. Does this affect
speed for brief operations (such as the one given in the example,
squares[i] = i * i)? I wonder if using an alias wouldn't be more
appropriate. Some performance numbers would be very useful in any case.


As I've said before, I strongly believe it's not worth it to give up the 
features (ref iteration, index variables, consistency with non-parallel 
syntax) that opApply offers, given that we also have map() and the 
performance hit is only a small constant that only matters for very 
fine-grained parallelism.  (I've shown it to be pretty small in absolute 
terms with benchmarks.  See previous posts.)




* Why is ThreadPool a class and what are the prospects of overriding its
members?



Made it final.  Done.


* Can't we define the behavior of break inside a parallel loop?


Done.  It now aborts the current work unit.  A goto from inside the loop 
to outside the loop is still undefined, though, because it makes even 
less sense and raises the possibility of conflicting gotos from 
different threads.




* I think it does make sense to evaluate a parallel map lazily by using
a finite buffer. Generally map looks the most promising so it may be
worth investing some more work in it to make it smart lazy.


I have added LazyMap, which rotates two buffers of a user-specified size 
and computes the map functions and dumps them to the back buffer while 
making the front buffer available via front() and popFront().  One can 
also achieve pipelining by chaining LazyMap objects.




* waitStop() - join()?


Done.



* The documentation should use more examples. Currently it uses entities
without defining them (Task, TaskPool etc.)


Can you elaborate on this?  I am having trouble thinking of things to 
add to the docs.


* why runCallable()? There's no runUncallable().


I made this into run().



* Should there be a makeAngel to undo makeDaemon?


Done.  Added makeAngel.

Re: David Simcha's std.parallelism

[dsimcha]

On 1/1/2011 6:07 PM, Andrei Alexandrescu wrote:

* parallel is templated on range, but not on operation. Does this affect
speed for brief operations (such as the one given in the example,
squares[i] = i * i)? I wonder if using an alias wouldn't be more
appropriate. Some performance numbers would be very useful in any case.


Ok, I did the benchmarks.  Since map is templated on the operation, I 
used that as a benchmark of the templating on operation scenario. 
Here's the benchmark:


import std.parallelism, std.stdio, std.datetime, std.range, std.conv,
std.math;

int fun1(int num) {
return roundTo!int(sqrt(num));
}

int fun2(int num) {
return num * num;
}

alias fun2 fun;

void main() {
auto foo = array(iota(10_000_000));
auto bar = new int[foo.length];

enum workUnitSize = 1_000_000;

auto sw = StopWatch(autoStart);
foreach(i, elem; parallel(foo, workUnitSize)) {
bar[i] = fun(elem);
}
writeln(Parallel Foreach:  , sw.peek.milliseconds);

sw = StopWatch(autoStart);
bar = taskPool.map!fun(foo, workUnitSize, bar);
writeln(Map:  , sw.peek.milliseconds);

sw = StopWatch(autoStart);
foreach(i, elem; foo) {
bar[i] = fun(elem);
}
writeln(Serial:  , sw.peek.milliseconds);
}


Results:

Parallel Foreach:  69.2988
Map:  29.1973
Serial:  40.2884


So obviously there's a huge penalty when the loop body is super cheap.

On the other hand, when I make fun1 the loop body instead (and it's 
still a fairly cheap body), the differences are buried in noise.


Now that I've given my honest report of the facts, though, I'd like to 
say that even so, I'm in favor of leaving things as-is, for the 
following reasons:


1.  Super cheap loop bodies are usually not worth parallelizing anyhow. 
 You get nowhere near a linear speedup due to memory bandwidth issues, 
etc., and if some super cheap loop body is your main bottleneck, it's 
probably being executed in in some outer loop and it may make more sense 
to parallelize the outer loop.  In all my experience with 
std.parallelism, I've **never** had the the need/desire to resort to 
parallelism fine grained enough that the limitations of delegate-based 
parallel foreach mattered in practice.


2.  If you really want to parallelize super cheap loop bodies, map() 
isn't going anywhere and that and/or reduce(), which also uses 
templates, will usually do what you need.  You can even use parallel map 
in place by simply passing in the same (writeable) range for both the 
input and the buffer.


3.  The foreach syntax makes the following very useful things (as in, I 
actually use them regularly) possible that wouldn't be possible if we 
used templates:


foreach(index, elem; parallel(range))
foreach(ref elem; parallel(range))

It also just plain looks nice.

4.  A major point of parallel foreach is that variables in the outer 
scope just work.  When passing blocks of code as aliases instead of 
delegates, this is still very buggy.


5.  I'm hoping I can convince Walter to implement an alias-based version 
of opApply, which is half-implemented and commented out in the DMD 
source code.  If this were implemented, I'd change std.parallelism to 
use it and this whole discussion would be moot.

Re: David Simcha's std.parallelism

[Andrej Mitrovic]

Here's my results:
Parallel Foreach:  29.6365
Map:  23.1849
Serial:  32.9265

This is with -release -inline -O -noboundscheck

I'm on a quad-core btw.

Re: David Simcha's std.parallelism

[Lars T. Kyllingstad]

On Sun, 02 Jan 2011 11:00:36 -0600, Andrei Alexandrescu wrote:

 On 1/2/11 10:39 AM, dsimcha wrote:
 On 1/1/2011 6:07 PM, Andrei Alexandrescu wrote:
 I think David Simcha's library is close to reviewable form. Code:

 http://dsource.org/projects/scrapple/browser/trunk/parallelFuture/
std_parallelism.d



 Documentation:

 http://cis.jhu.edu/~dsimcha/d/phobos/std_parallelism.html

 Here are a few comments:

 * parallel is templated on range, but not on operation. Does this
 affect speed for brief operations (such as the one given in the
 example, squares[i] = i * i)? I wonder if using an alias wouldn't be
 more appropriate. Some performance numbers would be very useful in any
 case.

 Will benchmark, though the issue here is that I like the foreach syntax
 and this can't be done with templates as you describe. Also, IIRC (I'll
 do the benchmark at some point to be sure) for such examples memory
 bandwidth is limiting, so the delegate call doesn't matter anyhow.
 
 I think anyone who wants to run something in parallel is motivated
 primarily by increasing efficiency, so focusing on aesthetics would be
 secondary and unsatisfactory for your target audience.

I've already been using David's parallelism library for quite a while, 
and parallel foreach is my favourite part of it!  I often have programs 
that look like this:

  foreach (point; grid) { point.doExpensiveComputation(); }

With David's library I just rewrite this into

  foreach (point; parallel(grid)) { ... }

Wow!  Typing 10 characters just gave me a 3-4 times speedup (assuming 
quad core).

So yeah, if using an alias parameter gives significant performance 
benefits for cheap operations (such as the squares example), by all 
means, add that.  But I don't think parallel foreach should be dropped in 
any case.

-Lars

Re: David Simcha's std.parallelism

[bearophile]

Andrei:

Overall it looks nice enough.

 * I think it does make sense to evaluate a parallel map lazily by using 
 a finite buffer. Generally map looks the most promising so it may be 
 worth investing some more work in it to make it smart lazy.

I agree.

Some more notes:

Later a parallel max() and min() (that work on sequences) may be good.

The syntax for the parallel foreach is not the best conceivable, but for now 
it's OK:
foreach(i; pool.parallel( iota(squares.length), 100)) {

This is about reduce, unfortunately there is no @associative annotations and 
it's not easy for a compiler to enforce the operator to be associative:
 Because this operation is being carried out in parallel, fun must be 
 associative. 

I think a parallel map() has to require the mapping function to be pure. It's 
not a strong requirement (and you can't add debug prints or logging), but 
logically it looks better.

A more general note regarding map() of std.algorithm too: 
pool.map!(a * a, -a)(numbers, 100, results);
In general the idea of multiple functions in map() doesn't look like a good 
idea. Instead of multiple functions I prefer (and I think it's more useful) a 
single function and optionally multiple iterables, as in Python and dlibs1, 
useful for functions that take more than one argument:
 s = [1, 2, 3]
 p = [2, 4, 6]
 map(pow, s, p)
[1, 16, 729]

Bye,
bearophile

Re: David Simcha's std.parallelism

[bearophile]

One more note: how are performances? Are benchmarks available? (I even suggest 
to add some benchmarks to the std.parallelism module unit tests).

Bye,
bearophile

Re: David Simcha's std.parallelism

[dsimcha]

On 1/1/2011 6:07 PM, Andrei Alexandrescu wrote:

I think David Simcha's library is close to reviewable form. Code:

http://dsource.org/projects/scrapple/browser/trunk/parallelFuture/std_parallelism.d


Documentation:

http://cis.jhu.edu/~dsimcha/d/phobos/std_parallelism.html

Here are a few comments:

* parallel is templated on range, but not on operation. Does this affect
speed for brief operations (such as the one given in the example,
squares[i] = i * i)? I wonder if using an alias wouldn't be more
appropriate. Some performance numbers would be very useful in any case.


Will benchmark, though the issue here is that I like the foreach syntax 
and this can't be done with templates as you describe.  Also, IIRC (I'll 
do the benchmark at some point to be sure) for such examples memory 
bandwidth is limiting, so the delegate call doesn't matter anyhow.




* Why is ThreadPool a class and what are the prospects of overriding its
members?


It's a class b/c I wanted to have reference semantics and a monitor.  I 
could make it a final class, as I didn't design for overriding anything.




* Can't we define the behavior of break inside a parallel loop?


I left the behavior of this undefined because I didn't have any good 
ideas about what it should do.  If you can suggest something that would 
be useful, I'll define it.  I'd rather not define it to do something 
completely arbitrary and not obviously useful b/c if we eventually come 
up w/ a useful semantic our hands will be tied.




* I think it does make sense to evaluate a parallel map lazily by using
a finite buffer. Generally map looks the most promising so it may be
worth investing some more work in it to make it smart lazy.


Can you elaborate on this?  I'm not sure what you're suggesting.



* waitStop() - join()?


I like waitStop() better, though join() is more standard and I've 
received this comment from some friends, too.  If there's a strong 
preference for join(), then I'll change it rather than wasting time on 
bikeshed issues.




* The documentation should use more examples. Currently it uses entities
without defining them (Task, TaskPool etc.)


Will do when I get a chance.



* why runCallable()? There's no runUncallable().


runCallable() is in the weird position of being exposed for the purpose 
of clarifying how running delegates and function pointers works, but not 
being a part of the public interface that is supposed to be used 
directly.  I gave it a verbose name for clarity.  Are you suggesting I 
just call it run?




* Should there be a makeAngel to undo makeDaemon?


Will fold into the next revision.  I've never had a use for it, but it's 
trivial to implement, orthogonal with the rest of the design, and 
symmetry is generally a good thing.




Overall, to prepare this for the review process, the documentation
should be grown considerably with motivating examples and relevant
benchmarks. We are modeling our review process after
http://www.boost.org/community/reviews.html. The first realization of
the process has been for std.datetime, and it seems to have been a success.


Andrei

Re: David Simcha's std.parallelism

[Andrei Alexandrescu]

On 1/2/11 10:39 AM, dsimcha wrote:

On 1/1/2011 6:07 PM, Andrei Alexandrescu wrote:

I think David Simcha's library is close to reviewable form. Code:

http://dsource.org/projects/scrapple/browser/trunk/parallelFuture/std_parallelism.d



Documentation:

http://cis.jhu.edu/~dsimcha/d/phobos/std_parallelism.html

Here are a few comments:

* parallel is templated on range, but not on operation. Does this affect
speed for brief operations (such as the one given in the example,
squares[i] = i * i)? I wonder if using an alias wouldn't be more
appropriate. Some performance numbers would be very useful in any case.


Will benchmark, though the issue here is that I like the foreach syntax
and this can't be done with templates as you describe. Also, IIRC (I'll
do the benchmark at some point to be sure) for such examples memory
bandwidth is limiting, so the delegate call doesn't matter anyhow.


I think anyone who wants to run something in parallel is motivated 
primarily by increasing efficiency, so focusing on aesthetics would be 
secondary and unsatisfactory for your target audience.



* Why is ThreadPool a class and what are the prospects of overriding its
members?


It's a class b/c I wanted to have reference semantics and a monitor. I
could make it a final class, as I didn't design for overriding anything.


Final should be it then.


* Can't we define the behavior of break inside a parallel loop?


I left the behavior of this undefined because I didn't have any good
ideas about what it should do. If you can suggest something that would
be useful, I'll define it. I'd rather not define it to do something
completely arbitrary and not obviously useful b/c if we eventually come
up w/ a useful semantic our hands will be tied.


As Sean said - abort current computation I guess.


* I think it does make sense to evaluate a parallel map lazily by using
a finite buffer. Generally map looks the most promising so it may be
worth investing some more work in it to make it smart lazy.


Can you elaborate on this? I'm not sure what you're suggesting.



* waitStop() - join()?


I like waitStop() better, though join() is more standard and I've
received this comment from some friends, too. If there's a strong
preference for join(), then I'll change it rather than wasting time on
bikeshed issues.


If they called it fork-waitStop parallelism I'd like waitStop better, 
too. But they call it fork-join parallelism.



* why runCallable()? There's no runUncallable().


runCallable() is in the weird position of being exposed for the purpose
of clarifying how running delegates and function pointers works, but not
being a part of the public interface that is supposed to be used
directly. I gave it a verbose name for clarity. Are you suggesting I
just call it run?


I'd think so, unless there are reasons not to.

Thanks for working on this!


Andrei

Re: David Simcha's std.parallelism

[Sean Kelly]

Andrei Alexandrescu Wrote:

 I think David Simcha's library is close to reviewable form. Code:
 
 http://dsource.org/projects/scrapple/browser/trunk/parallelFuture/std_parallelism.d
 
 Documentation:
 
 * why runCallable()? There's no runUncallable().

Does this even need to be public?  It looks like an implementation detail.

 * Should there be a makeAngel to undo makeDaemon?

Or an isDaemon property ala. Thread.  Though the chances of someone wanting to 
undo the isDaemon property once set are practically nil.

Re: David Simcha's std.parallelism

[bearophile]

dsimcha:

 Andrei:
  * I think it does make sense to evaluate a parallel map lazily by using
  a finite buffer. Generally map looks the most promising so it may be
  worth investing some more work in it to make it smart lazy.
 
 Can you elaborate on this?  I'm not sure what you're suggesting.

I think Andrei is talking about vectorized lazyness, I have explained the idea 
here two times in past. This isn't a replacement for the fully eager parallel 
map. Instead of computing the whole resulting array in parallel, you compute 
only a chunk of the result, in parallel, and you store it. When the code that 
uses the data lazily has exhausted that chunk, the lazy parallel map computes 
the next chunk and stores it inside, and so on.

Each chunk is large enough that performing it in parallel is advantageous, but 
not large enough to require a lot of memory.

An option is even self-tuning, let the library find the chunk size by itself, 
according to how much time each item computation (mapping function call) 
requires (this is an optional behaviour).

If you have a read-only memory mapped file that is readable from several 
threads in parallel, the map may perform some operation on the lines/records of 
the file. If the file is very large or huge, and you want to collect/summarize 
(reduce) the results of the mapping functions in some way, then a lazy parallel 
map is useful :-) This looks like a special case, but lot of heavy file 
processing (1 - 5000 gigabytes of data) can be done with this schema 
(map-reduce).

Bye,
bearophile

Re: David Simcha's std.parallelism [questions for Walter]

[dsimcha]

== Quote from Andrei Alexandrescu (seewebsiteforem...@erdani.org)'s article
 On 1/2/11 10:39 AM, dsimcha wrote:
  On 1/1/2011 6:07 PM, Andrei Alexandrescu wrote:
  * parallel is templated on range, but not on operation. Does this affect
  speed for brief operations (such as the one given in the example,
  squares[i] = i * i)? I wonder if using an alias wouldn't be more
  appropriate. Some performance numbers would be very useful in any case.
 
  Will benchmark, though the issue here is that I like the foreach syntax
  and this can't be done with templates as you describe. Also, IIRC (I'll
  do the benchmark at some point to be sure) for such examples memory
  bandwidth is limiting, so the delegate call doesn't matter anyhow.
 I think anyone who wants to run something in parallel is motivated
 primarily by increasing efficiency, so focusing on aesthetics would be
 secondary and unsatisfactory for your target audience.

Ok, next questions, and things I've wanted clarified for a while:

1.  How are template alias parameters that reference variables in local scope
going to work once fully debugged?  There's a laundry list of Bugzilla issues
related to such aliases and the spec isn't very clear.

2.  As I'd strongly prefer to keep the foreach syntax, will opApply ever be
allowed to work with an alias instead of a delegate?  I ask because I was 
reading
the DMD source code a while back and saw this feature half implemented.  If so,
I'll just leave it alone for now and switch to an alias when this is 
implemented.

David Simcha's std.parallelism

[Andrei Alexandrescu]

I think David Simcha's library is close to reviewable form. Code:

http://dsource.org/projects/scrapple/browser/trunk/parallelFuture/std_parallelism.d

Documentation:

http://cis.jhu.edu/~dsimcha/d/phobos/std_parallelism.html

Here are a few comments:

* parallel is templated on range, but not on operation. Does this affect 
speed for brief operations (such as the one given in the example, 
squares[i] = i * i)? I wonder if using an alias wouldn't be more 
appropriate. Some performance numbers would be very useful in any case.


* Why is ThreadPool a class and what are the prospects of overriding its 
members?


* Can't we define the behavior of break inside a parallel loop?

* I think it does make sense to evaluate a parallel map lazily by using 
a finite buffer. Generally map looks the most promising so it may be 
worth investing some more work in it to make it smart lazy.


* waitStop() - join()?

* The documentation should use more examples. Currently it uses entities 
without defining them (Task, TaskPool etc.)


* why runCallable()? There's no runUncallable().

* Should there be a makeAngel to undo makeDaemon?

Overall, to prepare this for the review process, the documentation 
should be grown considerably with motivating examples and relevant 
benchmarks. We are modeling our review process after 
http://www.boost.org/community/reviews.html. The first realization of 
the process has been for std.datetime, and it seems to have been a success.



Andrei