Re: review of std.parallelism

2011-03-21 Thread dsimcha
On 3/21/2011 11:58 AM, dsimcha wrote: == Quote from dsimcha (dsim...@yahoo.com)'s article == Quote from Michel Fortin (michel.for...@michelf.com)'s article On second thought, no, but for practical, not theoretical reasons: One, you can't introspect whether a foreach loop is using a ref or a val

Re: review of std.parallelism

2011-03-21 Thread dsimcha
== Quote from dsimcha (dsim...@yahoo.com)'s article > == Quote from Michel Fortin (michel.for...@michelf.com)'s article > > > On second thought, no, but for practical, not theoretical reasons: > > > One, you can't introspect whether a foreach loop is using a ref or a > > > value parameter. This is

Re: review of std.parallelism

2011-03-21 Thread dsimcha
== Quote from Michel Fortin (michel.for...@michelf.com)'s article > > On second thought, no, but for practical, not theoretical reasons: > > One, you can't introspect whether a foreach loop is using a ref or a > > value parameter. This is an issue with how opApply works. > Indeed a problem. Either

Re: review of std.parallelism

2011-03-21 Thread Michel Fortin
On 2011-03-21 09:50:09 -0400, dsimcha said: On 3/21/2011 8:37 AM, Michel Fortin wrote: Well, it'll work irrespective of whether shared delegates are used or not. I think you could add a compile-time check that the array element size is a multiple of the word size when the element is passed by

Re: review of std.parallelism

2011-03-21 Thread dsimcha
On 3/21/2011 8:37 AM, Michel Fortin wrote: Well, it'll work irrespective of whether shared delegates are used or not. I think you could add a compile-time check that the array element size is a multiple of the word size when the element is passed by ref in the loop and leave the clever trick as a

Re: review of std.parallelism

2011-03-21 Thread Michel Fortin
On 2011-03-20 23:21:49 -0400, dsimcha said: On 3/20/2011 10:44 PM, Michel Fortin wrote: I don't see a problem with the above. The array elements you modify are passed through parallel's opApply which can check easily whether it's safe or not to pass them by ref to different threads (by checki

Re: review of std.parallelism

2011-03-20 Thread dsimcha
On 3/20/2011 10:44 PM, Michel Fortin wrote: I don't see a problem with the above. The array elements you modify are passed through parallel's opApply which can check easily whether it's safe or not to pass them by ref to different threads (by checking the element's size) and allow or disallow th

Re: review of std.parallelism

2011-03-20 Thread Michel Fortin
On 2011-03-20 11:42:04 -0400, dsimcha said: On 3/19/2011 2:14 PM, Michel Fortin wrote: On 2011-03-19 13:20:00 -0400, dsimcha said: On 3/19/2011 1:09 PM, Michel Fortin wrote: For instance: void main() { int sum = 0; foreach (int value; taskPool.parallel([0,2,3,6,1,4,6,3,3,3,6])) { sum += v

Re: review of std.parallelism

2011-03-20 Thread Robert Jacques
On Sat, 19 Mar 2011 13:09:23 -0400, Michel Fortin wrote: On 2011-03-19 12:03:51 -0400, Andrei Alexandrescu said: * "Most of this module completely subverts..." Vague characterizations ("most", "completely", "some") don't belong in a technical documentation. (For example there's either

Re: review of std.parallelism

2011-03-20 Thread dsimcha
On 3/19/2011 10:16 PM, dsimcha wrote: * Again: speed of e.g. parallel min/max vs. serial, pi computation etc. on a usual machine? I **STRONGLY** believe this does not belong in API documentation because it's too machine specific, compiler specific, stack alignment specific, etc. and almost any

Re: review of std.parallelism

2011-03-20 Thread dsimcha
On 3/19/2011 2:14 PM, Michel Fortin wrote: On 2011-03-19 13:20:00 -0400, dsimcha said: On 3/19/2011 1:09 PM, Michel Fortin wrote: For instance: void main() { int sum = 0; foreach (int value; taskPool.parallel([0,2,3,6,1,4,6,3,3,3,6])) { sum += value; } writeln(sum); } The "+=" would need to

Re: review of std.parallelism

2011-03-20 Thread Lars T. Kyllingstad
On Sat, 19 Mar 2011 17:58:46 -0500, Andrei Alexandrescu wrote: > On 03/19/2011 04:25 PM, dsimcha wrote: >> On 3/19/2011 4:35 PM, Andrei Alexandrescu wrote: >>> I know you'd have no problem finding the right voice in this >>> discussion if you only frame it in the right light. Again, people are >>>

Re: review of std.parallelism

2011-03-19 Thread dsimcha
On 3/19/2011 4:35 PM, Andrei Alexandrescu wrote: On 03/19/2011 12:16 PM, dsimcha wrote: On 3/19/2011 12:03 PM, Andrei Alexandrescu wrote: On 03/19/2011 02:32 AM, dsimcha wrote: Ok, thanks again for clarifying **how** the docs could be improved. I've implemented the suggestions and generally gi

Re: review of std.parallelism

2011-03-19 Thread dsimcha
On 3/19/2011 8:48 PM, Jonathan M Davis wrote: On Saturday 19 March 2011 17:31:18 dsimcha wrote: On 3/19/2011 4:35 PM, Andrei Alexandrescu wrote: Furthermore, you should expect that the review process will prompt changes. My perception is that you consider the submission more or less final modul

Re: review of std.parallelism

2011-03-19 Thread Jonathan M Davis
On Saturday 19 March 2011 17:31:18 dsimcha wrote: > On 3/19/2011 4:35 PM, Andrei Alexandrescu wrote: > > Furthermore, you should expect that the review process will prompt > > changes. My perception is that you consider the submission more or less > > final modulo possibly a few minor nits. You sho

Re: review of std.parallelism

2011-03-19 Thread dsimcha
On 3/19/2011 4:35 PM, Andrei Alexandrescu wrote: Furthermore, you should expect that the review process will prompt changes. My perception is that you consider the submission more or less final modulo possibly a few minor nits. You shouldn't. I'm convinced you know much more about SMP than most o

Re: review of std.parallelism

2011-03-19 Thread dsimcha
On 3/19/2011 6:58 PM, Andrei Alexandrescu wrote: On 03/19/2011 04:25 PM, dsimcha wrote: On 3/19/2011 4:35 PM, Andrei Alexandrescu wrote: I know you'd have no problem finding the right voice in this discussion if you only frame it in the right light. Again, people are trying to help (however awk

Re: review of std.parallelism

2011-03-19 Thread dsimcha
On 3/19/2011 7:33 PM, Daniel Gibson wrote: Am 19.03.2011 22:45, schrieb dsimcha: IMHO all early termination that affects subsequent loop iterations as well (break, goto, labeled break and continue, but not regular continue) should just throw because they make absolutely no sense in a parallel co

Re: review of std.parallelism

2011-03-19 Thread Daniel Gibson
Am 19.03.2011 22:45, schrieb dsimcha: > IMHO all early termination that affects subsequent loop iterations as well > (break, goto, labeled break and continue, but not regular continue) should > just > throw because they make absolutely no sense in a parallel context. What about some kind of paral

Re: review of std.parallelism

2011-03-19 Thread Andrei Alexandrescu
On 03/19/2011 04:25 PM, dsimcha wrote: On 3/19/2011 4:35 PM, Andrei Alexandrescu wrote: I know you'd have no problem finding the right voice in this discussion if you only frame it in the right light. Again, people are trying to help (however awkwardly) and in no way is that ridiculous. Fair e

Re: review of std.parallelism

2011-03-19 Thread Jonathan M Davis
On Saturday 19 March 2011 09:03:51 Andrei Alexandrescu wrote: > On 03/19/2011 02:32 AM, dsimcha wrote: > > Ok, thanks again for clarifying **how** the docs could be improved. I've > > implemented the suggestions and generally given the docs a good reading > > over and clean up. The new docs are at:

Re: review of std.parallelism

2011-03-19 Thread dsimcha
On 3/19/2011 5:33 PM, Michel Fortin wrote: On 2011-03-19 15:36:24 -0400, dsimcha said: The only problem is that there's no easy, well-documented way to tell from the return value of opApply whether it was a break, a goto, a labeled break/continue, etc. This would be implementable only if I cha

Re: review of std.parallelism

2011-03-19 Thread Michel Fortin
On 2011-03-19 15:36:24 -0400, dsimcha said: On 3/19/2011 2:35 PM, Michel Fortin wrote: On 2011-03-19 13:16:02 -0400, dsimcha said: * "A goto from inside the parallel foreach loop to a label outside the loop will result in undefined behavior." Would this be a bug in dmd? No, it's because a

Re: review of std.parallelism

2011-03-19 Thread dsimcha
On 3/19/2011 4:35 PM, Andrei Alexandrescu wrote: On 03/19/2011 12:16 PM, dsimcha wrote: On 3/19/2011 12:03 PM, Andrei Alexandrescu wrote: On 03/19/2011 02:32 AM, dsimcha wrote: Ok, thanks again for clarifying **how** the docs could be improved. I've implemented the suggestions and generally gi

Re: review of std.parallelism

2011-03-19 Thread Andrei Alexandrescu
On 03/19/2011 12:16 PM, dsimcha wrote: On 3/19/2011 12:03 PM, Andrei Alexandrescu wrote: On 03/19/2011 02:32 AM, dsimcha wrote: Ok, thanks again for clarifying **how** the docs could be improved. I've implemented the suggestions and generally given the docs a good reading over and clean up. The

Re: review of std.parallelism

2011-03-19 Thread dsimcha
On 3/19/2011 3:36 PM, dsimcha wrote: On 3/19/2011 2:35 PM, Michel Fortin wrote: It gets rethrown when yieldWait()/spinWait()/workWait() is called. In the case of the higher-level primitives, it gets re-thrown to the calling thread at some non-deterministic point in the execution of these function

Re: review of std.parallelism

2011-03-19 Thread dsimcha
On 3/19/2011 2:35 PM, Michel Fortin wrote: On 2011-03-19 13:16:02 -0400, dsimcha said: * "A goto from inside the parallel foreach loop to a label outside the loop will result in undefined behavior." Would this be a bug in dmd? No, it's because a goto of this form has no reasonable, useful se

Re: review of std.parallelism

2011-03-19 Thread bearophile
dsimcha: > 1. I give up. > > 2. I wish someone had told me earlier. Please, don't give up. It's a lot of time from the first time I've asked a parallel map in Phobos :-) Bye, bearophile

Re: review of std.parallelism

2011-03-19 Thread dsimcha
On 3/19/2011 2:25 PM, Michel Fortin wrote: On 2011-03-19 14:14:51 -0400, Michel Fortin said: I'm not too convinced about the "I know what I'm doing" argument when I look at this example from asyncBuf's documentation: auto lines = File("foo.txt").byLine(); auto duped = map!"a.idup"(lines); //

Re: review of std.parallelism

2011-03-19 Thread Michel Fortin
On 2011-03-19 13:16:02 -0400, dsimcha said: * "A goto from inside the parallel foreach loop to a label outside the loop will result in undefined behavior." Would this be a bug in dmd? No, it's because a goto of this form has no reasonable, useful semantics. I should probably mention in the

Re: review of std.parallelism

2011-03-19 Thread Michel Fortin
On 2011-03-19 14:14:51 -0400, Michel Fortin said: I'm not too convinced about the "I know what I'm doing" argument when I look at this example from asyncBuf's documentation: auto lines = File("foo.txt").byLine(); auto duped = map!"a.idup"(lines); // Necessary b/c byLine() recycles b

Re: review of std.parallelism

2011-03-19 Thread Michel Fortin
On 2011-03-19 13:20:00 -0400, dsimcha said: On 3/19/2011 1:09 PM, Michel Fortin wrote: For instance: void main() { int sum = 0; foreach (int value; taskPool.parallel([0,2,3,6,1,4,6,3,3,3,6])) { sum += value; } writeln(sum); } The "+=" would need to be an atomic operation to avoid low-level r

Re: review of std.parallelism

2011-03-19 Thread dsimcha
On 3/19/2011 12:03 PM, Andrei Alexandrescu wrote: * "workUnitSize: The number of elements to evaluate in a single Task. Must be less than or equal to bufSize, and in practice should be a fraction of bufSize such that all worker threads can be used." Then why not specify a different parameter suc

Re: review of std.parallelism

2011-03-19 Thread dsimcha
On 3/19/2011 1:09 PM, Michel Fortin wrote: On 2011-03-19 12:03:51 -0400, Andrei Alexandrescu said: * "Most of this module completely subverts..." Vague characterizations ("most", "completely", "some") don't belong in a technical documentation. (For example there's either subversion going on or

Re: review of std.parallelism

2011-03-19 Thread dsimcha
On 3/19/2011 12:03 PM, Andrei Alexandrescu wrote: On 03/19/2011 02:32 AM, dsimcha wrote: Ok, thanks again for clarifying **how** the docs could be improved. I've implemented the suggestions and generally given the docs a good reading over and clean up. The new docs are at: http://cis.jhu.edu/~d

Re: review of std.parallelism

2011-03-19 Thread Michel Fortin
On 2011-03-19 12:03:51 -0400, Andrei Alexandrescu said: * "Most of this module completely subverts..." Vague characterizations ("most", "completely", "some") don't belong in a technical documentation. (For example there's either subversion going on or there isn't.) Also, std.concurrency and

Re: review of std.parallelism

2011-03-19 Thread Andrei Alexandrescu
On 03/19/2011 10:28 AM, dsimcha wrote: On 3/19/2011 10:54 AM, Andrei Alexandrescu wrote: Towards the bottom of the document there are overloads of task that don't have examples. You mean TaskPool.task()? Since these are such slight variations of the other overloads, I thought an example would

Re: review of std.parallelism

2011-03-19 Thread Andrei Alexandrescu
On 03/19/2011 02:32 AM, dsimcha wrote: Ok, thanks again for clarifying **how** the docs could be improved. I've implemented the suggestions and generally given the docs a good reading over and clean up. The new docs are at: http://cis.jhu.edu/~dsimcha/d/phobos/std_parallelism.html * When using

Re: review of std.parallelism

2011-03-19 Thread dsimcha
On 3/19/2011 10:54 AM, Andrei Alexandrescu wrote: On 03/18/2011 11:40 PM, dsimcha wrote: It should just be private. The fact that it's public is an artifact of when I was designing worker-local storage and didn't know how it was going to work yet. I never thought to revisit this until now. It r

Re: review of std.parallelism

2011-03-19 Thread Andrei Alexandrescu
On 03/18/2011 11:40 PM, dsimcha wrote: Thanks for the advice. You mentioned in the past that the documentation was inadequate but didn't give enough specifics as to how until now. As the author of the library, things seem obvious to me that don't seem obvious to anyone else, so I don't feel that

Re: review of std.parallelism

2011-03-19 Thread Simen kjaeraas
On Sat, 19 Mar 2011 13:51:56 +0100, dsimcha wrote: == Quote from Simen kjaeraas (simen.kja...@gmail.com)'s article On Sat, 19 Mar 2011 05:40:08 +0100, dsimcha wrote: > On 3/18/2011 11:29 PM, Andrei Alexandrescu wrote: >> 1. Library proper: >> >> * "In the case of non-random access ranges, par

Re: review of std.parallelism

2011-03-19 Thread dsimcha
== Quote from Simen kjaeraas (simen.kja...@gmail.com)'s article > On Sat, 19 Mar 2011 05:40:08 +0100, dsimcha wrote: > > On 3/18/2011 11:29 PM, Andrei Alexandrescu wrote: > >> 1. Library proper: > >> > >> * "In the case of non-random access ranges, parallel foreach is still > >> usable but buffers

Re: review of std.parallelism

2011-03-19 Thread Simen kjaeraas
On Sat, 19 Mar 2011 05:40:08 +0100, dsimcha wrote: On 3/18/2011 11:29 PM, Andrei Alexandrescu wrote: 1. Library proper: * "In the case of non-random access ranges, parallel foreach is still usable but buffers lazily to an array..." Wouldn't strided processing help? If e.g. 4 threads the first

Re: review of std.parallelism

2011-03-19 Thread dsimcha
Ok, thanks again for clarifying **how** the docs could be improved. I've implemented the suggestions and generally given the docs a good reading over and clean up. The new docs are at: http://cis.jhu.edu/~dsimcha/d/phobos/std_parallelism.html On 3/18/2011 11:29 PM, Andrei Alexandrescu wrote:

Re: review of std.parallelism

2011-03-18 Thread dsimcha
Thanks for the advice. You mentioned in the past that the documentation was inadequate but didn't give enough specifics as to how until now. As the author of the library, things seem obvious to me that don't seem obvious to anyone else, so I don't feel that I'm in a good position to judge the

review of std.parallelism

2011-03-18 Thread Andrei Alexandrescu
0. Overview and vote I think the library delivers the high-level goods (parallel foreach, map, reduce) but is a bit fuzzy in the lower level details. Documentation hurts understanding of the capabilities of the library and essentially is of inadequate quality. Entity documentation and example