Re: std.concurrency wrapper over MPI?

2011-08-11 Thread Jacob Carlborg
On 2011-08-11 13:07, dsimcha wrote: On 8/11/2011 4:14 AM, Jacob Carlborg wrote: On 2011-08-07 21:28, dsimcha wrote: In addition to the bug reports I filed, why is it necessary to write any serialization code to serialize through the base class? What's wrong with just doing something like: clas

Re: std.concurrency wrapper over MPI?

2011-08-11 Thread dsimcha
On 8/11/2011 7:07 AM, dsimcha wrote: On 8/11/2011 4:14 AM, Jacob Carlborg wrote: On 2011-08-07 21:28, dsimcha wrote: In addition to the bug reports I filed, why is it necessary to write any serialization code to serialize through the base class? What's wrong with just doing something like: cla

Re: std.concurrency wrapper over MPI?

2011-08-11 Thread dsimcha
On 8/11/2011 4:14 AM, Jacob Carlborg wrote: On 2011-08-07 21:28, dsimcha wrote: In addition to the bug reports I filed, why is it necessary to write any serialization code to serialize through the base class? What's wrong with just doing something like: class Base {} class Derived : Base {} vo

Re: std.concurrency wrapper over MPI?

2011-08-11 Thread Jacob Carlborg
On 2011-08-07 21:28, dsimcha wrote: In addition to the bug reports I filed, why is it necessary to write any serialization code to serialize through the base class? What's wrong with just doing something like: class Base {} class Derived : Base {} void main() { auto serializer = new Serializer(

Re: std.concurrency wrapper over MPI?

2011-08-09 Thread David Nadlinger
On 8/7/11 12:09 AM, dsimcha wrote: On 8/6/2011 5:38 PM, jdrewsen wrote: AFAIK David Nadlinger is handling serialization in his GSOC Thrift project that he is working on currently. Good to know, but what flavor? The most important thing to note, and the reason it could not be appropriate for

Re: std.concurrency wrapper over MPI?

2011-08-08 Thread Masahiro Nakagawa
On Mon, 08 Aug 2011 01:08:39 +0900, dsimcha wrote: On 8/7/2011 12:01 PM, Lutger Blijdestijn wrote: dsimcha wrote: On 8/7/2011 11:36 AM, Jacob Carlborg wrote: Currently, the only available format is XML. Ok, I'll look into writing a binary archiver that assumes that the CPU architecture on

Re: std.concurrency wrapper over MPI?

2011-08-07 Thread Jacob Carlborg
On 2011-08-07 21:28, dsimcha wrote: Yeah, I was trying to wrap my head around the whole "key" concept. I wasn't very successful. I also tried out Orange and filed a few bug reports. It may be that Orange isn't the right tool for the job for MPI, though modulo some bug fixing and polishing it coul

Re: std.concurrency wrapper over MPI?

2011-08-07 Thread Sean Kelly
This would probably work with the protobuf format. Sent from my iPhone On Aug 7, 2011, at 12:28 PM, dsimcha wrote: > On 8/7/2011 2:27 PM, Jacob Carlborg wrote: >> On 2011-08-07 17:45, dsimcha wrote: >>> On 8/7/2011 11:36 AM, Jacob Carlborg wrote: Currently, the only available format is XM

Re: std.concurrency wrapper over MPI?

2011-08-07 Thread dsimcha
On 8/7/2011 2:27 PM, Jacob Carlborg wrote: On 2011-08-07 17:45, dsimcha wrote: On 8/7/2011 11:36 AM, Jacob Carlborg wrote: Currently, the only available format is XML. Ok, I'll look into writing a binary archiver that assumes that the CPU architecture on the deserializing end is the same as t

Re: std.concurrency wrapper over MPI?

2011-08-07 Thread Jacob Carlborg
On 2011-08-07 18:01, Lutger Blijdestijn wrote: dsimcha wrote: On 8/7/2011 11:36 AM, Jacob Carlborg wrote: Currently, the only available format is XML. Ok, I'll look into writing a binary archiver that assumes that the CPU architecture on the deserializing end is the same as that on the seria

Re: std.concurrency wrapper over MPI?

2011-08-07 Thread Jacob Carlborg
On 2011-08-07 17:58, dsimcha wrote: On 8/7/2011 11:36 AM, Jacob Carlborg wrote: Good to know, but what flavor? As I see it there is a three-way tradeoff in serialization. In order of importance for distributed parallelism, the qualities are: I can answer these tradeoff for the Orange serializa

Re: std.concurrency wrapper over MPI?

2011-08-07 Thread Jacob Carlborg
On 2011-08-07 17:45, dsimcha wrote: On 8/7/2011 11:36 AM, Jacob Carlborg wrote: Currently, the only available format is XML. Ok, I'll look into writing a binary archiver that assumes that the CPU architecture on the deserializing end is the same as that on the serializing end. If it works, may

Re: std.concurrency wrapper over MPI?

2011-08-07 Thread Jacob Carlborg
On 2011-08-07 18:15, Sean Kelly wrote: I was mostly wondering if the serialized was all template code or if the archived portion used some form of polymorphism. Sounds like its the latter. The serializer uses template methods, the archive uses interfaces and virtual methods. Sent from my i

Re: std.concurrency wrapper over MPI?

2011-08-07 Thread dsimcha
On 8/6/2011 12:32 PM, Sean Kelly wrote: I'd love to be able to send classes between processes, but first we need a good serialization/deserialization mechanism. The more I think about it, the more I think that std.concurrency isn't quite the right interface for cluster parallelism. I'm think

Re: std.concurrency wrapper over MPI?

2011-08-07 Thread Sean Kelly
I was mostly wondering if the serialized was all template code or if the archived portion used some form of polymorphism. Sounds like its the latter. Sent from my iPhone On Aug 7, 2011, at 8:19 AM, Jacob Carlborg wrote: > On 2011-08-07 02:24, Sean Kelly wrote: >> Is the archive formatter dyna

Re: std.concurrency wrapper over MPI?

2011-08-07 Thread Sean Kelly
Nope. It would represent an external destination and defines the protocol. Sent from my iPhone On Aug 6, 2011, at 6:57 PM, dsimcha wrote: > On 8/6/2011 8:26 PM, Sean Kelly wrote: >> I'm hoping to simply extend the existing API. The crucial portion will be >> the addition of a Node (base) type

Re: std.concurrency wrapper over MPI?

2011-08-07 Thread dsimcha
On 8/7/2011 12:01 PM, Lutger Blijdestijn wrote: dsimcha wrote: On 8/7/2011 11:36 AM, Jacob Carlborg wrote: Currently, the only available format is XML. Ok, I'll look into writing a binary archiver that assumes that the CPU architecture on the deserializing end is the same as that on the seri

Re: std.concurrency wrapper over MPI?

2011-08-07 Thread Lutger Blijdestijn
dsimcha wrote: > On 8/7/2011 11:36 AM, Jacob Carlborg wrote: >> Currently, the only available format is XML. > > Ok, I'll look into writing a binary archiver that assumes that the CPU > architecture on the deserializing end is the same as that on the > serializing end. If it works, maybe Orange

Re: std.concurrency wrapper over MPI?

2011-08-07 Thread Lutger Blijdestijn
link for the D implementation: https://bitbucket.org/repeatedly/msgpack4d/

Re: std.concurrency wrapper over MPI?

2011-08-07 Thread dsimcha
On 8/7/2011 11:36 AM, Jacob Carlborg wrote: Good to know, but what flavor? As I see it there is a three-way tradeoff in serialization. In order of importance for distributed parallelism, the qualities are: I can answer these tradeoff for the Orange serialization library, http://dsource.org/proj

Re: std.concurrency wrapper over MPI?

2011-08-07 Thread dsimcha
On 8/7/2011 11:36 AM, Jacob Carlborg wrote: Currently, the only available format is XML. Ok, I'll look into writing a binary archiver that assumes that the CPU architecture on the deserializing end is the same as that on the serializing end. If it works, maybe Orange is a good choice.

Re: std.concurrency wrapper over MPI?

2011-08-07 Thread Jacob Carlborg
On 2011-08-07 00:09, dsimcha wrote: On 8/6/2011 5:38 PM, jdrewsen wrote: AFAIK David Nadlinger is handling serialization in his GSOC Thrift project that he is working on currently. /Jonas Good to know, but what flavor? As I see it there is a three-way tradeoff in serialization. In order of im

Re: std.concurrency wrapper over MPI?

2011-08-07 Thread Jacob Carlborg
On 2011-08-07 02:24, Sean Kelly wrote: Is the archive formatter dynamically pluggable? I'm not exactly sure what you mean but you can create new archive types and use them with the existing serializer. When creating a new serializer it takes an archive (as an interface) as a parameter. Sen

Re: std.concurrency wrapper over MPI?

2011-08-06 Thread dsimcha
On 8/6/2011 8:26 PM, Sean Kelly wrote: I'm hoping to simply extend the existing API. The crucial portion will be the addition of a Node (base) type. So Node would be the equivalent of Tid in the current API?

Re: std.concurrency wrapper over MPI?

2011-08-06 Thread Sean Kelly
I'm hoping to simply extend the existing API. The crucial portion will be the addition of a Node (base) type. Sent from my iPhone On Aug 6, 2011, at 2:38 PM, jdrewsen wrote: > Den 06-08-2011 05:51, dsimcha skrev: >> I've finally bitten the bullet and learned MPI >> (http://en.wikipedia.org/wi

Re: std.concurrency wrapper over MPI?

2011-08-06 Thread Sean Kelly
Is the archive formatter dynamically pluggable? Sent from my iPhone On Aug 6, 2011, at 11:51 AM, Jacob Carlborg wrote: > On 2011-08-06 18:32, Sean Kelly wrote: >> I'd love to be able to send classes between processes, but first we need a >> good serialization/deserialization mechanism. > > Ha

Re: std.concurrency wrapper over MPI?

2011-08-06 Thread dsimcha
On 8/6/2011 5:38 PM, jdrewsen wrote: AFAIK David Nadlinger is handling serialization in his GSOC Thrift project that he is working on currently. /Jonas Good to know, but what flavor? As I see it there is a three-way tradeoff in serialization. In order of importance for distributed parallel

Re: std.concurrency wrapper over MPI?

2011-08-06 Thread jdrewsen
Den 06-08-2011 05:51, dsimcha skrev: I've finally bitten the bullet and learned MPI (http://en.wikipedia.org/wiki/Message_passing_interface) for an ultra computationally intensive research project I've been working on lately. I wrote all the MPI-calling code in D against the C API, using a very q

Re: std.concurrency wrapper over MPI?

2011-08-06 Thread Jonathan M Davis
On Friday 05 August 2011 23:51:24 dsimcha wrote: > I've finally bitten the bullet and learned MPI > (http://en.wikipedia.org/wiki/Message_passing_interface) for an ultra > computationally intensive research project I've been working on lately. > I wrote all the MPI-calling code in D against the C

Re: std.concurrency wrapper over MPI?

2011-08-06 Thread Jacob Carlborg
On 2011-08-06 18:32, Sean Kelly wrote: I'd love to be able to send classes between processes, but first we need a good serialization/deserialization mechanism. Have a look at Orange, I don't know if it's considered good but it works for almost all types available in D, the only available arch

Re: std.concurrency wrapper over MPI?

2011-08-06 Thread Sean Kelly
I'd love to be able to send classes between processes, but first we need a good serialization/deserialization mechanism. Sent from my iPhone On Aug 5, 2011, at 8:51 PM, dsimcha wrote: > I've finally bitten the bullet and learned MPI > (http://en.wikipedia.org/wiki/Message_passing_interface)

Re: std.concurrency wrapper over MPI?

2011-08-06 Thread dsimcha
On 8/6/2011 2:57 AM, Russel Winder wrote: The main problem here is going to be that when anything gets released performance will be the only yardstick by which things are measured. Simplicity of code, ease of evolution of code, all the things professional developers value, will go out of the wind

Re: std.concurrency wrapper over MPI?

2011-08-06 Thread Russel Winder
On Sat, 2011-08-06 at 10:09 -0400, dsimcha wrote: [ . . . ] > Anyhow, D has one key advantage that makes it more tolerant of > communication overhead than most languages: std.parallelism. At least > the way things are set up on the cluster here at Johns Hopkins, each > node has 8 cores. The "

Re: std.concurrency wrapper over MPI?

2011-08-06 Thread dsimcha
On 8/6/2011 2:57 AM, Russel Winder wrote: The main problem here is going to be that when anything gets released performance will be the only yardstick by which things are measured. Simplicity of code, ease of evolution of code, all the things professional developers value, will go out of the win

Re: std.concurrency wrapper over MPI?

2011-08-06 Thread bearophile
dsimcha: > 1. Is anyone besides me interested in this? Other people are interested. > 3. Would this be Phobos material even though it would depend on MPI, or > would it better be kept as a 3rd party library? I'd like one or more Phobos modules built on top of the basic MPI, so I think it's

Re: std.concurrency wrapper over MPI?

2011-08-06 Thread Jacob Carlborg
On 2011-08-06 05:51, dsimcha wrote: 5. For passing complex object graphs, serialization would obviously be necessary. What's the current state of the art in serialization in D? I want something that's efficient and general first and foremost. I really don't care about human readability or standar

Re: std.concurrency wrapper over MPI?

2011-08-05 Thread Russel Winder
On Fri, 2011-08-05 at 23:51 -0400, dsimcha wrote: [ . . . ] > 1. Is anyone besides me interested in this? MPI may be ancient, it may be a bit daft in terms of its treatment of marshalling, unmarshalling and serializing, it may be only a Fortran and C thing bolted into C++ (quite well) but it is t

std.concurrency wrapper over MPI?

2011-08-05 Thread dsimcha
I've finally bitten the bullet and learned MPI (http://en.wikipedia.org/wiki/Message_passing_interface) for an ultra computationally intensive research project I've been working on lately. I wrote all the MPI-calling code in D against the C API, using a very quick-and-dirty (i.e. not releasabl