I'd be nice to see a distributed array implemented on top of MPI (or similar high perf distribution libs) like Fortran co-arrays but since I'm out of academia and do not have access to "real" supercomputers anymore I'm actually more interested in wrappers to cloud base distributed computing frameworks like Spark (which do have distributed datastructures abstraction especially distributed dataframe).
On Friday, May 1, 2015 at 4:20:13 AM UTC+2, Jake Bolewski wrote: > > Yes, performance will be largely the same on 0.4. > > If you have to do any performance sensitive code at scale MPI is really > the only option I can recomend now. I don't know what you are trying to do > but the MPI.jl library is a bit incomplete so it would be great if you used > it and could contribute back in some way. All the basic operations should > be covered. > > -Jake > > On Thursday, April 30, 2015 at 12:29:15 PM UTC-4, Ángel de Vicente wrote: >> >> Hi Jake, >> >> Jake Bolewski <jakebo...@gmail.com> writes: >> > DistributedArray performance is pretty bad. The reason for removing >> > them from base was to spur their development. All I can say at this >> > time is that we are actively working on making their performance >> > better. >> >> OK, thanks. Should I try with the DistributedArray package in 0.4-dev or >> for the moment the performance will be similar? >> >> >> > For every parallel program you have implicit serial overhead (this is >> > especially true with multiprocessing). The fraction of serial work to >> > parallel work determines your potential parallel speedup. The >> > parallel work / serial overhead in this case is really bad, so I don't >> > think your observation is really surprising. If this is on a shared >> > memory machine I would try using SharedArray's as the serial >> > communication overhead will be lower, and the potential parallel >> > speedup much higher. DistributedArrays only really make sense if they >> > are in fact distributed over multiple machines. >> >> I will try SharedArray's, but the goal is to be able to run the code >> (not this one :-)) over distributed machines. For the moment my only >> hope is MPI.jl then? >> >> Thanks, >> -- >> Ángel de Vicente >> http://www.iac.es/galeria/angelv/ >> >
