On Mon, Jul 2, 2018 at 7:42 PM Andrew Nelson <andyf...@gmail.com> wrote:
> <snip> > > On Tue, 3 Jul 2018 at 09:31, Charles R Harris <charlesr.har...@gmail.com> > wrote: > >> >> ISTR that some parallel processing applications sent pickled arrays >> around to different processes, I don't know if that is still the case, but >> if so, no copy might be a big gain for them. >> > > That is very much correct. One example is using MCMC, which is massively > parallel. I do parallelisation with mpi4py, and this requires distribution > of pickled data of a reasonable size to the entire MPI world. This pickling > introduces quite a bit of overhead. > Doesn’t mpi4py have support for buffered low-level communication of numpy arrays? See e.g. https://mpi4py.scipy.org/docs/usrman/tutorial.html Although I guess with Antoine’s proposal uses of the “lowercase” mpi4py API where data might get pickled will see speedups. _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion >
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
