On 05/11/2016 06:48 PM, Sturla Molden wrote: > Elliot Hallmark <permafact...@gmail.com> wrote: >> Strula, this sounds brilliant! To be clear, you're talking about >> serializing the numpy array and reconstructing it in a way that's faster >> than pickle? > > Yes. We know the binary format of NumPy arrays. We don't need to invoke the > machinery of pickle to serialize an array and write the bytes to some IPC > mechanism (pipe, tcp socket, unix socket, shared memory). The choise of IPC > mechanism might not even be relevant, and could even be deferred to a > library like ZeroMQ. The point is that if multiple peocesses are to > cooperate efficiently, we need a way to let them communicate NumPy arrays > quickly. That is where using multiprocessing hurts today, and shared memory > does not help here. > > Sturla
You probably already know this, but I just wanted to note that the mpi4py module has worked around pickle too. They discuss how they efficiently transfer numpy arrays in mpi messages here: http://pythonhosted.org/mpi4py/usrman/overview.html#communicating-python-objects-and-array-data Of course not everyone is able to install mpi easily. _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion