On 12/04/14 01:07, Sturla Molden wrote: >> ATM the only other way to work with >> a data set that's larger than memory-divided-by-numcpus is to >> explicitly set up shared memory, and this is *really* hard for >> anything more complicated than a single flat array. > > > Not difficult. You just go to my GitHub site and grab the code ;) > > (I have some problems running it on my MBP though, not sure why, but it > used to work on Linux and Windows, and possibly still does.) > > https://github.com/sturlamolden/sharedmem-numpy
Hmm, today it works fine on my MBP too... Good. :) import multiprocessing as mp import numpy as np import sharedmem as shm def proc(qin, qout): print("grabbing array from queue") a = qin.get() print(a) print("putting array in queue") b = shm.zeros(10) print(b) qout.put(b) print("waiting for array to be updated by another process") a = qin.get() print(b) if __name__ == "__main__": qin = mp.Queue() qout = mp.Queue() p = mp.Process(target=proc, args=(qin,qout)) p.start() a = shm.zeros(4) qin.put(a) b = qout.get() b[:] = range(10) qin.put(None) p.join() sturla$ python example.py grabbing array from queue [ 0. 0. 0. 0.] putting array in queue [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.] waiting for array to be updated by another process [ 0. 1. 2. 3. 4. 5. 6. 7. 8. 9.] Sturla _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion