Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
Hi! I was in fact experimenting with this. The solution seemed to lie in simple memmap as it is implemented in Windows: import numpy as N def arrSharedMemory(shape, dtype, tag=PriithonSharedMemory): Windows only ! share memory between different processes if same `tag` is used. itemsize = N.dtype(dtype).itemsize count = N.product(shape) size = count * itemsize import mmap sharedmem = mmap.mmap(0, size, tag) a=N.frombuffer(sharedmem, dtype, count) a.shape = shape return a For explaintion look up the microsoft site for the mmap documentation. And/or the Python-doc for mmap. (( I have to mention, that I could crash a process while testing this ... )) If anyone here would know an equivalent way of doing this on Linux/OS-X we were back to a cross-platfrom function. Hope this helps, Sebastian Haase On 10/9/07, David Cournapeau [EMAIL PROTECTED] wrote: Ray S wrote: Is anyone sharing arrays between processes on Windows? I tried compiling the posh sources (once, so far) with the new MS toolkit and failed... What other solutions are in use? Have a second process create an array view from an address would suffice for this particular purpose. I could pass the address as a parameter of the second process's argv. I've also tried things like pb=pythonapi.PyBuffer_FromReadWriteMemory(9508824, 9*sizeof(c_int)) N.frombuffer(pb, N.int32) which fails since pb is and int. What are my options? (disclaimer: I know nothing about windows idiosyncraties) Could not this be because you compiled the posh sources with a compiler/runtime which is different than the other extensions and python interpreter ? I don't know the details, but since most of the posix functions related to files and processes are broken beyond despair in windows, and in particular, many posix handles cannot cross dll boundaries compiled by different compilers, I would not be surprised if this cause some trouble. The fact that POSH is said to be posix-only on python.org (http://wiki.python.org/moin/ParallelProcessing) would imply that people do not care much about windows, too (but again, this is just from reading what posh is about; I have never used it personnally). cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
Sebastian Haase wrote: Hi! I was in fact experimenting with this. The solution seemed to lie in simple memmap as it is implemented in Windows: import numpy as N def arrSharedMemory(shape, dtype, tag=PriithonSharedMemory): Windows only ! share memory between different processes if same `tag` is used. itemsize = N.dtype(dtype).itemsize count = N.product(shape) size = count * itemsize import mmap sharedmem = mmap.mmap(0, size, tag) a=N.frombuffer(sharedmem, dtype, count) a.shape = shape return a For explaintion look up the microsoft site for the mmap documentation. And/or the Python-doc for mmap. (( I have to mention, that I could crash a process while testing this ... )) If anyone here would know an equivalent way of doing this on Linux/OS-X we were back to a cross-platfrom function. AFAIK, the tag thing is pretty much windows specific, so why not just ignoring it on non windows platforms ? (or interpreting the tag argument as the flag argument for mmap, which would be consistent with python mmap API ?) cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
On 10/9/07, David Cournapeau [EMAIL PROTECTED] wrote: Sebastian Haase wrote: Hi! I was in fact experimenting with this. The solution seemed to lie in simple memmap as it is implemented in Windows: import numpy as N def arrSharedMemory(shape, dtype, tag=PriithonSharedMemory): Windows only ! share memory between different processes if same `tag` is used. itemsize = N.dtype(dtype).itemsize count = N.product(shape) size = count * itemsize import mmap sharedmem = mmap.mmap(0, size, tag) a=N.frombuffer(sharedmem, dtype, count) a.shape = shape return a For explaintion look up the microsoft site for the mmap documentation. And/or the Python-doc for mmap. (( I have to mention, that I could crash a process while testing this ... )) If anyone here would know an equivalent way of doing this on Linux/OS-X we were back to a cross-platfrom function. AFAIK, the tag thing is pretty much windows specific, so why not just ignoring it on non windows platforms ? (or interpreting the tag argument as the flag argument for mmap, which would be consistent with python mmap API ?) As I recollect, the tag thing was the key for turning the mmap into a not really memmaped file, that is, a memmap without a corresponding file on the disk. In other words, isn't a mmap ( without(!) tag ) always bound to a real file in the file system ? -Sebastian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
At 05:22 AM 10/9/2007, David Cournapeau wrote: Could not this be because you compiled the posh sources with a compiler/runtime which is different than the other extensions and python interpreter ? It definitely was - since my 2.4 wanted the free 7.1 compiler, I (and anyone else who didn't download it in time) are now seemingly SOL since it is no longer available. I saw much discussion of this as well, but even 2.5 is now fixed on 7.1 and reports of compiling distutil modules with the new MS SDK and having them work at all with 2.4 were very mixed. I also tried GCC and had a litany of other errors with the posh. Sebastian Haase added: I was in fact experimenting with this. The solution seemed to lie in simple memmap as it is implemented in Windows: snip I had just found and started to write some tests with that MS function. If I can truly write to the array in one process and instantly read it in the other I'll be happy. Did you find that locks or semaphores were needed? (( I have to mention, that I could crash a process while testing this ... )) That was one of my first results! I also found that using ctypes to create arrays from the other process's address and laying a numpy array on top was prone to that in experimentation. But I had the same issue as Mark Heslep http://aspn.activestate.com/ASPN/Mail/Message/ctypes-users/3192422 of creating a numpy array from a raw address (not a c_array). Thanks, Ray Schumacher ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
On 10/9/07, Ray Schumacher [EMAIL PROTECTED] wrote: At 05:22 AM 10/9/2007, David Cournapeau wrote: Could not this be because you compiled the posh sources with a compiler/runtime which is different than the other extensions and python interpreter ? It definitely was - since my 2.4 wanted the free 7.1 compiler, I (and anyone else who didn't download it in time) are now seemingly SOL since it is no longer available. I saw much discussion of this as well, but even 2.5 is now fixed on 7.1 and reports of compiling distutil modules with the new MS SDK and having them work at all with 2.4 were very mixed. I also tried GCC and had a litany of other errors with the posh. Sebastian Haase added: I was in fact experimenting with this. The solution seemed to lie in simple memmap as it is implemented in Windows: snip I had just found and started to write some tests with that MS function. If I can truly write to the array in one process and instantly read it in the other I'll be happy. Did you find that locks or semaphores were needed? Maybe that's why it crashed ;-) !? But for simple use it seems fine. (( I have to mention, that I could crash a process while testing this ... )) That was one of my first results! I also found that using ctypes to create arrays from the other process's address and laying a numpy array on top was prone to that in experimentation. But I had the same issue as Mark Heslep http://aspn.activestate.com/ASPN/Mail/Message/ctypes-users/3192422 of creating a numpy array from a raw address (not a c_array). I assume this is a different issue, but haven't looked into it yet. -Sebastian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] diff of find of diff trick
Hello, As a converting MATLAB user I am bringing over some of my code. A core function I need is to sample the probability distribution of a given set of data. Sometimes these sets are large so I would like to make it as efficient as possible. (the data are integers representing members of a discrete space) In MATLAB the best way I found was the diff of find of diff trick which resulted in the completely vectorised solution (below). Does it make sense to translate this into numpy? I don't have a feel yet for what is fast/slow - are the functions below built in and so quick (I thought diff might not be). Is there a better/more pythonic way to do it? function Pr=prob(data, nR) Pr= zeros(nR,1); % diff of find of diff trick for counting number of elements temp = sort(data(data0));% want vector excluding P(0) dtemp= diff([temp;max(temp)+1]); count = diff(find([1;dtemp])); indx = temp(dtemp0); Pr(indx)= count ./ numel(data);% probability Thanks Robin ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Warnings
How can I get the line number of where a numpy warning message is envoked in my code? Cheers Tommy ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
On 10/9/07, Sebastian Haase replied: Did you find that locks or semaphores were needed? Maybe that's why it crashed ;-) !? But for simple use it seems fine. I just did some code (below) that does read/write to the array AFAP, and there is no crash, or any other issue (Win2000, py2.4, numpy 1.0b1). Without the print statements, it does max both processors; with printing I/O only 58%. Both processes can modify the array without issue either. I'll experiment with I had seen the Win mmap in this thread: http://objectmix.com/python/127666-shared-memory-pointer.html and here: http://www.codeproject.com/cpp/embedpython_2.asp Note also that the Python mmap docs read In either case you must provide a file descriptor for a file opened for update. and no mention of the integer zero descriptor option. Access options behave as presented. Because *NIX has MAP_SHARED as an option you'd think that there might be cross-platform share behavior with some platform checking if statements. Without a tag though, how does another process reference the same memory on NIX, a filename? (It seems) But I had the same issue as Mark Heslep http://aspn.activestate.com/ASPN/Mail/Message/ctypes-users/3192422 of creating a numpy array from a raw address (not a c_array). I assume this is a different issue, but haven't looked into it yet. Yes, a different methodology attempt. It would be interesting to know anyway how to create a numpy array from an address; it's probably buried in the undocumented C-API that I don't grok, and likely frowned upon. Thanks, Ray ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
