Re: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation
On 2013-09-19 23:12, Christoph Gohlke wrote:
On 9/19/2013 1:06 AM, Henry Gomersall wrote:
On 19/09/13 09:05, Henry Gomersall wrote:
I've had feedback that this is possible. Give me a few hours and I'll
see what I can do...
I mean that it builds under win 64-bit. I'll prob push a .exe.
Cheers,
Henry
It's probbly the wrong place to suggest/discuss this... To build PyFFTW
with MSVC one could replace the C99 headers and types with their numpy
counterparts, e.g. `double complex`-`np.complex128`. There would be no
need for `complex.h` or `stdint.h`. IIRC, with MSVC the `long double`
type is identical to `double`. PyFFTW will likely not work with mingw
compiled libfftw3l-3.dll and msvc compiled numpy. Maybe an option to
map
all fftwl functions to their fftw counterparts would work. That way it
will also be possible to link PyFFTW to Intel's MKL (instead of
libfftw3*-3.dll), which does not support `long double` types.
Yeah, these are the problems I'm trying to sort out. One strategy is to
simply leave out the long double transform schemes when
numpy.dtype('longdouble') == numpy.dtype('float64'), which means that
pyFFTW will work just fine with the mingw dlls as both numpy.longdouble
and numpy.float64 arrays would map to the double precision fftw
functions. This _should_ be seamless and transparent to the user.
A remaining issue is that there would be an ABI incompatibility between
pyFFTW and the (never called) fftwl binaries. It probably wouldn't
matter in practice, but it would upset me. Perhaps using MSVC compiled
FFTW is a better solution...
Henry
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation
On 18/09/13 01:51, Antony Lee wrote: While I realize that this is certainly tweaking multiprocessing beyond its specifications, I would like to use it on Windows to start a 32-bit Python process from a 64-bit Python process (use case: I need to interface with a 64-bit DLL and use an extension (pyFFTW) for which I can only find a 32-bit compiled version (yes, I could try to install MSVC and compile it myself but I'm trying to avoid that...)) There is now a release on PyPI including installers for both 32- and 64-bit Python 2.7, 3.2 and 3.3. The long double schemes are ignored as on 64-bit windows that type simply maps to double (though it should be seamless from the Python/Numpy end). All tests satisfied :) (that was some work!) Cheers, Henry ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation
Thanks a lot! Antony 2013/9/20 Henry Gomersall h...@cantab.net On 18/09/13 01:51, Antony Lee wrote: While I realize that this is certainly tweaking multiprocessing beyond its specifications, I would like to use it on Windows to start a 32-bit Python process from a 64-bit Python process (use case: I need to interface with a 64-bit DLL and use an extension (pyFFTW) for which I can only find a 32-bit compiled version (yes, I could try to install MSVC and compile it myself but I'm trying to avoid that...)) There is now a release on PyPI including installers for both 32- and 64-bit Python 2.7, 3.2 and 3.3. The long double schemes are ignored as on 64-bit windows that type simply maps to double (though it should be seamless from the Python/Numpy end). All tests satisfied :) (that was some work!) Cheers, Henry ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation
On Wed, Sep 18, 2013 at 2:51 AM, Antony Lee antony@berkeley.edu wrote: While I realize that this is certainly tweaking multiprocessing beyond its specifications, I would like to use it on Windows to start a 32-bit Python process from a 64-bit Python process (use case: I need to interface with a 64-bit DLL and use an extension (pyFFTW) for which I can only find a 32-bit compiled version (yes, I could try to install MSVC and compile it myself but I'm trying to avoid that...)) In the end, this may be the less painful solution, especially if speed is a requirement. Stéfan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation
On Wed, 18 Sep 2013 18:40:44 -0700 Antony Lee antony@berkeley.edu wrote: Thanks, I didn't know that multiprocessing Managers could be used with processes not started by multiprocessing itself... I will give them a try. I just need to compute FFTs, but speed is a real issue for me (I am using the results for real-time feedback). To be honest I don't know yet if the FFTs are going to be the limiting step but I thought I may as well give pyFFTW a try and ran into that issue... If FFTs are needed as well as speed, I got good results with PyCuda + CuFFT (and a GPU from Nvidia). -- Jérôme Kieffer On-Line Data analysis / Software Group ISDD / ESRF tel +33 476 882 445 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation
On 18/09/13 01:51, Antony Lee wrote: I need to interface with a 64-bit DLL and use an extension (pyFFTW) for which I can only find a 32-bit compiled version (yes, I could try to install MSVC and compile it myself but I'm trying to avoid that...)) I'm now in a position that I might be able to release a 64-bit Windows version of pyFFTW (I actually have a 64-bit installation of windows) Rather stupidly, one of the problems at the moment is that I can't upload the .msi files created under win7 to pypi! (the magic bytes check is failing due to a version mismatch). I've had feedback that this is possible. Give me a few hours and I'll see what I can do... Cheers, Henry ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation
On 19/09/13 09:05, Henry Gomersall wrote: I've had feedback that this is possible. Give me a few hours and I'll see what I can do... I mean that it builds under win 64-bit. I'll prob push a .exe. Cheers, Henry ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation
On Thu, Sep 19, 2013 at 2:40 AM, Antony Lee antony@berkeley.edu wrote: Thanks, I didn't know that multiprocessing Managers could be used with processes not started by multiprocessing itself... I will give them a try. I just need to compute FFTs, but speed is a real issue for me (I am using the results for real-time feedback). I am pretty sure that the overhead of communicating a large array from one process to another will vastly overwhelm any speed gains you get by using pyFFTW over numpy.fft. To be honest I don't know yet if the FFTs are going to be the limiting step but I thought I may as well give pyFFTW a try and ran into that issue... In that case, thinking about multiprocessing or even pyFFTW is far too premature. Implement your code with numpy.fft and see what performance you actually get. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation
On 2013-09-19 17:58, Antony Lee wrote: Henry: thanks a lot, that would be very appreciated regardless of whether I end up using it in this specific project or not. Other replies below. I've actually spent rather too long fiddling with Windows on this one! I can't for the life of me get cl.exe to compile the code. It has a long list of cryptic errors that gcc seems to not find at all :( I'll keep at it! hen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation
2013/9/19 Robert Kern robert.k...@gmail.com On Thu, Sep 19, 2013 at 5:58 PM, Antony Lee antony@berkeley.edu wrote: Henry: thanks a lot, that would be very appreciated regardless of whether I end up using it in this specific project or not. Other replies below. Antony 2013/9/19 Robert Kern robert.k...@gmail.com On Thu, Sep 19, 2013 at 2:40 AM, Antony Lee antony@berkeley.edu wrote: Thanks, I didn't know that multiprocessing Managers could be used with processes not started by multiprocessing itself... I will give them a try. I just need to compute FFTs, but speed is a real issue for me (I am using the results for real-time feedback). I am pretty sure that the overhead of communicating a large array from one process to another will vastly overwhelm any speed gains you get by using pyFFTW over numpy.fft. I would have hoped that the large arrays are simply written (from the beginning) to shared memory (what multiprocessing.sharedctypes.Array seems to do(?)) and that interprocess communication would be cheap enough (but what do I know about that). It certainly won't be automatic just by passing a numpy array to the manager. You will have to manually create the shared memory, pass its handle to the other process, and copy into it. But even the copy of the array may overwhelm the speed gains between PyFFTW and numpy.fft. If you can set it up such that the subprocess owns the shared memory for both input and output and the GUI process always writes into the input shared array directly and reads out the output shared array, then might work out okay. This works well when the inputs/outputs are always the same size. The arrays would always be the same size, and there is no array copy involved, as (I think that) I can have the C dll directly write whatever data needs to be analyzed to the shared memory array -- basically what you're suggesting. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation
On Thu, Sep 19, 2013 at 5:58 PM, Antony Lee antony@berkeley.edu wrote: Henry: thanks a lot, that would be very appreciated regardless of whether I end up using it in this specific project or not. Other replies below. Antony 2013/9/19 Robert Kern robert.k...@gmail.com On Thu, Sep 19, 2013 at 2:40 AM, Antony Lee antony@berkeley.edu wrote: Thanks, I didn't know that multiprocessing Managers could be used with processes not started by multiprocessing itself... I will give them a try. I just need to compute FFTs, but speed is a real issue for me (I am using the results for real-time feedback). I am pretty sure that the overhead of communicating a large array from one process to another will vastly overwhelm any speed gains you get by using pyFFTW over numpy.fft. I would have hoped that the large arrays are simply written (from the beginning) to shared memory (what multiprocessing.sharedctypes.Array seems to do(?)) and that interprocess communication would be cheap enough (but what do I know about that). It certainly won't be automatic just by passing a numpy array to the manager. You will have to manually create the shared memory, pass its handle to the other process, and copy into it. But even the copy of the array may overwhelm the speed gains between PyFFTW and numpy.fft. If you can set it up such that the subprocess owns the shared memory for both input and output and the GUI process always writes into the input shared array directly and reads out the output shared array, then might work out okay. This works well when the inputs/outputs are always the same size. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation
Henry: thanks a lot, that would be very appreciated regardless of whether I end up using it in this specific project or not. Other replies below. Antony 2013/9/19 Robert Kern robert.k...@gmail.com On Thu, Sep 19, 2013 at 2:40 AM, Antony Lee antony@berkeley.edu wrote: Thanks, I didn't know that multiprocessing Managers could be used with processes not started by multiprocessing itself... I will give them a try. I just need to compute FFTs, but speed is a real issue for me (I am using the results for real-time feedback). I am pretty sure that the overhead of communicating a large array from one process to another will vastly overwhelm any speed gains you get by using pyFFTW over numpy.fft. I would have hoped that the large arrays are simply written (from the beginning) to shared memory (what multiprocessing.sharedctypes.Array seems to do(?)) and that interprocess communication would be cheap enough (but what do I know about that). To be honest I don't know yet if the FFTs are going to be the limiting step but I thought I may as well give pyFFTW a try and ran into that issue... In that case, thinking about multiprocessing or even pyFFTW is far too premature. Implement your code with numpy.fft and see what performance you actually get. There is another (and, in fact, main) reason for me to use multiprocessing: the main app runs a GUI and running the data analysis in the same process just makes it painfully slow (I have tried that). Instead, running the data analysis in a separate process keeps the GUI responsive. Now whether the data analysis process should use numpy.fft or pyFFTW is a separate question; I realize that the gains from pyFFTW may probably be negligible compared to the other costs (... including the costs of tweaking multiprocessing beyond its specifications) but I was just giving it a try when I ran into the issue and was just puzzled by the error message I had never seen before. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation
On 9/19/2013 1:06 AM, Henry Gomersall wrote: On 19/09/13 09:05, Henry Gomersall wrote: I've had feedback that this is possible. Give me a few hours and I'll see what I can do... I mean that it builds under win 64-bit. I'll prob push a .exe. Cheers, Henry It's probbly the wrong place to suggest/discuss this... To build PyFFTW with MSVC one could replace the C99 headers and types with their numpy counterparts, e.g. `double complex`-`np.complex128`. There would be no need for `complex.h` or `stdint.h`. IIRC, with MSVC the `long double` type is identical to `double`. PyFFTW will likely not work with mingw compiled libfftw3l-3.dll and msvc compiled numpy. Maybe an option to map all fftwl functions to their fftw counterparts would work. That way it will also be possible to link PyFFTW to Intel's MKL (instead of libfftw3*-3.dll), which does not support `long double` types. Christoph ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation
On Wed, Sep 18, 2013 at 1:51 AM, Antony Lee antony@berkeley.edu wrote:
Hi all,
While I realize that this is certainly tweaking multiprocessing beyond
its specifications, I would like to use it on Windows to start a 32-bit
Python process from a 64-bit Python process (use case: I need to interface
with a 64-bit DLL and use an extension (pyFFTW) for which I can only find a
32-bit compiled version (yes, I could try to install MSVC and compile it
myself but I'm trying to avoid that...))
Just use subprocess to start up the 32-bit Python. If you want to use the
multiprocessing tools for communicating data, use a Manager server in the
32-bit Python to communicate over a socket.
http://docs.python.org/2/library/multiprocessing#managers
http://docs.python.org/2/library/multiprocessing#using-a-remote-manager
It is possible that this won't work if the protocol assumes that the
bitness is the same between server and client (e.g. struct.pack('Q', ...)),
but I suspect this is not the case.
You may also consider writing a small server using pyzmq or similar. I am
guessing that you are just calling one function from pyFFTW and getting the
result back. A simple REQ/REP server is easy to write with pyzmq. Do you
need to use pyFFTW for some specific functionality that is not available in
numpy.fft or scipy.fftpack?
--
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] multiprocessing, numpy and 32-64 bit cohabitation
Thanks, I didn't know that multiprocessing Managers could be used with
processes not started by multiprocessing itself... I will give them a try.
I just need to compute FFTs, but speed is a real issue for me (I am using
the results for real-time feedback). To be honest I don't know yet if the
FFTs are going to be the limiting step but I thought I may as well give
pyFFTW a try and ran into that issue...
Antony
2013/9/18 Robert Kern robert.k...@gmail.com
On Wed, Sep 18, 2013 at 1:51 AM, Antony Lee antony@berkeley.edu
wrote:
Hi all,
While I realize that this is certainly tweaking multiprocessing beyond
its specifications, I would like to use it on Windows to start a 32-bit
Python process from a 64-bit Python process (use case: I need to interface
with a 64-bit DLL and use an extension (pyFFTW) for which I can only find a
32-bit compiled version (yes, I could try to install MSVC and compile it
myself but I'm trying to avoid that...))
Just use subprocess to start up the 32-bit Python. If you want to use the
multiprocessing tools for communicating data, use a Manager server in the
32-bit Python to communicate over a socket.
http://docs.python.org/2/library/multiprocessing#managers
http://docs.python.org/2/library/multiprocessing#using-a-remote-manager
It is possible that this won't work if the protocol assumes that the
bitness is the same between server and client (e.g. struct.pack('Q', ...)),
but I suspect this is not the case.
You may also consider writing a small server using pyzmq or similar. I am
guessing that you are just calling one function from pyFFTW and getting the
result back. A simple REQ/REP server is easy to write with pyzmq. Do you
need to use pyFFTW for some specific functionality that is not available in
numpy.fft or scipy.fftpack?
--
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
