Re: [Numpy-discussion] Best way to insert C code in numpy code
René Dudfield wrote: > On Mon, Sep 21, 2009 at 8:12 PM, David Warde-Farley > wrote: > >> On 21-Sep-09, at 2:55 PM, Xavier Gnata wrote: >> >> >>> Should I read that to learn you cython and numpy interact? >>> Or is there another best documentation (with examples...)? >>> >> You should have a look at the Bresenham algorithm thread you posted. I >> went to the trouble of converting some Python code for Bresenham's >> algorithm to Cython, and a pointer to the Cython+NumPy tutorial: >> >> http://wiki.cython.org/tutorials/numpy >> >> David >> > > I don't know about the best way... but here are two approaches I like... > > Another way is to make your C function then load it with ctypes(or > wrap it with something else) and pass it pointers with > array.ctype.data. You can find the shape of the array in python, and > pass it to your C function. The benefit is it's just C code, and you > can avoid the GIL too if you want. Then if you keep your C code > separate from python stuff other people can use your C code in other > languages more easily. > > cinpy is another one(similar to weave) which can be good for fast > prototyping... http://www.cs.tut.fi/~ask/cinpy/ or for changing the > code to fit your data. > > Well I would have to find/read docs to be able to try that solution. cython looks easy :) cheers ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to insert C code in numpy code
On Tue, Sep 22, 2009 at 3:45 PM, Sturla Molden wrote:
> Xavier Gnata skrev:
>> I have a large 2D numpy array as input and a 1D array as output.
>> In between, I would like to use C code.
>> C is requirement because it has to be fast and because the algorithm
>> cannot be written in a numpy oriented way :( (no way...really).
>>
> There are certain algorithms that cannot be vectorized, particularly
> those that are recursive/iterative.
Hi,
one thing you can do is guess(predict) what the previous answer is and
continue on a path from that guess. Say you have 1000 processing
units, you could get the other 999 working on guesses for the answers
and go from there. If one of your guess paths is right, you might be
able to skip a bunch of steps. That's what cpus do with their
'speculative execution and branch prediction'. even more OT, you can
take advantage of this sometimes by getting the cpu to work out
multiple things for you at once by putting in well placed if/elses,
but that's fairly cpu specific. It's also used with servers... you
ask say 5 servers to give you a result, and wait for the first one to
give you the answer.
That cython pure module looks cool, thanks for pointing it out.
I wonder if anyone has tried using that with traces? So common paths
in your code record the types, and then can be used by the cython pure
module to try and generate the types for you? It could generate a
file containing a {callable : @cython.locals} mapping to be used on
compilation. Similar to how psyco works, but with a separate run
program/compilation step.
cheers,
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to insert C code in numpy code
Xavier Gnata skrev: > I have a large 2D numpy array as input and a 1D array as output. > In between, I would like to use C code. > C is requirement because it has to be fast and because the algorithm > cannot be written in a numpy oriented way :( (no way...really). > There are certain algorithms that cannot be vectorized, particularly those that are recursive/iterative. One example is MCMC methods such as the Gibbs sampler. You can get around it by running multiple Markov chains in parallel, and vectorizing this parallelism with NumPy. But you cannot vectorize one long chain. Vectorizing with NumPy only applies to data parallel problems. But then there is a nice tool you should not about: Cython in pure Python mode. You just add some annotations to the Python code, and the .py file can be compiled to efficient C. http://wiki.cython.org/pure This is quite similar in spirit to the optional static typing that makes certain implementations of Common Lisp (CMUCL, SBCL, Franz) so insanely fast. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to insert C code in numpy code
René Dudfield skrev: > Another way is to make your C function then load it with ctypes Also one should beware that ctypes is a stable part of the Python standard library. Cython is still unstable and in rapid development. Pyrex is more stabile than Cython, but interfacing with ndarrays is harder. If you have a requirement on not using experimental code, then Cython is not an option. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to insert C code in numpy code
René Dudfield skrev: > Another way is to make your C function then load it with ctypes(or > wrap it with something else) and pass it pointers with > array.ctype.data. numpy.ctypeslib.ndpointer is preferred when using ndarrays with ctypes. > You can find the shape of the array in python, and > pass it to your C function. The benefit is it's just C code, and you > can avoid the GIL too if you want. Then if you keep your C code > separate from python stuff other people can use your C code in other > languages more easily. You can do this with Cython as well, just use Cython for the glue code. The important difference is this: Cython is a language for writing C extensions, ctypes is a module for calling DLLs. One important advantage of Cython is deterministic clean-up code. If you put a __dealloc__ method in a "cdef class", it will be called on garbage collection. Another nice way of interfacing C with numpy is f2py. It also works with C, not just Fortran. Yet another way (Windows specific) is to use win32com.client and pass array.ctype.data. That is nice if you have an ActiveX control; for Windows you often get commercial libraries like that. Also if you have .NET or Java objects, you can easily expose them to COM. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to insert C code in numpy code
On Mon, Sep 21, 2009 at 8:12 PM, David Warde-Farley wrote: > On 21-Sep-09, at 2:55 PM, Xavier Gnata wrote: > >> Should I read that to learn you cython and numpy interact? >> Or is there another best documentation (with examples...)? > > You should have a look at the Bresenham algorithm thread you posted. I > went to the trouble of converting some Python code for Bresenham's > algorithm to Cython, and a pointer to the Cython+NumPy tutorial: > > http://wiki.cython.org/tutorials/numpy > > David I don't know about the best way... but here are two approaches I like... Another way is to make your C function then load it with ctypes(or wrap it with something else) and pass it pointers with array.ctype.data. You can find the shape of the array in python, and pass it to your C function. The benefit is it's just C code, and you can avoid the GIL too if you want. Then if you keep your C code separate from python stuff other people can use your C code in other languages more easily. cinpy is another one(similar to weave) which can be good for fast prototyping... http://www.cs.tut.fi/~ask/cinpy/ or for changing the code to fit your data. cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to insert C code in numpy code
I give my vote to cython as well. I have a program which uses cython for a portion simply because it was easier using a simple C for-loop to do what i wanted rather than beating numpy into submission. It was an order of magnitude faster as well. Cheers, Chris On Mon, Sep 21, 2009 at 9:12 PM, David Warde-Farley wrote: > On 21-Sep-09, at 2:55 PM, Xavier Gnata wrote: > >> Should I read that to learn you cython and numpy interact? >> Or is there another best documentation (with examples...)? > > You should have a look at the Bresenham algorithm thread you posted. I > went to the trouble of converting some Python code for Bresenham's > algorithm to Cython, and a pointer to the Cython+NumPy tutorial: > > http://wiki.cython.org/tutorials/numpy > > David > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to insert C code in numpy code
On 21-Sep-09, at 2:55 PM, Xavier Gnata wrote: > Should I read that to learn you cython and numpy interact? > Or is there another best documentation (with examples...)? You should have a look at the Bresenham algorithm thread you posted. I went to the trouble of converting some Python code for Bresenham's algorithm to Cython, and a pointer to the Cython+NumPy tutorial: http://wiki.cython.org/tutorials/numpy David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to insert C code in numpy code
Xavier Gnata wrote: > David Cournapeau wrote: >> That's only a data point, but I almost always use cython in those cases, I'm a second data point, but I think there are many more. Judging from the SciPy conference, Cython is the preferred method for new projects. > Should I read that to learn you cython and numpy interact? > http://wiki.cython.org/tutorials/numpy That's probably the best starting point. Also look online for the videos of the presentations at the SciPy2009 conference -- there were a few Cython ones. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to insert C code in numpy code
David Cournapeau wrote: > Xavier Gnata wrote: > >> Hi, >> >> I have a large 2D numpy array as input and a 1D array as output. >> In between, I would like to use C code. >> C is requirement because it has to be fast and because the algorithm >> cannot be written in a numpy oriented way :( (no way...really). >> >> Which tool should I use to achieve that? waeve.inline? pyrex? What is >> the baseline? >> >> > > That's only a data point, but I almost always use cython in those cases, > unless I need 'very advanced' features of the C API in which case I just > do it manually. > > cheers, > > David > Ok :) Should I read that to learn you cython and numpy interact? Or is there another best documentation (with examples...)? Xavier http://wiki.cython.org/tutorials/numpy ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to insert C code in numpy code
Xavier Gnata wrote: > Hi, > > I have a large 2D numpy array as input and a 1D array as output. > In between, I would like to use C code. > C is requirement because it has to be fast and because the algorithm > cannot be written in a numpy oriented way :( (no way...really). > > Which tool should I use to achieve that? waeve.inline? pyrex? What is > the baseline? > That's only a data point, but I almost always use cython in those cases, unless I need 'very advanced' features of the C API in which case I just do it manually. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
