On Tue, Aug 25, 2009 at 4:21 PM, Fernando Perez<[email protected]> wrote:
> Hi folks,
>
> On Tue, Aug 25, 2009 at 12:08 PM, Kurt Smith<[email protected]> wrote:
>> Weave-like support in Cython -- Fernando Perez (not at the BoF)
>> suggested that Cython could assume the same functionality of
>> scipy.weave (see: http://www.scipy.org/Weave), since scipy.weave is
>> poorly documented and could use more maintenance than it is currently
>> getting. I'm personally very interested in this -- it seems that
>> there is much overlap between scipy.weave and the combination of
>> pyximport & the pure-python mode of Cython. It seems that with a bit
>> of work from an interested user and some guidance from yours truly, we
>> could improve pyximport & Cython's pure-python mode to incorporate the
>> good stuff from scipy.weave, but better ;-) I intend to write-up a
>> CEP with my thoughts sometime in the near future.
>>
>
> Kurt, thanks for resurrecting this comment. Indeed, weave is an
> extremely useful piece of functionality but one that has been somewhat
> neglected for years. There's a lot of good in there though, and the
> need for it is significant. It's *really* cool to do things like:
>
> def prod(m, v):
> """Matrix-vector multiply via weave/blitz++"""
> nrows, ncolumns = m.shape
> assert v.ndim==1 and ncolumns==v.shape[0],"Shape mismatch in prod"
>
> res = np.zeros(nrows, float)
> code = r"""
> for (int i=0; i<nrows; i++)
> {
> for (int j=0; j<ncolumns; j++)
> {
> res(i) += m(i,j)*v(j);
> }
> }
> """
> err = inline(code,['nrows', 'ncolumns', 'res', 'm', 'v'], verbose=2,
> type_converters=converters.blitz)
> return res
Here's what I'm envisioning for the above usecase -- I admit to not
using the pure-python mode that much, so there may be errors (and some
of what I'm using isn't implemented yet). But it shows how much is
already there in Cython's pure-python mode.
@cython.compile # this is new & where most of the work is required!
@cython.locals(m=cython.float[:,::1], v=cython.float[::1])
def prod(m, v):
cython.declare(
i=cython.int,
j=cython.int,
nrows=cython.int,
ncolumns=cython.int)
nrows = m.shape[0]; ncolumns = m.shape[1]
cython.declare(res=cython.float[::1])
res = np.zeros(nrows, np.float)
for i in range(nrows):
for j in range(ncolumns):
res[i] += m[i,j] * v[j]
return res
Once the new array type / memoryview slice is merged into cython-devel
and we get it integrated into pure-python mode, this will be close to
working. If you just want things fast in the function body (what
weave gives you currently) then the above is all you need.
The hard part is getting things to compile & run automatically for
just this function (the cython.compile decorator). This overlaps with
pyximport a good amount, and it will take some thought as to how it
will all work together. Perhaps the core devs won't like so much
going on behind the scenes -- cython is principally a utility for
generating a C extension module, not compiling & loading the extension
module.
Weave compiles individual snippets into their own extension module,
whereas Cython has always compiled entire modules into extension
modules, and requires a separate compilation step (whether through the
commandline or through pyximport). The cython.compile decorator
would be different, if it were to be just like weave -- it would
compile a single function into its own extension module, without an
explicit compilation step. Much more is automated.
This seems to be the main thing that weave offers -- fast code only
where you want it, without the explicit compilation step. Pyximport
overlaps here, but pyximport still compiles an entire module, not just
a part.
So those are some of my thoughts. They'll be in a CEP sometime soon :-)
>
> This makes interactive experimentation and the writing of
> loop/indexing-intensive code really easy. But by itself, weave is in
> serious danger of bitrot, and right now Cython has the momentum and
> concentrated effort for this problem domain.
>
> Everyone is extremely impressed with the amazing work you've all done,
> and rooting for cython to continue to improve, as you are solving a
> number of key problems to make python a really complete platform for
> scientific computing. It's been lots of fun to see cython steadily
> attack a number of problems that have been a real issue for years.
>
> So if Cython absorbs/extends/integrates/develops/whatever the good
> ideas and code from weave, so that this type of workflow is also
> supported, it would be great. I am really happy to see f2py grow
> into fwrap, and the weave machinery (or at least parts of it) seems
> like a logical addition to the cython-based toolkit.
>
> One particular use case for which weave.inline was invaluable to me in
> the past was the generation of fast looping code for various
> dimensions. I had to write function evaluation codes where the user
> could supply a C expression (as a string, that would #define NDIM to
> indicate the dimensionality of the problem). I could then easily
> generate a few methods for D=1..6 (the range I needed to cover) that
> would do certain key operations very fast. The bulk of the code used
> numpy arrays to be dimension agnostic, but a few methods really needed
> explicit loops and with weave, it was made completely transparent to
> the user, all of it being done at runtime (for us, needing gcc at
> runtime was OK). I mention this to give you some feedback on the use
> cases for weave (there are others), but I hope that in the end, all of
> this class of functionality will be available through a unified entry
> point, and if that's cython, all the better.
>
Thanks for the encouragement, and for the examples. I don't have time
to absorb all this just yet, but I'll return to it. And get other
usecases for weave from you and others.
Kurt
> Thanks again to the team for your great work!
>
> Cheers,
>
> f
> _______________________________________________
> Cython-dev mailing list
> [email protected]
> http://codespeak.net/mailman/listinfo/cython-dev
>
_______________________________________________
Cython-dev mailing list
[email protected]
http://codespeak.net/mailman/listinfo/cython-dev
