Yep -- building something like that directly into pyximport as a general utility would be the first step; then we could see after that.
There's already unmerged patches sitting for pyximport though. The real problem here is that pyximport doesn't have a testsuite, which makes making changes to it seem overwhelmingly daunting... Adding that would definitely be my first step in all of this. Since pyximport is expected to run in pure Python, I think it is really just about writing the tests; no test-compilation framework needed. Any volunteers? :-) Dag Sverre Seljebotn -----Original Message----- From: [email protected] Date: Wednesday, Aug 26, 2009 4:28 am Subject: Re: [Cython] scipy'09 cython BoF To: [email protected]: [email protected] On Tue, Aug 25, 2009 at 9:16 PM, Kurt Smith<[email protected]> wrote: >> On Tue, Aug 25, 2009 at 4:21 PM, Fernando Perez<[email protected]> wrote: >>> Hi folks, >>> >>> On Tue, Aug 25, 2009 at 12:08 PM, Kurt Smith<[email protected]> wrote: >>>> Weave-like support in Cython -- Fernando Perez (not at the BoF) >>>> suggested that Cython could assume the same functionality of >>>> scipy.weave (see: http://www.scipy.org/Weave), since scipy.weave is >>>> poorly documented and could use more maintenance than it is currently >>>> getting. I'm personally very interested in this -- it seems that >>>> there is much overlap between scipy.weave and the combination of >>>> pyximport & the pure-python mode of Cython. It seems that with a bit >>>> of work from an interested user and some guidance from yours truly, we >>>> could improve pyximport & Cython's pure-python mode to incorporate the >>>> good stuff from scipy.weave, but better ;-) I intend to write-up a >>>> CEP with my thoughts sometime in the near future. >>>> >>> >>> Kurt, thanks for resurrecting this comment. Indeed, weave is an >>> extremely useful piece of functionality but one that has been somewhat >>> neglected for years. There's a lot of good in there though, and the >>> need for it is significant. It's *really* cool to do things like: >>> >>> def prod(m, v): >>> """Matrix-vector multiply via weave/blitz++""" >>> nrows, ncolumns = m.shape >>> assert v.ndim==1 and ncolumns==v.shape[0],"Shape mismatch in prod" >>> >>> res = np.zeros(nrows, float) >>> code = r""" >>> for (int i=0; i<nrows; i++) >>> { >>> for (int j=0; j<ncolumns; j++) >>> { >>> res(i) += m(i,j)*v(j); >>> } >>> } >>> """ >>> err = inline(code,['nrows', 'ncolumns', 'res', 'm', 'v'], verbose=2, >>> type_converters=converters.blitz) >>> return res >> >> Here's what I'm envisioning for the above usecase -- I admit to not >> using the pure-python mode that much, so there may be errors (and some >> of what I'm using isn't implemented yet). But it shows how much is >> already there in Cython's pure-python mode. >> >> @cython.compile # this is new & where most of the work is required! >> @cython.locals(m=cython.float[:,::1], v=cython.float[::1]) >> def prod(m, v): >> cython.declare( >> i=cython.int, >> j=cython.int, >> nrows=cython.int, >> ncolumns=cython.int) >> nrows = m.shape[0]; ncolumns = m.shape[1] >> cython.declare(res=cython.float[::1]) >> res = np.zeros(nrows, np.float) >> for i in range(nrows): >> for j in range(ncolumns): >> res[i] += m[i,j] * v[j] >> return res >> >> Once the new array type / memoryview slice is merged into cython-devel >> and we get it integrated into pure-python mode, this will be close to >> working. If you just want things fast in the function body (what >> weave gives you currently) then the above is all you need. >> >> The hard part is getting things to compile & run automatically for >> just this function (the cython.compile decorator). This overlaps with >> pyximport a good amount, and it will take some thought as to how it >> will all work together. Perhaps the core devs won't like so much >> going on behind the scenes -- cython is principally a utility for >> generating a C extension module, not compiling & loading the extension >> module. >> >> Weave compiles individual snippets into their own extension module, >> whereas Cython has always compiled entire modules into extension >> modules, and requires a separate compilation step (whether through the >> commandline or through pyximport). The cython.compile decorator >> would be different, if it were to be just like weave -- it would >> compile a single function into its own extension module, without an >> explicit compilation step. Much more is automated. >> >> This seems to be the main thing that weave offers -- fast code only >> where you want it, without the explicit compilation step. Pyximport >> overlaps here, but pyximport still compiles an entire module, not just >> a part. >> >> So those are some of my thoughts. They'll be in a CEP sometime soon :-) >> >>> >>> This makes interactive experimentation and the writing of >>> loop/indexing-intensive code really easy. But by itself, weave is in >>> serious danger of bitrot, and right now Cython has the momentum and >>> concentrated effort for this problem domain. >>> >>> Everyone is extremely impressed with the amazing work you've all done, >>> and rooting for cython to continue to improve, as you are solving a >>> number of key problems to make python a really complete platform for >>> scientific computing. It's been lots of fun to see cython steadily >>> attack a number of problems that have been a real issue for years. >>> >>> So if Cython absorbs/extends/integrates/develops/whatever the good >>> ideas and code from weave, so that this type of workflow is also >>> supported, it would be great. I am really happy to see f2py grow >>> into fwrap, and the weave machinery (or at least parts of it) seems >>> like a logical addition to the cython-based toolkit. >>> >>> One particular use case for which weave.inline was invaluable to me in >>> the past was the generation of fast looping code for various >>> dimensions. I had to write function evaluation codes where the user >>> could supply a C expression (as a string, that would #define NDIM to >>> indicate the dimensionality of the problem). I could then easily >>> generate a few methods for D=1..6 (the range I needed to cover) that >>> would do certain key operations very fast. The bulk of the code used >>> numpy arrays to be dimension agnostic, but a few methods really needed >>> explicit loops and with weave, it was made completely transparent to >>> the user, all of it being done at runtime (for us, needing gcc at >>> runtime was OK). I mention this to give you some feedback on the use >>> cases for weave (there are others), but I hope that in the end, all of >>> this class of functionality will be available through a unified entry >>> point, and if that's cython, all the better. >>> >> >> Thanks for the encouragement, and for the examples. I don't have time >> to absorb all this just yet, but I'll return to it. And get other >> usecases for weave from you and others. >> >> Kurt >> >>> Thanks again to the team for your great work! >>> >>> Cheers, >>> >>> f > >(A comment from a non-expert) > >A more literal approach to imitating weave inline compilation, seems >already possible with cython and pyximport. To try out cython and to >reduce the time in debugging my (many) mistakes, a added the >compilation step directly to the script. This seems closer to the >usage of weave that Fernando was deswcribing and what I saw of weave >examples. > >Cheers, > >Josef >
_______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
