Re: [Numpy-discussion] [SciPy08] Documentation BoF
Mon, 25 Aug 2008 00:05:06 +0200, Ondrej Certik wrote: > On Fri, Aug 22, 2008 at 10:26 AM, Stéfan van der Walt <[EMAIL PROTECTED]> > wrote: [clip] >> - Path towards getting SciPy documented using the online documentation >> editor >> - Intersphinx: linking Sphinx-using projects - Culture / methodology: >> what do we expect from code accepted into NumPy/SciPy? >> - Tools for reading documentation (Help 2.0) - The complete SciPy book >> -- how to get there - NumPy Documentation Standard: is it useful beyond >> NumPy? [clip] > > +1 > > Currently sphinx can't handle scipy docstrings, can it? It didn't for me > at least. It'd be nice if whatever format you agre upon, could work with > sphinx's autodoc. Sphinx offers enough hooks in autodoc for proper mangling of the docstrings, so the numpy format we have already agreed on can be used without problems. What we have now can be found here: https://code.launchpad.net/~pauli-virtanen/scipy/numpy-refguide https://code.launchpad.net/~stefanv/scipy/numpy-refguide Splitting the docstrings so that there's one docstring per page proved more difficult to do, especially if we want informative listings of the functions available. Currently our autosummary:: directive + automatically generated RST files can manage it, but it's not very elegant. Another question that pops up is how to integrate Travis's newly free Guide to Numpy to numpy documentation, as there is already some overlap with the docstrings and the reference part of the Guide. The main questions I see is that - What were the conclusions about this in the BoF and other discussions in Scipy'08? - Should we have a separate User manual and a Reference manual, or only a single manual? - What to do with the numpy.doc.* docstrings and Travis's book? There is obviously some overlap here. How to combine the material? - I vaguely remember someone mentioned having done numpybook -> RST conversion. If so, is the result somewhere available? Was something done towards this in the Scipy'08 sprints? I already wrote a rough LyX -> RST converter myself, but would like to avoid further duplication of work... > Also I am very interested in your web based editing of docstrings, so > that people can just fix and improve the docstrings easily. I can > imagine something like > > In [1]: e.series??? > > in ipython and it would fireup a browser pointing to the docstring and > one would just fix it. Since you did most of the work already, maybe the > above won't be that hard to do either. Pointing the browser at %(BASEURL)s/doc/%(obj.__name__)s/ should do it. -- Pauli Virtanen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Remove ParametricTestCase from numpy.testing
On Sun, Aug 24, 2008 at 3:20 PM, Jarrod Millman <[EMAIL PROTECTED]> wrote: > On Sun, Aug 24, 2008 at 3:20 PM, Jarrod Millman <[EMAIL PROTECTED]> wrote: >> On Sat, Aug 23, 2008 at 10:23 PM, Alan McIntyre <[EMAIL PROTECTED]> wrote: >>> Actually, it was removed right after the nose framework was working, >>> but I think a decision was made to keep it around until 1.3 in case >>> somebody was using it, so I put it back. :) After the 1.2 release it >>> can just be deleted without any issues, I think. >> >> Could you through a deprecation warning in? > > I suppose throw makes more sense. > Sorry, > J Done. :) ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Report from SciPy
Robert Kern wrote: > On Sun, Aug 24, 2008 at 10:58, Bruce Southey <[EMAIL PROTECTED]> wrote: > >> Hi, >> I think this is a great idea but I am curious about what NumPy will be >> doing with Python 3. The Python 3 final is scheduled for 1st October >> release so is there a policy on handling the migration to Python 3 or >> dual support of the 2 and 3 series? >> > > We're still evaluating the situation. Rahul Garg has spent some time > yesterday at the sprint looking at it. > > http://www.scipy.org/Python3k > > The situation is complicated for us because we have C code that uses > some of the removed parts of the C API. The recommended approach > (single 2.x source tree, and convert using 2to3 and manual patches for > 3.x) isn't quite germane. If the necessary changes are small enough, > then it is possible that manual patches or some #ifdefs will be > sufficient. Given Rahul's initial findings (thank you, Rahul!), I > think this will probably be feasible. If it is not, then we have a > significant problem. If we have to have two different code bases, none > of the alternatives are very appealing. > > We do want to support Python 3.0 as soon as possible, but we need more > hands and eyes on the code. If Python 3.0 support is important to you, > please take a look at Rahul's findings and think about how to address > them. > > Thanks. > > Thanks Rahul for that work as it is very interesting! At present my concern for the Python 3k is only one of inevitability. Bruce ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Report from SciPy
> As a footnote to this query, has the Scons community > addressed the problem Scons had working with VS 2008 (=VS9)? No. I'm still facing the same issues, but I'll try to help David Cournapeau with his work on that (he is trying to simplify all Visual Studio supports). Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] doing zillions of 3x3 determinants fast
On Sun, 24 Aug 2008 23:49:45 -0600, Charles R Harris wrote: > On Sun, Aug 24, 2008 at 9:48 PM, Daniel Lenski <[EMAIL PROTECTED]> > wrote: > >> Hi all, >> I need to take the determinants of a large number of 3x3 matrices, in >> order to determine for each of N points, in which of M tetrahedral >> cells they lie. I arrange the matrices in an ndarray of shape >> (N,M,5,3,3). >> >> > If you step back a bit and describe the actual problem you have, rather > than your current solution, it might be that there are algorithms in > scipy to solve it. > Hi Charles, I have an irregular/unstructured mesh of tetrahedral cells, and I need to interpolate some data over this mesh, at arbitrary points within the complete volume. So this is basically 3D interpolation. I've done some looking into this already, and it seems that the only facility for 3D interpolation in Scipy is map_coordinates, which only works on a regular grid. So I have to roll my own interpolation! I start by trying to figure out in which of the tetrahedral cells each interpolation point lies. The standard way to do this is to check that for every three vertices of each tetrahedron (v1,v2,v3), the point in question lies on the same side as the fourth point (v4). This can be checked with: | v1x-xv1y-y v1z-z | | v1x-v4xv1y-v4y v1z-v4z | | v2x-xv2y-y v2z-z | = | v2x-v4xv2y-v4y v2z-v4z | | v3x-xv3y-y v3z-z | | v3x-v4xv3y-v4y v3z-v4z | (Here's a description of a nearly identical algorithm: http:// steve.hollasch.net/cgindex/geometry/ptintet.html) Dan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy08] Documentation BoF
On Mon, Aug 25, 2008 at 4:18 AM, Pauli Virtanen <[EMAIL PROTECTED]> wrote: > - I vaguely remember someone mentioned having done numpybook -> RST > conversion. If so, is the result somewhere available? > Was something done towards this in the Scipy'08 sprints? Yes. Gabriel Gellner is working on this and is pretty far along. He is traveling at the moment, but should be checking something in later this week. Cheers, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy08] Documentation BoF
Pauli Virtanen wrote: > > - I vaguely remember someone mentioned having done numpybook -> RST > conversion. If so, is the result somewhere available? > Was something done towards this in the Scipy'08 sprints? > > Yes, Gabriel Gellner made progress on this during the sprints, and I asked him to post his scripts for doing it.I don't have his email address currently, but perhaps he will respond. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Inplace dot and blas access
Hi all, I'm sorry for what is probably a very simple question but despite looking I didn't manage to find anything. I have some code (solving linear differential equation) which is doing the following operation lots of times: S[:] = dot(A,S)+C here S is a (M,N) matrix, A is (M,M) and C is (M,1). M is usually smallish and N is largish. My question is, can this be turned into an inplace operation to save memory and time? As far as I can see, although some array functions allow you to specify the destination of the output, dot always generates a new array. So I would like to have something like this: dot(A,S, out=S) S+=C Ideally, I could do this in one line. I believe the BLAS DGEMM routine does what I want (if I converted C to a matrix rather than a vector). Is there any way to access this using Python and NumPy (e.g. something like numpy.blas.DGEMM) or would I need to use C code to do this? Many thanks in advance, Dan Goodman ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] subsampling array without loops
This almost works. Is there a way to do some masking on tiles, for instance taking the maximum height of each 2x2 square that is an odd number? I've tried playing around with masking and where, but they don't return an array of the original size and shape of "tiles" below. Catherine > Perhaps you could take the relevant slices of the arrays with > appropriate striding: > > a = numpy.arange(128*512).reshape((128, 512)) > top_left = a[::2, ::2] > top_right = a[::2, 1::2] > bottom_left = a[1::2, ::2] > bottom_right = a[1::2, 1::2] > > tiles = numpy.array([top_left, top_right, bottom_left, bottom_right]) > maxes = tiles.max(axis=0) > > Similarly if you want overlapping tiles, you could leave out the > final :2 in the slice specifications above. > > Zach > > > >> I'm looking for a way to acccomplish the following task without lots >> of loops involved, which are really slowing down my code. >> >> I have a 128x512 array which I want to break down into 2x2 squares. >> Then, for each 2x2 square I want to do some simple calculations >> such as finding the maximum value, which I then store in a 64x256 >> array. >> >> Here is the actual code involved. It's only slightly more complex >> than what I described above, since it also involves doing a bit of >> masking on the 2x2 sub-arrays. >> >> Any hints are much appreciated! An inordinate amount of time is >> being spent in this function and another one like it. >> >> Catherine >> >> def calc_sdcm_at_rlra(self,iblock): >> >> npixl = self.nlineht/self.nlinerl >> npixs = self.nsmpht/self.nsmprl >> >> sdcm_out = numpy.array([Constants.CLOUDMASK_NR] \ >>*self.nlinerl*self.nsmprl,'int8') \ >> .reshape(self.nlinerl,self.nsmprl) >> >> for ilinerl in range(0,self.nlinerl): >> for ismprl in range(0,self.nsmprl): >> >> height = self.data[iblock].height >> [ilinerl*2:ilinerl*2+2, \ >> ismprl*2:ismprl*2+2] >> sdcm = self.data[iblock].sdcm[ilinerl*2:ilinerl*2 >> +2, \ >> ismprl*2:ismprl*2+2] >> source = self.data[iblock].heightsrc >> [ilinerl*2:ilinerl*2+2, \ >> >> ismprl*2:ismprl*2+2] >> >> mask1 = (source == Constants.HEIGHT_STEREO) >> mask2 = ( (source == Constants.HEIGHT_SURFACE) | \ >> (source == Constants.HEIGHT_DEFAULT) ) >> >> if (mask1.any()): >> loc = height[mask1].argmax() >> sdcm_out[ilinerl,ismprl] = sdcm[mask1].ravel() >> [loc] >> elif (mask2.any()): >> loc = height[mask2].argmax() >> sdcm_out[ilinerl,ismprl] = sdcm[mask2].ravel() >> [loc] >> >> return sdcm_out > ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] subsampling array without loops
On Monday 25 August 2008 14:29:49 Catherine Moroney wrote: > This almost works. Is there a way to do some masking on tiles, for > instance taking the maximum height of each 2x2 square that is an > odd number? I've tried playing around with masking and where, but > they don't return an array of the original size and shape of "tiles" > below. Catherine, Sorry, I'm not following you: what were you *actually* doing with masking and where ? A short, self-contained example would be great. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] subsampling array without loops
> This almost works. Is there a way to do some masking on tiles, for > instance taking the maximum height of each 2x2 square that is an > odd number? I've tried playing around with masking and where, but > they don't return an array of the original size and shape of "tiles" > below. Could you provide an example (maybe not code, just descriptive) of what you want to do? I'm not sure what specifically you mean by "height" and "odd" above... Zach > Catherine > >> Perhaps you could take the relevant slices of the arrays with >> appropriate striding: >> >> a = numpy.arange(128*512).reshape((128, 512)) >> top_left = a[::2, ::2] >> top_right = a[::2, 1::2] >> bottom_left = a[1::2, ::2] >> bottom_right = a[1::2, 1::2] >> >> tiles = numpy.array([top_left, top_right, bottom_left, bottom_right]) >> maxes = tiles.max(axis=0) >> >> Similarly if you want overlapping tiles, you could leave out the >> final :2 in the slice specifications above. >> >> Zach >> >> >> >>> I'm looking for a way to acccomplish the following task without lots >>> of loops involved, which are really slowing down my code. >>> >>> I have a 128x512 array which I want to break down into 2x2 squares. >>> Then, for each 2x2 square I want to do some simple calculations >>> such as finding the maximum value, which I then store in a 64x256 >>> array. >>> >>> Here is the actual code involved. It's only slightly more complex >>> than what I described above, since it also involves doing a bit of >>> masking on the 2x2 sub-arrays. >>> >>> Any hints are much appreciated! An inordinate amount of time is >>> being spent in this function and another one like it. >>> >>> Catherine >>> >>>def calc_sdcm_at_rlra(self,iblock): >>> >>>npixl = self.nlineht/self.nlinerl >>>npixs = self.nsmpht/self.nsmprl >>> >>>sdcm_out = numpy.array([Constants.CLOUDMASK_NR] \ >>> *self.nlinerl*self.nsmprl,'int8') \ >>>.reshape(self.nlinerl,self.nsmprl) >>> >>>for ilinerl in range(0,self.nlinerl): >>>for ismprl in range(0,self.nsmprl): >>> >>>height = self.data[iblock].height >>> [ilinerl*2:ilinerl*2+2, \ >>> ismprl*2:ismprl*2+2] >>>sdcm = self.data[iblock].sdcm[ilinerl*2:ilinerl*2 >>> +2, \ >>>ismprl*2:ismprl*2+2] >>>source = self.data[iblock].heightsrc >>> [ilinerl*2:ilinerl*2+2, \ >>> >>> ismprl*2:ismprl*2+2] >>> >>>mask1 = (source == Constants.HEIGHT_STEREO) >>>mask2 = ( (source == Constants.HEIGHT_SURFACE) | \ >>> (source == Constants.HEIGHT_DEFAULT) ) >>> >>>if (mask1.any()): >>>loc = height[mask1].argmax() >>>sdcm_out[ilinerl,ismprl] = sdcm[mask1].ravel() >>> [loc] >>>elif (mask2.any()): >>>loc = height[mask2].argmax() >>>sdcm_out[ilinerl,ismprl] = sdcm[mask2].ravel() >>> [loc] >>> >>>return sdcm_out >> > ___ > Numpy-discussion mailing list > Numpy-discussion@scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy08] Documentation BoF
On Monday 25 August 2008 10:36:03 Travis E. Oliphant wrote: > Pauli Virtanen wrote: > > - I vaguely remember someone mentioned having done numpybook -> RST > > conversion. If so, is the result somewhere available? > > Was something done towards this in the Scipy'08 sprints? > > Yes, Gabriel Gellner made progress on this during the sprints, and I > asked him to post his scripts for doing it.I don't have his email > address currently, but perhaps he will respond. Note that a set of latex->reST converters seem to already exist there: http://svn.python.org/view/doctools/converter/ (haven't tried myself, so, YMMV) cheers, sebastien. -- ### # Dr. Sebastien Binet # Lawrence Berkeley National Lab. # 1 Cyclotron Road # Berkeley, CA 94720 ### ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Converting audio data to array for further processing
Hello all, I need an efficient way to convert 24 bit signed audio data to a numpy array for further processing. The data will be in a .wav file, and can be recovered via the python wave module. At that point it is a byte string - likely in little endian order. Somewhere in that data will be a 1 second length tone of prescribed but varying amplitude and frequency which must be measured for amplitude, frequency, duration, thd% etc. Also need to make the conversion quickly - over relatively large amounts of data. Something like 5 seconds sampled at 32000 samples/second. This same job will have to be repeated > 300 times as part of a test module. Thanks in advance for any and all ideas. Dan Colesworthy Otologics LLC ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] subsampling array without loops
2008/8/22 Catherine Moroney <[EMAIL PROTECTED]>: > I'm looking for a way to acccomplish the following task without lots > of loops involved, which are really slowing down my code. > > I have a 128x512 array which I want to break down into 2x2 squares. > Then, for each 2x2 square I want to do some simple calculations > such as finding the maximum value, which I then store in a 64x256 > array. You should be able to do some of this with reshape and transpose: In [1]: import numpy as np In [3]: A = np.zeros((128,512)) In [4]: B = np.reshape(A,(64,2,256,2)) In [5]: C = np.transpose(B,(0,2,1,3)) In [6]: C.shape Out[6]: (64, 256, 2, 2) Now C[i,j] is the 2 by 2 array [ [A[2*i,2*j], A[2*i,2*j+1]], [A[2*i+1,2*j], A[2*i+1, 2*j+1]] ] (or maybe I got those indices the wrong way around.) Now most operations you want to do on the two-by-two matrices can be done, using ufuncs, on the whole array at once. For example if you wanted the max of each two-by-two matrix, you'd write: In [7]: D = np.amax(np.amax(C,axis=-1),axis=-1) In [8]: D.shape Out[8]: (64, 256) Anne ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Buildbot failure
In case it hasn't been noted yet, three of the buildbots (Windows_XP_x86_64_MSVC, Linux_SPARC_64_Debian, Linux_SPARC_64_Debian_gcc4) are failing to build. The Linux builds have the error: gcc: build/src.linux-sparc64-2.4/numpy/core/src/umathmodule.c numpy/core/src/umathmodule.c.src:332: error: static declaration of 'trunc' follows non-static declaration The MSVC build appears to fail with build\src.win32-2.5\numpy\core\include\numpy\__umath_generated.c(580) : error C2065: 'truncf' : undeclared identifier ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Buildbot failure
On Mon, Aug 25, 2008 at 9:16 PM, Alan McIntyre <[EMAIL PROTECTED]>wrote: > In case it hasn't been noted yet, three of the buildbots > (Windows_XP_x86_64_MSVC, Linux_SPARC_64_Debian, > Linux_SPARC_64_Debian_gcc4) are failing to build. The Linux builds > have the error: > > gcc: build/src.linux-sparc64-2.4/numpy/core/src/umathmodule.c > numpy/core/src/umathmodule.c.src:332: error: static declaration of > 'trunc' follows non-static declaration > > The MSVC build appears to fail with > > build\src.win32-2.5\numpy\core\include\numpy\__umath_generated.c(580) > : error C2065: 'truncf' : undeclared identifier > _ Blame David ;) Anyway, I expect he will fix it up soon. Maybe we should shove some of these trickier fixes off to the next release? Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Converting audio data to array for further processing
On Mon, Aug 25, 2008 at 4:25 PM, Dan Colesworthy <[EMAIL PROTECTED]> wrote: > Hello all, > > > > I need an efficient way to convert 24 bit signed audio data to a numpy array > for further processing. The data will be in a .wav file, and can be > recovered via the python wave module. At that point it is a byte string – > likely in little endian order. There are some wav io facilities in scipy.io. If you need more advanced facilities and ndo not ming using LGPL-licensed code, there is scikits.audiolab which is a wrapper around sndfile and supports a large range of audio file formats: http://www.mega-nerd.com/libsndfile/ http://projects.scipy.org/scipy/scikits david ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
