Re: [Numpy-discussion] Report from SciPy
Travis E. Oliphant wrote: ... * Thus 1.2 will not break ABI compatibility but will add new API features. This is really great news (amongst the other good things). Many thanks for keeping the ABI compatible! All the best, Jon . ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Report from SciPy
Hi, I think this is a great idea but I am curious about what NumPy will be doing with Python 3. The Python 3 final is scheduled for 1st October release so is there a policy on handling the migration to Python 3 or dual support of the 2 and 3 series? Thanks Bruce On Sat, Aug 23, 2008 at 6:39 PM, Travis E. Oliphant [EMAIL PROTECTED] wrote: Hi everybody, Robert K, Chuck H, Stefan VdW, Jarrod M, David C, and I had a nice discussion about the future directions of NumPy. We resolved some things and would like community feedback on them if there are opinions. * we will be moving to time-based releases (at least 2 times a year -- November / May) with major changes not accepted about 4 weeks before the release. * The releases will be numbered major.minor.bugfix * There will be no ABI changes in minor releases * There will be no API changes in bugfix releases * Any API changes in minor releases will be done in a backward compatible fashion (possibly with deprecations). * Thus 1.2 will not break ABI compatibility but will add new API features. * We will push the generalized ufuncs off to 1.3 * NumPy 2.0 will be a library and will not automagically import numpy.fft * We will suggest that other libraries use from numpy import fft instead of import numpy as np; np.fft * We will apply most of Andrew Dalke's speed up patches but will keep ctypeslib import * We will remove automatic import of numpy.doc from trunk * NumPy 1.2 will come out shortly after the conference (rc1 on Sunday). * SciPy 0.7b1 will come out after the sprints. If there is anything else I missed from those who were there, please let us all know. By the way, as promised, the NumPy book is now available for download and the source to the book is checked in to the numpy SVN tree: http://svn.scipy.org/svn/numpy/trunk/numpy/doc/numpybook Best regards, -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Report from SciPy
Bruce Southey wrote: I think this is a great idea but I am curious about what NumPy will be doing with Python 3. The Python 3 final is scheduled for 1st October release so is there a policy on handling the migration to Python 3 or dual support of the 2 and 3 series? As a footnote to this query, has the Scons community addressed the problem Scons had working with VS 2008 (=VS9)? Cheers, Alan Isaac ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Report from SciPy
On Sun, Aug 24, 2008 at 10:58, Bruce Southey [EMAIL PROTECTED] wrote: Hi, I think this is a great idea but I am curious about what NumPy will be doing with Python 3. The Python 3 final is scheduled for 1st October release so is there a policy on handling the migration to Python 3 or dual support of the 2 and 3 series? We're still evaluating the situation. Rahul Garg has spent some time yesterday at the sprint looking at it. http://www.scipy.org/Python3k The situation is complicated for us because we have C code that uses some of the removed parts of the C API. The recommended approach (single 2.x source tree, and convert using 2to3 and manual patches for 3.x) isn't quite germane. If the necessary changes are small enough, then it is possible that manual patches or some #ifdefs will be sufficient. Given Rahul's initial findings (thank you, Rahul!), I think this will probably be feasible. If it is not, then we have a significant problem. If we have to have two different code bases, none of the alternatives are very appealing. We do want to support Python 3.0 as soon as possible, but we need more hands and eyes on the code. If Python 3.0 support is important to you, please take a look at Rahul's findings and think about how to address them. Thanks. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy08] Documentation BoF
On Fri, Aug 22, 2008 at 10:26 AM, Stéfan van der Walt [EMAIL PROTECTED] wrote: Hi all, This is my personal recollection of the documentation BoF. Feel free to comment or correct the text below. Regards Stéfan Summary of the Documentation Birds-of-a-Feather Session === Topics proposed for discussion -- - Path towards getting SciPy documented using the online documentation editor - Intersphinx: linking Sphinx-using projects - Culture / methodology: what do we expect from code accepted into NumPy/SciPy? - Tools for reading documentation (Help 2.0) - The complete SciPy book -- how to get there - NumPy Documentation Standard: is it useful beyond NumPy? Conclusions reached --- Documentation format - Question: could we standardise the descriptions of data-types between docs of different projects? They should overlap to some extent. - If we could agree on some common standard (or at least a common subset of a format) across projects, IPython could parse and display docstrings with special markup. - Matplotlib should be involved in this conversation. They have already written a vast amount of documentation, and their needs may be different to those of the SciPy projects. Formal requirements ``` For functions going into NumPy and SciPy, we have a certain minimum documentation requirements. However, we'd rather take a somewhat liberal, open approach, and accept code with somwhat inferior docstrings, working with the author to improve them. This way, we remain accessible as a community, while striving to improve the quality of the code base. Marketing SciPy ``` The current entry point for SciPy on the web (http://www.scipy.org) may be confusing to beginners. The web page should be redesigned, in a similar fashion as the new Sage and code.enthought.com pages, linking to the messy wiki behind it in a consistent fashion. Joe Harrington may be able to hire someone to work on this important aspect of SciPy marketing. Cross-project documentation searching ` Ideally, one should be able to perform a documentation search across several projects. In order to achieve this, we should negotiate a common convention for installing Sphinx-generated documentation into a standard location. A javascript-backed webpage can then be provided to do a search, using the json indices, or some other means. +1 Currently sphinx can't handle scipy docstrings, can it? It didn't for me at least. It'd be nice if whatever format you agre upon, could work with sphinx's autodoc. Also I am very interested in your web based editing of docstrings, so that people can just fix and improve the docstrings easily. I can imagine something like In [1]: e.series??? in ipython and it would fireup a browser pointing to the docstring and one would just fix it. Since you did most of the work already, maybe the above won't be that hard to do either. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Remove ParametricTestCase from numpy.testing
On Sat, Aug 23, 2008 at 10:23 PM, Alan McIntyre [EMAIL PROTECTED] wrote: Actually, it was removed right after the nose framework was working, but I think a decision was made to keep it around until 1.3 in case somebody was using it, so I put it back. :) After the 1.2 release it can just be deleted without any issues, I think. Could you through a deprecation warning in? Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Remove ParametricTestCase from numpy.testing
On Sun, Aug 24, 2008 at 3:20 PM, Jarrod Millman [EMAIL PROTECTED] wrote: On Sat, Aug 23, 2008 at 10:23 PM, Alan McIntyre [EMAIL PROTECTED] wrote: Actually, it was removed right after the nose framework was working, but I think a decision was made to keep it around until 1.3 in case somebody was using it, so I put it back. :) After the 1.2 release it can just be deleted without any issues, I think. Could you through a deprecation warning in? I suppose throw makes more sense. Sorry, J ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy08] Documentation BoF
On Sun, Aug 24, 2008 at 15:05, Ondrej Certik [EMAIL PROTECTED] wrote: Currently sphinx can't handle scipy docstrings, can it? It didn't for me at least. It'd be nice if whatever format you agre upon, could work with sphinx's autodoc. We do some preprocessing, I believe. Also I am very interested in your web based editing of docstrings, so that people can just fix and improve the docstrings easily. I can imagine something like In [1]: e.series??? in ipython and it would fireup a browser pointing to the docstring and one would just fix it. Since you did most of the work already, maybe the above won't be that hard to do either. The idea of having a this_docstring_sucks() function was certainly bounced around at the BoF. :-) -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] weights parameter of np.average() doesn't work (BUG?)
Hi all, Is there a good reason why the weights parameter of np.average() doesn't broadcast properly? This is with the Ubuntu Hardy x86_64 numpy package, version 1.0.4. In [293]: a=arange(100).reshape(10,10) # Things work fine when weights have the exact same shape as a In [297]: average(a, axis=1, weights=ones((10,10))) Out[297]: array([ 4.5, 14.5, 24.5, 34.5, 44.5, 54.5, 64.5, 74.5, 84.5, 94.5]) # Bizarre and incorrect result with length-10 weight array In [298]: average(a, axis=1, weights=ones(10)) Out[298]: array([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19.], [ 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 30., 31., 32., 33., 34., 35., 36., 37., 38., 39.], [ 40., 41., 42., 43., 44., 45., 46., 47., 48., 49.], [ 50., 51., 52., 53., 54., 55., 56., 57., 58., 59.], [ 60., 61., 62., 63., 64., 65., 66., 67., 68., 69.], [ 70., 71., 72., 73., 74., 75., 76., 77., 78., 79.], [ 80., 81., 82., 83., 84., 85., 86., 87., 88., 89.], [ 90., 91., 92., 93., 94., 95., 96., 97., 98., 99.] ) Doing the weighted-sum explicitly works fine for me: In [311]: sum(a*ones(10), axis=-1)/sum(ones(10)) Out[311]: array([ 4.5, 14.5, 24.5, 34.5, 44.5, 54.5, 64.5, 74.5, 84.5, 94.5]) This seems like a bug, especially since average.__doc__ states that: If weights are given, result is: sum(a * weights,axis) / sum(weights,axis), where the weights must have a's shape or be 1D with length the size of a in the given axis. Integer weights are converted to Float. Not specifying weights is equivalent to specifying weights that are all 1. Frankly, I don't even see why weights is constrained to be 1D or the same shape as a... why not anything that's broadcastable to the same shape as a? Dan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] weights parameter of np.average() doesn't work (BUG?)
On Sun, Aug 24, 2008 at 8:03 PM, Dan Lenski [EMAIL PROTECTED] wrote: Hi all, Is there a good reason why the weights parameter of np.average() doesn't broadcast properly? This is with the Ubuntu Hardy x86_64 numpy package, version 1.0.4. In [293]: a=arange(100).reshape(10,10) # Things work fine when weights have the exact same shape as a In [297]: average(a, axis=1, weights=ones((10,10))) Out[297]: array([ 4.5, 14.5, 24.5, 34.5, 44.5, 54.5, 64.5, 74.5, 84.5, 94.5]) # Bizarre and incorrect result with length-10 weight array In [298]: average(a, axis=1, weights=ones(10)) Out[298]: array([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19.], [ 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 30., 31., 32., 33., 34., 35., 36., 37., 38., 39.], [ 40., 41., 42., 43., 44., 45., 46., 47., 48., 49.], [ 50., 51., 52., 53., 54., 55., 56., 57., 58., 59.], [ 60., 61., 62., 63., 64., 65., 66., 67., 68., 69.], [ 70., 71., 72., 73., 74., 75., 76., 77., 78., 79.], [ 80., 81., 82., 83., 84., 85., 86., 87., 88., 89.], [ 90., 91., 92., 93., 94., 95., 96., 97., 98., 99.] ) This has been fixed in later versions: In [2]: a=arange(100).reshape(10,10) In [3]: average(a, axis=1, weights=ones(10)) Out[3]: array([ 4.5, 14.5, 24.5, 34.5, 44.5, 54.5, 64.5, 74.5, 84.5, 94.5]) However, broadcasting in the axis=None case is not allowed. The thinking is that the weights must be fully specified for the axis (or None) that is being averaged over. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] doing zillions of 3x3 determinants fast
Hi all, I need to take the determinants of a large number of 3x3 matrices, in order to determine for each of N points, in which of M tetrahedral cells they lie. I arrange the matrices in an ndarray of shape (N,M,5,3,3). As far as I can tell, Numpy doesn't have a function to do determinants over a specific set of axes... so I can't say: dets = np.linalg.det(a, axis1=-1, axis2=-1) Using an explicit Python loop is very slow... doing the determinants of 100,000 random 3x3 matrices in a loop and inserting the results into a preallocated results array takes about 5.1 seconds. So instead I coded up my own routine to do the determinants over the last 2 axes, based on the naive textbook formula for a 3x3 determinant: def det3(ar): a=ar[...,0,0]; b=ar[...,0,1]; c=ar[...,0,2] d=ar[...,1,0]; e=ar[...,1,1]; f=ar[...,1,2] g=ar[...,2,0]; h=ar[...,2,1]; i=ar[...,2,2] return (a*e*i+b*f*g+c*d*h)-(g*e*c+h*f*a+i*d*b) This is *much much* faster... it does the same 100,000 determinants in 0.07 seconds. And the results differ from linalg.det's results by less than 1 part in 10^15 on average (w/ a std dev of about 1 part in 10^13). Does anyone know of any well-written routines to do lots and lots of determinants of a fixed size? My routine has a couple problems I can see: * it's fast enough for 100,000 determinants, but it bogs due to all the temporary arrays when I try to do 1,000,000 determinants (=72 MB array) * I've heard the numerical stability of the naive determinant algorithms is fairly poor, so I'm reluctant to use this function on real data. Any advice will be appreciated! Dan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] weights parameter of np.average() doesn't work (BUG?)
On Sun, 24 Aug 2008 20:57:43 -0600, Charles R Harris wrote: On Sun, Aug 24, 2008 at 8:03 PM, Dan Lenski [EMAIL PROTECTED] wrote: This has been fixed in later versions: In [2]: a=arange(100).reshape(10,10) In [3]: average(a, axis=1, weights=ones(10)) Out[3]: array([ 4.5, 14.5, 24.5, 34.5, 44.5, 54.5, 64.5, 74.5, 84.5, 94.5]) Ah, good to know! I guess it's time to upgrade. However, broadcasting in the axis=None case is not allowed. The thinking is that the weights must be fully specified for the axis (or None) that is being averaged over. Hmm, that's a pity :-( My real-world use case has a 2D array of weights, shape (a,b), that I'd like to broadcast to a 3D array, shape (n,a,b), which I average over the first dimension. Currently, I just use repeat() to make it the right size... but this is less-than-ideal for huge arrays. Dan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] doing zillions of 3x3 determinants fast
On Mon, 25 Aug 2008 03:48:54 +, Daniel Lenski wrote: * it's fast enough for 100,000 determinants, but it bogs due to all the temporary arrays when I try to do 1,000,000 determinants (=72 MB array) I've managed to reduce the memory usage significantly by getting the number of temporary arrays down to exactly two: def det3(ar): a=ar[...,0,0]; b=ar[...,0,1]; c=ar[...,0,2] d=ar[...,1,0]; e=ar[...,1,1]; f=ar[...,1,2] g=ar[...,2,0]; h=ar[...,2,1]; i=ar[...,2,2] t=a.copy(); t*=e; t*=i; tot =t t=b.copy(); t*=f; t*=g; tot+=t t=c.copy(); t*=d; t*=h; tot+=t t=g.copy(); t*=e; t*=c; tot-=t t=h.copy(); t*=f; t*=a; tot-=t t=i.copy(); t*=d; t*=b; tot-=t return tot Now it runs very fast with 1,000,000 determinants to do (10X the time required to do 100,000) but I'm still worried about the stability with real-world data. Dan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] doing zillions of 3x3 determinants fast
2008/8/25 Daniel Lenski [EMAIL PROTECTED]: On Mon, 25 Aug 2008 03:48:54 +, Daniel Lenski wrote: * it's fast enough for 100,000 determinants, but it bogs due to all the temporary arrays when I try to do 1,000,000 determinants (=72 MB array) I've managed to reduce the memory usage significantly by getting the number of temporary arrays down to exactly two: def det3(ar): a=ar[...,0,0]; b=ar[...,0,1]; c=ar[...,0,2] d=ar[...,1,0]; e=ar[...,1,1]; f=ar[...,1,2] g=ar[...,2,0]; h=ar[...,2,1]; i=ar[...,2,2] t=a.copy(); t*=e; t*=i; tot =t t=b.copy(); t*=f; t*=g; tot+=t t=c.copy(); t*=d; t*=h; tot+=t t=g.copy(); t*=e; t*=c; tot-=t t=h.copy(); t*=f; t*=a; tot-=t t=i.copy(); t*=d; t*=b; tot-=t return tot Now it runs very fast with 1,000,000 determinants to do (10X the time required to do 100,000) but I'm still worried about the stability with real-world data. As far as I know, that's the best you can do (though come to think of it, a 3D determinant is a cross-product followed by a dot product, so you might be able to cheat and use some built-in routines). It's a current limitation of numpy that there is not much support for doing many linear algebra problems. We do have perennial discussions about improving support for array-of-matrices calculations, and in fact currently in numpy SVN is code to allow C code doing a 3x3 determinant to be easily made into a generalized ufunc that does ufunc-like broadcasting and iteration over all but the last two axes. Anne ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] weights parameter of np.average() doesn't work (BUG?)
On Sun, Aug 24, 2008 at 9:56 PM, Daniel Lenski [EMAIL PROTECTED] wrote: On Sun, 24 Aug 2008 20:57:43 -0600, Charles R Harris wrote: On Sun, Aug 24, 2008 at 8:03 PM, Dan Lenski [EMAIL PROTECTED] wrote: This has been fixed in later versions: In [2]: a=arange(100).reshape(10,10) In [3]: average(a, axis=1, weights=ones(10)) Out[3]: array([ 4.5, 14.5, 24.5, 34.5, 44.5, 54.5, 64.5, 74.5, 84.5, 94.5]) Ah, good to know! I guess it's time to upgrade. However, broadcasting in the axis=None case is not allowed. The thinking is that the weights must be fully specified for the axis (or None) that is being averaged over. Hmm, that's a pity :-( My real-world use case has a 2D array of weights, shape (a,b), that I'd like to broadcast to a 3D array, shape (n,a,b), which I average over the first dimension. Currently, I just use repeat() to make it the right size... but this is less-than-ideal for huge arrays. I expect you can do this explicitly, which is pretty much what average does anyway. Average is pure python, so you won't be losing any efficiency. If you were a bit clearer with your use case I could be more explicit. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] doing zillions of 3x3 determinants fast
On Sun, Aug 24, 2008 at 9:48 PM, Daniel Lenski [EMAIL PROTECTED] wrote: Hi all, I need to take the determinants of a large number of 3x3 matrices, in order to determine for each of N points, in which of M tetrahedral cells they lie. I arrange the matrices in an ndarray of shape (N,M,5,3,3). If you step back a bit and describe the actual problem you have, rather than your current solution, it might be that there are algorithms in scipy to solve it. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
