[Numpy-discussion] minimal numpy ?
for use in binary distribution where I need only basics and fast startup/low memory footprint, I try to isolate the minimal ndarray type and what I need.. with "import numpy" or "import numpy.core.multiarray" almost the whole numpy package tree is imported, _dotblas etc. cxFreeze produces some 10MB numpy baggage (4MB zipped) yet when copying and using the multiarray DLL only, I can create arrays, but he most things fail: >>> import multiarray >>> x=multiarray.array([5,6]) >>> x+x Traceback (most recent call last): File "", line 1, in TypeError: unsupported operand type(s) for +: 'numpy.ndarray' and 'numpy.ndarray' while this works: >>> b=numpy.core.multiarray.array([3,9]) >>> b+b array([ 6, 18]) I added some things from core.__init__.py like this: import umath import _internal # for freeze programs import numerictypes as nt multiarray.set_typeDict(nt.sctypeDict) .. but the problem of failed type self-recognition remains. What is this? What to do? ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] minimal numpy ?
David Cournapeau wrote: Robert wrote: for use in binary distribution where I need only basics and fast startup/low memory footprint, I try to isolate the minimal ndarray type and what I need.. [..] I think you need at least umath to make this work: when doing import numpy.core.multiarray, you pull out the whole numpy (because import foo.bar induces import foo I believe), whereas import multiarray just imports the multiarray C extension. So my suggestion would be to modify numpy such as you can do import numpy after having removed most directories inside numpy. The big ones are distutils and f2py, which should already save 2.5 Mb and are not used at all in numpy itself. IIRC, the only problematic package is numpy.lib (we import numpy.lib in numpy.core IIRC). Did like this - keeping a /numpy/core folder structure. In attachment is a README-minimal-numpy.txt Maybe thats interesting for many users / inclusion somewhere in the docs. Result is: some 300kB compressed. And startup very fast. Strange: most imports in the package are relative - which is good for (flat) repackaging. Just one absolutue "from numpy.core.multiarray import ..." in a py file. Yet the 2 remaining DLL's obviously contain absolute imports of each other. Just because of that it is not possible to have the "minimal numpy" in a separate package folder with other name (without recompiling), but one needs to rename/remove the original numpy from the PYTHONPATH :-( Maybe the absolute imports could be removed out of the DLLs in future. By #ifdef or so in the C code the newer Pythons also can be forced to do precisely relative import. Robert HOW TO create a minimal numpy ? (for fast import or freezing) == rk 2009-05-16 * Make a copy of the original numpy folder tree * ( Rename the original tree / remove from sys.path ) * Keep only those files/folders: numpy/ -rw-rw-rw- 1 user group 82 May 16 10:58 __init__.py -rw-rw-rw- 1 user group 593 Apr 28 21:16 version.py numpy/core/ -rw-rw-rw- 1 user group 961 May 16 11:15 __init__.py -rw-rw-rw- 1 user group8938 Apr 28 21:16 _internal.py -rw-rw-rw- 1 user group 16814 May 16 11:31 arrayprint.py -rw-rw-rw- 1 user group 61754 Apr 28 21:16 fromnumeric.py (optional) -rw-rw-rw- 1 user group4635 Apr 28 21:16 info.py -rw-rw-rw- 1 user group 505132 Apr 28 21:16 multiarray.pyd (.so) -rw-rw-rw- 1 user group 56471 May 12 22:12 numeric.py -rw-rw-rw- 1 user group 20774 May 11 23:48 numerictypes.py -rw-rw-rw- 1 user group 318670 Apr 28 21:16 umath.pyd (.so) (about 300kB compressed on win32) * change numpy/__init__.py to simply: == numpy/__init__.py """ a minimal numpy - only essential stuff of the core """ #$1 import core from core import * == end of numpy/__init__.py = * run the patch below in numpy/core/ * (the strip-off regarding 'fromnumeric' is optional) * from there re-add just the stuff you need == patch in numpy/core/ === diff -ur -x *.pyc -x *.pyo ..\..\numpyxx\core\__init__.py .\__init__.py --- ..\..\numpyxx\core\__init__.py Tue Apr 28 21:16:32 2009 +++ .\__init__.py Sat May 16 11:15:35 2009 @@ -1,36 +1,34 @@ - +# minimal numpy! # pyXpy transposer to original version: #$1 from info import __doc__ from numpy.version import version as __version__ import multiarray +from multiarray import * #$1 import umath import _internal # for freeze programs import numerictypes as nt multiarray.set_typeDict(nt.sctypeDict) -import _sort +#$1 import _sort +import numeric #$1 from numeric import * -from fromnumeric import * -from defmatrix import * -import defchararray as char -import records as rec -from records import * -from memmap import * -from defchararray import * -import scalarmath -del nt - -from fromnumeric import amax as max, amin as min, \ - round_ as round -from numeric import absolute as abs +#import fromnumeric #$1 +#$1 from fromnumeric import * +#$1 from defmatrix import * +#$1 import defchararray as char +#$1 import records as rec +#$1 from records import * +#$1 from memmap import * +#$1 from defchararray import * +#$1 import scalarmath +#$1 del nt + +#$1 from fromnumeric import amax as max, amin as min, \ +#$1 round_ as round +#$1 from numeric import absolute as abs -__all__ = ['char','rec','memmap'] +__all__ = [] #$1 ['char','rec','memmap'] __all__ += numeric.__all__ -__all__ += fromnumeric.__all__ -__all__ += defmatrix.__all__ -__all__ += rec.__all__ -__all__ += char.__all__ - - -from numpy.testing import Tester -test = Tester().test -bench = Tester().bench +#$1 __all__ += fromnum
[Numpy-discussion] interleaving arrays
whats the right way to efficiently weave arrays like this ? : >>> n array([1, 2, 3, 4]) >>> m array([11, 22, 33, 44]) >>> o array([111, 222, 333, 444]) => [ 1, 11, 111, 2, 22, 222, 3, 33, 333, 4, 44, 444] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Interleaved Arrays and
Ian Mallett wrote:
>
> n = #blah
> testlist = []
> for x in xrange(n):
> for y in xrange(n):
> testlist.append([x,y])
> testlist.append([x+1,y])
>
> If "testlist" is an array (i.e., I could go: "array(testlist)"), it
> works nicely. However, my Python method is certainly improveable with
> numpy. I suspect the best way is interleaving the arrays [x,y->yn] and
> [x+1,y->yn] n times, but I couldn't figure out how to do that...
>
e.g with column_stack
>>> n = 10
>>> xx = np.ones(n)
>>> yy = np.arange(n)
>>> aa = np.column_stack((xx,yy))
>>> bb = np.column_stack((xx+1,yy))
>>> aa
array([[ 1., 0.],
[ 1., 1.],
[ 1., 2.],
[ 1., 3.],
[ 1., 4.],
[ 1., 5.],
[ 1., 6.],
[ 1., 7.],
[ 1., 8.],
[ 1., 9.]])
>>> bb
array([[ 2., 0.],
[ 2., 1.],
[ 2., 2.],
[ 2., 3.],
[ 2., 4.],
[ 2., 5.],
[ 2., 6.],
[ 2., 7.],
[ 2., 8.],
[ 2., 9.]])
>>> np.column_stack((aa,bb))
array([[ 1., 0., 2., 0.],
[ 1., 1., 2., 1.],
[ 1., 2., 2., 2.],
[ 1., 3., 2., 3.],
[ 1., 4., 2., 4.],
[ 1., 5., 2., 5.],
[ 1., 6., 2., 6.],
[ 1., 7., 2., 7.],
[ 1., 8., 2., 8.],
[ 1., 9., 2., 9.]])
>>> cc = _
>>> cc.reshape((n*2,2))
array([[ 1., 0.],
[ 2., 0.],
[ 1., 1.],
[ 2., 1.],
[ 1., 2.],
[ 2., 2.],
[ 1., 3.],
[ 2., 3.],
[ 1., 4.],
[ 2., 4.],
[ 1., 5.],
[ 2., 5.],
[ 1., 6.],
[ 2., 6.],
[ 1., 7.],
[ 2., 7.],
[ 1., 8.],
[ 2., 8.],
[ 1., 9.],
[ 2., 9.]])
>>>
However I feel too, there is a intuitive abbrev function like
'interleave' or so missing in numpy shape_base or so.
Robert
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Interleaved Arrays and
Neil Martinsen-Burrell wrote: > On 06/16/2009 02:18 PM, Robert wrote: >> >>> n = 10 >> >>> xx = np.ones(n) >> >>> yy = np.arange(n) >> >>> aa = np.column_stack((xx,yy)) >> >>> bb = np.column_stack((xx+1,yy)) >> >>> aa >> array([[ 1., 0.], >> [ 1., 1.], >> [ 1., 2.], >> [ 1., 3.], >> [ 1., 4.], >> [ 1., 5.], >> [ 1., 6.], >> [ 1., 7.], >> [ 1., 8.], >> [ 1., 9.]]) >> >>> bb >> array([[ 2., 0.], >> [ 2., 1.], >> [ 2., 2.], >> [ 2., 3.], >> [ 2., 4.], >> [ 2., 5.], >> [ 2., 6.], >> [ 2., 7.], >> [ 2., 8.], >> [ 2., 9.]]) >> >>> np.column_stack((aa,bb)) >> array([[ 1., 0., 2., 0.], >> [ 1., 1., 2., 1.], >> [ 1., 2., 2., 2.], >> [ 1., 3., 2., 3.], >> [ 1., 4., 2., 4.], >> [ 1., 5., 2., 5.], >> [ 1., 6., 2., 6.], >> [ 1., 7., 2., 7.], >> [ 1., 8., 2., 8.], >> [ 1., 9., 2., 9.]]) >> >>> cc = _ >> >>> cc.reshape((n*2,2)) >> array([[ 1., 0.], >> [ 2., 0.], >> [ 1., 1.], >> [ 2., 1.], >> [ 1., 2.], >> [ 2., 2.], >> [ 1., 3.], >> [ 2., 3.], >> [ 1., 4.], >> [ 2., 4.], >> [ 1., 5.], >> [ 2., 5.], >> [ 1., 6.], >> [ 2., 6.], >> [ 1., 7.], >> [ 2., 7.], >> [ 1., 8.], >> [ 2., 8.], >> [ 1., 9.], >> [ 2., 9.]]) >> >>> >> >> >> However I feel too, there is a intuitive abbrev function like >> 'interleave' or so missing in numpy shape_base or so. > > Using fancy indexing, you can set strided portions of an array equal to > another array. So:: > > In [2]: aa = np.empty((10,2)) > > In [3]: aa[:, 0] = 1 > > In [4]: aa[:,1] = np.arange(10) > > In [5]: bb = np.empty((10,2)) > > In [6]: bb[:,0] = 2 > > In [7]: bb[:,1] = aa[:,1] # this works > > In [8]: cc = np.empty((20,2)) > > In [9]: cc[::2,:] = aa > > In [10]: cc[1::2,:] = bb > > In [11]: cc > Out[11]: > array([[ 1., 0.], > [ 2., 0.], > [ 1., 1.], > [ 2., 1.], > [ 1., 2.], > [ 2., 2.], > [ 1., 3.], > [ 2., 3.], > [ 1., 4.], > [ 2., 4.], > [ 1., 5.], > [ 2., 5.], > [ 1., 6.], > [ 2., 6.], > [ 1., 7.], > [ 2., 7.], > [ 1., 8.], > [ 2., 8.], > [ 1., 9.], > [ 2., 9.]]) > > Using this syntax, interleave could be a one-liner. > > -Neil that method of 'filling an empty with a pattern' was mentioned in the other (general) interleaving question. It requires however a lot of particular numbers and :'s in the code, and requires even more statements which can hardly be written in functional style - in one line?. The other approach is more jount, free of fancy indexing assignments. The general interleaving should work efficiently in one like this: np.column_stack/concatenate((r,g,b,), axis=...).reshape(..) But as all this is not intuitive, something like this should be in numpy perhaps? : def interleave( tup_arrays, axis = None ) Robert ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Advanced indexing advice?
well its not really slow. yet with np.where it seems to be 2x faster for large arrays : a[1:4,1:4] = np.where(mask,b,a[1:4,1:4]) otherwise consider Cython: http://docs.cython.org/docs/numpy_tutorial.html#tuning-indexing-further Robert Cristi Constantin wrote: > > Thank you so much for your prompt answer, Stéfan. > It's a very very interesting method. I will keep it for future. :) > > But, i tested it with a few examples and the speed of execution is just > a tiny bit slower than what i told you i was using. So it's not faster, > it's about the same speed. > > Thank you again. I will play with your method a little more. > > --- On *Thu, 6/18/09, Stéfan van der Walt //* wrote: > > From: Stéfan van der Walt > Subject: Re: [Numpy-discussion] Advanced indexing advice? > To: "Discussion of Numerical Python" > Date: Thursday, June 18, 2009, 2:16 AM > > Hi Cristi > > 2009/6/18 Cristi Constantin >: > > I have a question about advanced indexing. > > > > I have 2 matrices : > > > >>>> > > a=array([[ 0, 1, 2, 3, 4, 5], > > [ 6, 7, 8, 9, 10, 11], > > [12, 13, 14, 15, 16, 17], > > [18, 19, 20, 21, 22, 23]]) > > > > b=array([[1, 0, 1], > > [0, 2, 0], > > [0, 0, 3]]) > >>>> > > > > I want to put all NON-zero elements from array B into array A, > but use > > offset! > > Here's a solution using views: > > offset = np.array([1,1]) > slices = [slice(*x) for x in zip(offset, offset + b.shape)] > c = a[slices] > mask = (b != 0) > c[mask] = b[mask] > > Regards > Stéfan > > > > > > ___ > Numpy-discussion mailing list > Numpy-discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Advanced indexing advice?
or if your mask is thin and constant, then consider np.put ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Plans for Numpy 1.4.0 and scipy 0.8.0
Lou Pecora wrote:
> I'm still using 2.4, but I plan to go to 2.5 when the project we're
> doing now reaches a stable point later this year. Not sure after that.
> I know it's real work to keep several versions going, but I sense there
> are a lot of people in the 2.4 - 2.5 window. I guess 2.6 is a mini step
> toward 3.0. The problem with each step is that all the libraries we
> rely on have to be ugraded to that step or we might lose the
> functionality of that library. For me that's a killer. I have to take a
> good look at all of them before the upgrade or a big project will take a
> fatal hit.
>
+1
I'd like even support for Python 2.3. Many basic libraries still
support 2.3 . Recently I wanted to download the latest numpy for
2.3 which I need for some projects and :-( . Just since
2008-08-01 they dropped both 2.3 and 2.4. Is there a serious reason?
And numpy is a very basic library. And what is in numpy (scipy)
that requires advanced language syntax? Its just about numbers and
slices. A few ifdef's for new concepts like new base classes. It
needs nothing of the real news of advanced Pythons. A thing like
numpy/scipy is most simple regarding code structure. It should be
easy to offer it for old Pythons - at least 2.3 which is the
inofficial "Python 2" baseline for library producers.
Even a complex GUI app like Pythonwin (where it is very tempting
to use advanced sugar) is still supported for even 2.2
Regarding Python 2 -> 3 Migration. Look at e.g. Cython - it
produces C code with a few #ifdef's and macros and which compiles
both in Py2 (2.3+) and Py3. Its quite simple to maintain. Also
Python code can be written so, that it can be auto-transposed from
2 -> 3 for long time: by 2to3 + pyXpy comment transposition
language like
print "x" #$3 print("x")
#$3 only_py3_func()
only_py2_func() #$3
It would be drastic to force the basis for numpy to Py3 so early -
unless a back direction is offered. And numpy should/could in my
opinion be one of the last libraries which cuts off support for
old Pythons.
-
One of the itching problems when using numpy with smaller apps,
scripts, web and with freezers is, that import is very slow, and
it rams almost the whole numpy into the memory. Many fat dlls like
_dotblas, etc. And there are strange (unnecessary) dependencies
between the branches and files. See thread "minimal numpy?".
That all threatens usabilty of numpy in a many areas. Its designed
like "loading numpy/scipe in the lab in the morning and exiting in
the evening". That is too sloopy for a basic library which adds a
efficient key array class to Python. numpy should be usable in a
much lighter way.
If "import numpy" shall by default still import most of numpy -
ok, but there should perhaps be at least an easy mechanism to stop
this "all-in-one" behavior and dependencies with an easy switch.
Or with "import numpy.base"
Or in my opinion the reverse behavior would be more decent for a
library:
"import numpy" imports only the minimum (like in that other
thread) and "import numpy.anth[ology] as np" or so may draw the
whole mess as before.
inter-DLL-imports like multiarray.pyd <-> umath.pyd should use
relative imports (like the py modules next to them).
Such cleanup of the code organization and improving usability etc
I think is more important than playing a role as "leading py3
enforcer"
Robert
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ndarray from column data
Elaine Angelino wrote: > Hi there -- > > Is there a fast way to make a numpy ndarray from column data? > > For example, suppose I want to make an ndarray with 2 rows and 3 columns > of different data types based on the following column data: > > C0 = [1,2] > C1 = ['a','b'] > C2 = [3.3,4.4] > > I could create an empty ndarray and fill the columns one by one: > > X = numpy.core.ndarray((2,), dtype=' X['f0'] = C0 > X['f1'] = C1 > X['f2'] = C2 > > The result is the same as: > > X = numpy.array([(1,'a',3.3), (2,'b',4.4)], dtype=' > but I would like to make X directly from the column data. > > [ I know that numpy.core.records.fromarrays will produce a numpy > recarray from column data, but this of course is a recarray and not a > ndarray! For ex: > > X = numpy.numpy.core.records.fromarrays([C0,C1,C2]) ] > > Thanks for any help, > > Elaine > >>> np.array(zip(C0,C1,C2), dtype=' 2D there is >>> np.column_stack((C0,C1,C2)) array([['1', 'a', '3.3'], ['2', 'b', '4.4']], dtype='|S8') Robert ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] memory address of array data?
Is there a function to get the memory address (int) of (contigious) ndarray data on Python level - like array.array.buffer_info() ? I'd need it to pass it to a camera function. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Why is the truth value of ndarray not simply size>0 ?
Is there a reason why ndarray truth tests (except scalars)
deviates from the convention of other Python iterables
list,array.array,str,dict,... ?
Furthermore there is a surprising strange exception for arrays
with size 1 (!= scalars).
I often run into exceptions and unexpected bahavior like shown
below, when developing or transcribing from other types to numpy.
Robert
---
>>> a=np.array([12,4,5])
>>> if a: print 2
...
Traceback (most recent call last):
File "", line 1, in
ValueError: The truth value of an array with more than one element
is ambiguous. Use a.any() or a.all()
>>> a=np.array([12])
>>> if a: print 2
...
2
>>> a=np.array([0])
>>> if a: print 2
...
>>> a=[0]
>>> if a: print 2
...
2
>>> a=[]
>>> if a: print 2
...
>>> import array
>>> a=array.array('i',[12,1,3])
>>> if a: print 2
...
2
>>> a=array.array('i',[0])
>>> if a: print 2
...
2
>>> a=array.array('i',[])
>>> if a: print 2
...
>>> bool(np.array(0))
False
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Why is the truth value of ndarray not simply size>0 ?
Neil Martinsen-Burrell wrote: > On 2009-09-07 07:11 , Robert wrote: >> Is there a reason why ndarray truth tests (except scalars) >> deviates from the convention of other Python iterables >> list,array.array,str,dict,... ? >> >> Furthermore there is a surprising strange exception for arrays >> with size 1 (!= scalars). > > Historically, numpy's predecessors used "not equal to zero" as the > meaning for truth (consistent with numerical types in Python). However, > this introduces an ambiguity as both any(a != 0) and all(a != 0) are > reasonable interpretations of the truth value of a sequence of numbers. well, I can familiarize with that "not equal to zero" philosophy for a math-centric array type (different from a container / size>0 philosophy) However I don't see that all(a) (or "all(a != 0)") is something which anybody would ever expect with .__nonzero__() / if a: ... . Does anybody? And the current behavior with all those strange exceptions and exceptions from exceptions still seems awkward and unnecessary. The any() interpretion is outstandingly "right" in my opinion, and doesn't need to be guessed: anything/any part non-zero disturbs the clean "zeroness". Zero must be wholly pure zero. This is so everywhere in math and informatics. a number/memory is zero when all bits/bytes are zero. a matrix is a zero matrix when all elements are zero... This way only the test is also seamlessly consistent with a zero length array (while all(zerolengtharray != 0) would be True surprisingly!) This kind of any(a) truth test (only) is also often needed, and it would be also fast executable this way. It would be compatible with None/False init/default variable tests during code evolution in Python style and would behave well everywhere as far as I can see. It would also not break old code. Would a feature request in that direction have any chance? Robert > Numpy refuses to guess and raises the exception shown below. For > sequences with a single item, there is no ambiguity and numpy does the > (numerically) ordinary thing. > > The ndarray type available in Numpy is not conceptually an extension of > Python's iterables. If you'd like to help other Numpy users with this > issue, you can edit the documentation in the online documentation editor > at http://docs.scipy.org/numpy/docs/numpy-docs/user/index.rst > > -Neil ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Why is the truth value of ndarray not simply size>0 ?
Robert wrote: > Neil Martinsen-Burrell wrote: >> On 2009-09-07 07:11 , Robert wrote: >>> Is there a reason why ndarray truth tests (except scalars) >>> deviates from the convention of other Python iterables >>> list,array.array,str,dict,... ? >>> >>> Furthermore there is a surprising strange exception for arrays >>> with size 1 (!= scalars). >> Historically, numpy's predecessors used "not equal to zero" as the >> meaning for truth (consistent with numerical types in Python). However, >> this introduces an ambiguity as both any(a != 0) and all(a != 0) are >> reasonable interpretations of the truth value of a sequence of numbers. > > > well, I can familiarize with that "not equal to zero" philosophy > for a math-centric array type (different from a container / size>0 > philosophy) > > However I don't see that all(a) (or "all(a != 0)") is something > which anybody would ever expect with .__nonzero__() / if a: ... . > Does anybody? And the current behavior with all those strange > exceptions and exceptions from exceptions still seems awkward and > unnecessary. > > The any() interpretion is outstandingly "right" in my opinion, and > doesn't need to be guessed: anything/any part non-zero disturbs > the clean "zeroness". Zero must be wholly pure zero. This is so > everywhere in math and informatics. a number/memory is zero when > all bits/bytes are zero. a matrix is a zero matrix when all > elements are zero... This way only the test is also seamlessly > consistent with a zero length array (while all(zerolengtharray != > 0) would be True surprisingly!) > This kind of any(a) truth test (only) is also often needed, and it > would be also fast executable this way. It would be compatible > with None/False init/default variable tests during code evolution > in Python style and would behave well everywhere as far as I can > see. It would also not break old code. > Would a feature request in that direction have any chance? > > Robert > > >> Numpy refuses to guess and raises the exception shown below. For >> sequences with a single item, there is no ambiguity and numpy does the >> (numerically) ordinary thing. >> coming to mind another way to see it: I'm not aware of any other python type which doesn't definitely know if it is __nonzero__ or not (unless there is an IOError or so). And everywhere: if there is *any* logical doubt at all, the default is: True! - not an Exception. For example .__nonzero__/.__bool__ for a custom class defaults to True. A behavior where an object throws an exception upon __nonzero__ test just because of principal doubts seems not to fit into the Python world. The non-zero test must definitely go through. Only 2 ways seem to be consistently Pythonic and logical: "size > 0"; or "any(a)" (*); and the later option may be more 'numerical'. Robert * .__nonzero__() and perhaps .any() too should not fail upon flexible types like currently: >>> np.array(["","",""]).any() Traceback (most recent call last): File "", line 1, in TypeError: cannot perform reduce with flexible type >>> ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] scipy Windows binary compatible recent numpy 1.0 ?
Since some months now the latest scipy (0.5.1) binary (for Windows) is not compatible with current numpy (1.0) - obviously because of a C-library mismatch. Does somebody have a compatible binary - or when is the next scipy (binary) planned? Robert ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy.random.shuffle
is this an error when shuffle doubles&looses items on a 2-d array? : >>> r=arange(20) >>> rr=ziparrays(r,r) #numpy.vstack(*args).transpose() >>> rr array([[ 0, 0], [ 1, 1], [ 2, 2], [ 3, 3], [ 4, 4], [ 5, 5], [ 6, 6], [ 7, 7], [ 8, 8], [ 9, 9], [10, 10], [11, 11], [12, 12], [13, 13], [14, 14], [15, 15], [16, 16], [17, 17], [18, 18], [19, 19]]) >>> numpy.random.shuffle(rr) >>> rr array([[ 0, 0], [ 0, 0], [ 2, 2], [ 2, 2], [ 3, 3], [ 3, 3], [ 5, 5], [ 2, 2], [ 0, 0], [ 8, 8], [10, 10], [ 1, 1], [10, 10], [11, 11], [12, 12], [ 2, 2], [10, 10], [15, 15], [ 6, 6], [16, 16]]) >>> on 1-d arrays it seems to be consistent as expected: >>> numpy.random.shuffle(r) >>> r array([ 6, 1, 3, 17, 0, 11, 15, 7, 14, 10, 9, 2, 8, 12, 5, 4, 19, 16, 13, 18]) >>> Robert ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] approx. replacement for scipy.special.betainc ?
for the computation of a p_value I need the betainc function, which draws the big _cephes module into a (numpy-only) app - the only reason for this import. I need it just for the computation of a p-value like: p_value = scipy.special.betainc(0.5*nf, 0.5, nf/(nf+t_value*t_value)) Is there a simple (python-only) replacement func for betainc ? Robert ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy.random.shuffle
Robert Kern wrote: > Tim Hochberg wrote: >> Robert Kern wrote: > >>> One possibility is to check if the object is an ndarray (or subclass) and >>> use >>> .copy() if so; otherwise, use the current implementation and hope that you >>> didn't pass it a Numeric or numarray array (or some other view-based >>> object). >>> >> I think I would invert this test and instead check if the object is a >> Python list and *not* copy in that case. Otherwise, use copy.copy to >> copy the object whatever it is. This looks like it would be more robust >> in that it would work in all sensible case, and just be a tad slower in >> some of them. > > I don't want to assume that the only two sequence types are lists and arrays. > The problem with using copy.copy() on non-arrays is that it, well, makes > copies > of the elements. The objects in the shuffled sequence are not the same objects > before and after the shuffling. I consider that to be a violation of the spec. > > Views are rare outside of numpy/Numeric/numarray, partially because Guido > considers them to be evil. I'm beginning to see why. > >> Another possible refinement / complication would be to special case 1D >> arrays so that they run fastish. >> >> A third possibility involves rewriting this in this form: >> >> indices = arange(len(x)) >> _shuffle_core(indices) # This just does what current shuffle now does >> x[:] = take(x, indices, 0) > > That's problematic since the elements all turn into numpy scalar objects: > > In [1]: from numpy import * > > In [2]: a = range(9,-1,-1) > > In [3]: idx = arange(len(a)) > > In [4]: a[:] = take(a, idx, 0) > > In [5]: a > Out[5]: [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] > > In [6]: type(a[0]) > Out[6]: > a[:]=take(asarray(a,object),idx,0) ? works also correct with ndarray's even if I didn't dig the reason why... all element will be probably re-casted twice. Think the take-method on shuffled indizes is basically right and natural for a numpy-shuffler. The example is just possibly another vote against the default behavior of letting numpy.scalar types out of arrays, which are set up with a "harmless" type. >>> array([1,2,3],float) array([ 1., 2., 3.]) >>> type(_[0]) >>> is just ill as it I think. In (Guido's) Python objects should probably come out of collections best as typy as they went in. Currently numpy-scalars will just "infect" the whole app almost like a virus (and kill performance and pickle's etc.) Of course views are essential for an efficient array type, but type-altering possibly not. For rare cases for generalized algs (I need to think hard to find even an example), where the array-interface is needed on elements (and a array(obj) cast is too uncomfortable), there could be still the different possibilty: >>> array([1,2,3],numpy.float64) then its natural that numpy.float64, numpy.int32 come out, as the programmer would even expect it so. Thus maybe for array types: * float!=numpy.float64 (but common base class (or 'float' itself) maybe) * int !=numpy.intXX * complex !=numpy.complex128 * default array type is (python.)float * default array type from list of ints is (python.)int * default array type from list of complex is (python.)complex * default array type of other lists is always currently this is also problematic: >>> array([1,2,"3",[]]) array(['1', '2', '3', '[]'], dtype='|S4') and even >>> array([1,2,"3ef",'wefwfewoiwjefo iwjef']) array(['1', '2', '3ef', 'wefwfewoiwjefo iwjef'], dtype='|S20') >>> _[0]='woeifjwo woie pwioef wliuefh lwieufh wleifuh welfiu ' >>> _ array(['woeifjwo woie pwioef', '2', '3ef', 'wefwfewoiwjefo iwjef'], dtype='|S20') is rarely what a Pythoneer would expect. Guess fix string arrays should only be created explicitely Robert ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Wanted: Numeric/NumPy compatible array creation expression
[EMAIL PROTECTED] wrote: > I am looking for a way to create an array of the same type as another > given array (but of different shape) that works both with Numeric and > NumPy without being unreasonably slow. In other words, I am looking > for a a replacement for the expression > > array2 = Numeric.zeros(shape, array1.typecode()) > > that will *also* (not *only*) work under NumPy (where typecode() > became dtype.char). An obvious idea is reshaping array1 and > multiplying by 0., but that can become quite costly. timeit if *0 is really costly ... you'll not get it really cheaper I think. As you write just "same type": .copy().reshape() or so maybe ok too, in case you'll fill in stuff into this array anyway > Any ideas? > > Konrad. > -- > - > Konrad Hinsen > Centre de Biophysique Moléculaire, CNRS Orléans > Synchrotron Soleil - Division Expériences > Saint Aubin - BP 48 > 91192 Gif sur Yvette Cedex, France > Tel. +33-1 69 35 97 15 > E-Mail: [EMAIL PROTECTED] > - ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Wanted: Numeric/NumPy compatible array creation expression
[EMAIL PROTECTED] wrote:
> On 23.11.2006, at 16:29, Robert wrote:
>
>>> that will *also* (not *only*) work under NumPy (where typecode()
>>> became dtype.char). An obvious idea is reshaping array1 and
>>> multiplying by 0., but that can become quite costly.
>> timeit if *0 is really costly ... you'll not get it really cheaper
>> I think.
>
> What I expect to be costly is the reshape (resize in most
> situations), not the multiplication. I didn't do any timings though...
>
its simple with:
a=...
import timeit
timeit.Timer("b=a.reshape",glbls=globals()).timeit(1000)
reshaping is usually very cheap (numpy), as only an new view with other index
geometry is created.
A (unavoidable?) copy is probably the most expensive but just test ...
Robert
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] can int and float exists in one array?(about difference in indexing Matlab matrix and Numpy array)
Zhang Sam wrote:
> Hi, there
>
> I have a practical problem. For some reason, I hope int and float can
> exist one array as shown in matlab code.
You can use 'object' as array type:
>>> a=numpy.array([1,2.0,2])
>>> b=numpy.array([1,2.0,2],object)
>>> a
array([ 1., 2., 2.])
>>> b
array([1, 2.0, 2], dtype=object)
>>>
strangely this often leads to even faster code compared to using float/int
arrays. Especially when you do much computations in Python - because with numpy
you cannot really create int/float arrays, but only numpy.int32/numpy.float64
arrays. Even if you write "numpy.array([1,2.0,2],float)", you'll get item
instances of another type out, which compute a lot slower and cause other
problems. So currently the only way to put int's & float's into an array anyway
requires to use type 'object'.
>>> import timeit
>>> timeit.Timer('a[0]+a[1]',glbls=globals()).timeit(1)
0.069539285020861596
>>> timeit.Timer('a[0]+a[1]',glbls=globals()).timeit(1)
0.037078734867140639
>>> timeit.Timer('a[0]+a[1]',glbls=globals()).timeit(1)
0.038550709657232396
>>> timeit.Timer('b[0]+b[1]',glbls=globals()).timeit(1)
0.013452827105121301
>>> timeit.Timer('b[0]+b[1]',glbls=globals()).timeit(10000)
0.01344221123075684
>>> b.sum()
5.0
>>> timeit.Timer('a.sum()',glbls=globals()).timeit(1)
0.18172588974296389
>>> timeit.Timer('b.sum()',glbls=globals()).timeit(1)
0.15798208990248241
>>>
Robert
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy: azip ?
I often need a function to zip 2 (1D) arrays together to a 2D array - similar as python's zip() does. Found no function in numpy to do that directly without thinking a lot. Or is there one? Otherwise such thing would be helpful in future numpy. I have this in my tools: def azip( *ll ): return numpy.vstack(ll).transpose() Robert ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Future Python 2.3 support ? - Re: Numpy and Python 2.2 on RHEL3
Robert Kern wrote: > David Bogen wrote: >> All: >> >> Is it possible to build Numpy using Python 2.2? I haven't been able to >> find anything that explicitly lists the versions of Python with which >> Numpy functions so I've been working under the assumption that the two >> bits will mesh together somehow. > > numpy requires Python 2.3 . > hope Python2.3 support will not be dropped too early. There is not much cost overall when going from 2.2 to 2.3. Yet from Py2.3 to Py2.4 there is a tremendous increase in memory footprint, in distributable file sizes, load time, cgi start time, compiler troubles on Win etc. .. - not really balanced by comparable improvements. For most types of practical applications Py2.3 is still the "good Python" for me. Robert ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
On Sat, Feb 18, 2012 at 04:54, Charles R Harris wrote: > I found this , which references 0mq (used by ipython) as an example of a C++ > library with a C interface. It seems enums can have different sizes in > C/C++, so that is something to watch. One of the ways they manage to do this is by scrupulously avoiding exceptions even in the internal, never-touches-C zone. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
>> >> >> > It's not that the support WAS buggy, it's that it wasn't clear to >>> >> >> > me >>> >> >> > what was going on and where my performance bottleneck was. Even >>> >> >> > after >>> >> >> > microbenchmarking with ipython, using timeit and prun, and using >>> >> >> > the >>> >> >> > cython code visualization tool. Ultimately I don't think it was >>> >> >> > cython, so perhaps my comment was a bit unfair. But it was >>> >> >> > unfortunately difficult to verify that. Of course, as you say, >>> >> >> > diagnosing and solving such issues would become easier to resolve >>> >> >> > with >>> >> >> > more cython experience. >>> >> >> > >>> >> >> >>> The C files generated by cython were >>> >> >> >>> enormous and difficult to read. They really weren't meant for >>> >> >> >>> human >>> >> >> >>> consumption. >>> >> >> >> >>> >> >> >> Yes, it takes some practice to get used to what Cython will do, >>> >> >> >> and >>> >> >> >> how to optimize the output. >>> >> >> >> >>> >> >> >>> As Sturla has said, regardless of the quality of the >>> >> >> >>> current product, it isn't stable. >>> >> >> >> >>> >> >> >> I've personally found it more or less rock solid. Could you say >>> >> >> >> what >>> >> >> >> you mean by "it isn't stable"? >>> >> >> >> >>> >> >> > >>> >> >> > I just meant what Sturla said, nothing more: >>> >> >> > >>> >> >> > "Cython is still 0.16, it is still unfinished. We cannot base >>> >> >> > NumPy >>> >> >> > on >>> >> >> > an unfinished compiler." >>> >> >> >>> >> >> Y'all mean, it has a zero at the beginning of the version number and >>> >> >> it is still adding new features? Yes, that is correct, but it seems >>> >> >> more reasonable to me to phrase that as 'active development' rather >>> >> >> than 'unstable', because they take considerable care to be backwards >>> >> >> compatible, have a large automated Cython test suite, and a major >>> >> >> stress-tester in the Sage test suite. >>> >> >> >>> >> > >>> >> > Matthew, >>> >> > >>> >> > No one in their right mind would build a large performance library >>> >> > using >>> >> > Cython, it just isn't the right tool. For what it was designed for - >>> >> > wrapping existing c code or writing small and simple things close to >>> >> > Python >>> >> > - it does very well, but it was never designed for making core C/C++ >>> >> > libraries and in that role it just gets in the way. >>> >> >>> >> I believe the proposal is to refactor the lowest levels in pure C and >>> >> move the some or most of the library superstructure to Cython. >>> > >>> > >>> > Go for it. >>> >>> The proposal of moving to a core C + cython has been discussed by >>> multiple contributors. It is certainly a valid proposal. *I* have >>> worked on this (npymath, separate compilation), although certainly not >>> as much as I would have wanted to. I think much can be done in that >>> vein. Using the "shut up if you don't do it" is a straw man (and >>> uncalled for). >> >> >> OK, I was annoyed. > > By what? Your misunderstanding of what was being discussed. The proposal being discussed is implementing the core of numpy in C++, wrapped in C to be usable as a C library that other extensions can use, and then exposed to Python in an unspecified way. Cython was raised as an alternative for this core, but as Chuck points out, it doesn't really fit. Your assertion that what was being discussed was putting the core in C and using Cython to wrap it was simply a non-sequitur. Discussion of alternatives is fine. You weren't doing that. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
On Sat, Feb 18, 2012 at 22:06, Matthew Brett wrote: > Hi, > > On Sat, Feb 18, 2012 at 2:03 PM, Robert Kern wrote: >> On Sat, Feb 18, 2012 at 21:51, Matthew Brett wrote: >>> On Sat, Feb 18, 2012 at 1:40 PM, Charles R Harris >>> wrote: >>>> >>>> >>>> On Sat, Feb 18, 2012 at 2:17 PM, David Cournapeau >>>> wrote: >>>>> >>>>> On Sat, Feb 18, 2012 at 8:45 PM, Charles R Harris >>>>> wrote: >>>>> > >>>>> > >>>>> > On Sat, Feb 18, 2012 at 1:39 PM, Matthew Brett >>>>> > wrote: >>>>> >> >>>>> >> Hi, >>>>> >> >>>>> >> On Sat, Feb 18, 2012 at 12:35 PM, Charles R Harris >>>>> >> wrote: >>>>> >> > >>>>> >> > >>>>> >> > On Sat, Feb 18, 2012 at 12:21 PM, Matthew Brett >>>>> >> > >>>>> >> > wrote: >>>>> >> >> >>>>> >> >> Hi. >>>>> >> >> >>>>> >> >> On Sat, Feb 18, 2012 at 12:18 AM, Christopher Jordan-Squire >>>>> >> >> wrote: >>>>> >> >> > On Fri, Feb 17, 2012 at 11:31 PM, Matthew Brett >>>>> >> >> > wrote: >>>>> >> >> >> Hi, >>>>> >> >> >> >>>>> >> >> >> On Fri, Feb 17, 2012 at 10:18 PM, Christopher Jordan-Squire >>>>> >> >> >> wrote: >>>>> >> >> >>> On Fri, Feb 17, 2012 at 8:30 PM, Sturla Molden >>>>> >> >> >>> >>>>> >> >> >>> wrote: >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> Den 18. feb. 2012 kl. 05:01 skrev Jason Grout >>>>> >> >> >>>> : >>>>> >> >> >>>> >>>>> >> >> >>>>> On 2/17/12 9:54 PM, Sturla Molden wrote: >>>>> >> >> >>>>>> We would have to write a C++ programming tutorial that is >>>>> >> >> >>>>>> based >>>>> >> >> >>>>>> on >>>>> >> >> >>>>>> Pyton knowledge instead of C knowledge. >>>>> >> >> >>>>> >>>>> >> >> >>>>> I personally would love such a thing. It's been a while since >>>>> >> >> >>>>> I >>>>> >> >> >>>>> did >>>>> >> >> >>>>> anything nontrivial on my own in C++. >>>>> >> >> >>>>> >>>>> >> >> >>>> >>>>> >> >> >>>> One example: How do we code multiple return values? >>>>> >> >> >>>> >>>>> >> >> >>>> In Python: >>>>> >> >> >>>> - Return a tuple. >>>>> >> >> >>>> >>>>> >> >> >>>> In C: >>>>> >> >> >>>> - Use pointers (evilness) >>>>> >> >> >>>> >>>>> >> >> >>>> In C++: >>>>> >> >> >>>> - Return a std::tuple, as you would in Python. >>>>> >> >> >>>> - Use references, as you would in Fortran or Pascal. >>>>> >> >> >>>> - Use pointers, as you would in C. >>>>> >> >> >>>> >>>>> >> >> >>>> C++ textbooks always pick the last... >>>>> >> >> >>>> >>>>> >> >> >>>> I would show the first and the second method, and perhaps >>>>> >> >> >>>> intentionally forget the last. >>>>> >> >> >>>> >>>>> >> >> >>>> Sturla >>>>> >> >> >>>> >>>>> >> >> >> >>>>> >> >> >>> On the flip side, cython looked pretty...but I did
Re: [Numpy-discussion] Proposed Roadmap Overview
On Sat, Feb 18, 2012 at 22:29, Matthew Brett wrote: > Hi, > > On Sat, Feb 18, 2012 at 2:20 PM, Robert Kern wrote: >> On Sat, Feb 18, 2012 at 22:06, Matthew Brett wrote: >>> Hi, >>> >>> On Sat, Feb 18, 2012 at 2:03 PM, Robert Kern wrote: >>>> Your misunderstanding of what was being discussed. The proposal being >>>> discussed is implementing the core of numpy in C++, wrapped in C to be >>>> usable as a C library that other extensions can use, and then exposed >>>> to Python in an unspecified way. Cython was raised as an alternative >>>> for this core, but as Chuck points out, it doesn't really fit. Your >>>> assertion that what was being discussed was putting the core in C and >>>> using Cython to wrap it was simply a non-sequitur. Discussion of >>>> alternatives is fine. You weren't doing that. >>> >>> You read David's email? Was he also being annoying? >> >> Not really, because he was responding on-topic to the bizarro-branch >> of the conversation that you spawned about the merits of moving from >> hand-written C extensions to a Cython-wrapped C library. Whatever >> annoyance his email might inspire is your fault, not his. The >> discussion was about whether to use C++ or Cython for the core. Chuck >> argued that Cython was not a suitable implementation language for the >> core. You responded that his objections to Cython didn't apply to what >> you thought was being discussed, using Cython to wrap a pure-C >> library. As Pauli (Wolfgang, not our Pauli) once phrased it, you were >> "not even wrong". It's hard to respond coherently to someone who is >> breaking the fundamental expectations of discourse. Even I had to >> stare at the thread for a few minutes to figure out where things went >> off the rails. > > I'm sorry but this seems to me to be aggressive, offensive, and unjust. > > The discussion was, from the beginning, mainly about the relative > benefits of rewriting the core with C / Cython, or C++. > > I don't think anyone was proposing writing every line of the numpy > core in Cython. Ergo (sorry to use the debating term), the proposal > to use Cython was always to take some of the higher level code out of > C and leave some of it in C. It does indeed make the debate > ridiculous to oppose a proposal that no-one has made. > > Now I am sure it is obvious to you, that the proposal to refactor the > current C code to into low-level C libraries, and higher level Cython > wrappers, is absurd and off the table. It isn't obvious to me. I > don't think I broke a fundamental rule of polite discourse to clarify > that is what I meant, It's not off the table, but it's not what this discussion was about. The proposal is to implement the core in C++. Regardless of whether the core is separated out as an independent non-Python library or not. Some people want to use higher level language features in the core. Cython was brought up as an alternative. If they were bringing up Cython in the context of C-core+Cython-wrapper, then they were also misunderstanding what the proposal was about. The discussion is about a C++-core versus a C-core (either the current one or a refactored one). If you want to argue for a C-core over a C++-core, that's great, but talking about Cython features and stability is not relevant to that discussion. It's an entirely orthogonal issue to what is motivating the request to use C++ in the core. C-core+Cython-wrapper is still a viable alternative, but the relevant bit of that is "C-core". I would wager that after any refactoring of the core, regardless of whether it is implemented in C++ or C, we would then wrap it in Cython. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
On Mon, Feb 20, 2012 at 19:55, Paul Anton Letnes wrote: > > On 20. feb. 2012, at 16:29, Sturla Molden wrote: >>> - in newer standards it has some nontrivial mathematical functions: gamma, >>> bessel, etc. that numpy lacks right now >> >> That belongs to SciPy. > > I don't see exactly why. Why should numpy have exponential but not gamma > functions? The division seems kinda arbitrary. Not that I am arguing > violently for bessel functions in numpy. The semi-arbitrary dividing line that we have settled on is C99. If a special function is in the C99 standard, we'll accept an implementation for it in numpy. Part (well, most) of the rationale is just to have a clear dividing line even if it's fairly arbitrary. The other part is that if a decidedly non-mathematically-focused standard like C99 includes a special function in its standard library, then odds are good that it's something that is widely used enough as a building block for other things. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers
On Thu, Feb 23, 2012 at 21:09, Gael Varoquaux wrote: > On Thu, Feb 23, 2012 at 04:07:04PM -0500, Wes McKinney wrote: >> In this last case for example, around 500 MB of RAM is taken up for an >> array that should only be about 80-90MB. If you're a data scientist >> working in Python, this is _not good_. > > But why, oh why, are people storing big data in CSV? Because everyone can read it. It's not so much "storage" as "transmission". -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np.longlong casts to int
On Feb 24, 2012, at 7:43 AM, Pierre Haessig wrote: > Hi, > Le 24/02/2012 01:00, Matthew Brett a écrit : >> Right - no proposal to change float64 because it's not ambiguous - it >> is both binary64 IEEE floating point format and 64 bit width. > All right ! Focusing the renaming only on those "extended precision" > float types makes sense. >> The confusion here is for float128 - which is very occasionally IEEE >> binary128 and can be at least two other things (PPC twin double, and >> Intel 80 bit padded to 128 bits). Some of us were also surprised to >> find float96 is the same precision as float128 (being an 80 bit Intel >> padded to 96 bits). >> >> The renaming is an attempt to make it less confusing. Do you agree >> the renaming is less confusing? Do you have another proposal? >> >> Preferring 'longdouble' is precisely to flag up to people that they >> may need to do some more research to find out what exactly that is. >> Which is correct :) > > The renaming scheme you mentionned (float80_96, float80_128, > float128_ieee, float_pair_128 ) is very informative, maybe too much ! > (In this list, I would shorten float128_ieee -> float128 though). > > So in the end, I may concur with you on "longdouble" as a good name for > "extended precision" in the Intel 80 bits sense. (Should "longfloat" be > deprecated ?). > float128 may be kept for ieee definition only, since it looks like the > natural extension of float64. Maybe one day it will be available on our > "standard" machines ? > > Also I just browsed Wikipedia's page [1] to get a bit of background and > I wonder what is the use case of these 80 bits numbers apart from what > is described as "keeping intermediate results" when performing > exponentiation on doubles ? In AIFF audio files, the sample rate is stored in the Common Chunk as an 80-bit "extended" floating-point number. The field allocated for this is exactly 80 bits wide (i.e., no padding to 96 or 128). The 1989 Apple document defining AIFF can be found at I once wrote my own "save as AIFF" routine and I remember it was a pain to format the 80-bit extended float. Bob Pyle > > Best, > Pierre > > [1] http://en.wikipedia.org/wiki/Extended_precision > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy uint16 array to unicode string
On Sat, Feb 25, 2012 at 19:50, Janwillem wrote:
> I have a buffer as a numpy array of uint16. Chunks of this buffer are
> unicode characters. How do I get these chunks, say data[start:end], in a
> string?
data[sart:end].tostring().decode('UTF-16LE')
or 'UTF-16BE' if they are big-endian.
> Other chunks are 32bit integers how do I get these chunks into a numpy
> array of int32?
data[start:end].view(np.int32)
--
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [NumPy-Tickets] [NumPy] #2067: when x is an N-D array, list(x) produces surprising results
Off-Trac comments should probably go to numpy-discussion rather than back to numpy-tickets. I'm not sure why it's not read-only, but it should be. On Tue, Feb 28, 2012 at 01:15, Phillip Feldman wrote: > I'd appreciate a pointer to something in the NumPy reference material > that states that a N-dim ndarray is treated as a collection of > (N-1)-dim ndarrays for purposes of iteration. I'm not sure anything states that about iteration specifically, but it's implicit from the documented behavior when given a single integer index: http://docs.scipy.org/doc/numpy/user/basics.indexing.html#single-element-indexing > Phillip > > On Mon, Feb 27, 2012 at 11:45 AM, wrote: >> #2067: when x is an N-D array, list(x) produces surprising results >> +--- >> Reporter: pfeldman | Owner: somebody >> Type: defect | Status: new >> Priority: normal | Milestone: Unscheduled >> Component: numpy.core | Version: 1.6.1 >> Keywords: | >> +--- >> >> Comment(by pv): >> >> The behavior does make sense, and it is what would naturally be expected. >> From the point of view of iteration, a N-dim ndarray *is* a collection of >> (N-1)-dim sub-arrays. The list constructor merely retains the consistency >> between `list(a)[j]` and `list(a[j])`. I don't see any reason to change >> this behavior, as raveling is a separate operation. >> >> -- >> Ticket URL: <http://projects.scipy.org/numpy/ticket/2067#comment:2> >> NumPy <http://projects.scipy.org/numpy> >> My example project > ___ > NumPy-Tickets mailing list > numpy-tick...@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-tickets -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Determining the 'viewness' of an array, and flags.owndata confusion
On Tue, Feb 28, 2012 at 23:01, Kurt Smith wrote: > For an arbitrary numpy array 'a', what does 'a.flags.owndata' indicate? > > I originally thought that owndata is False iff 'a' is a view. But > that is incorrect. > > Consider the following: > > In [119]: a = np.zeros((3,3)) > > In [120]: a.flags.owndata # should be True; zeros() creates and > returns a non-view array. > Out[120]: True > > In [121]: a_view1 = a[2:, :] > > In [122]: a_view1.flags.owndata # expected to be False > Out[122]: False > > In [123]: a_fancy1 = a[[0,1], :] > > In [124]: a_fancy1.flags.owndata # expected to be True, a_fancy1 is a > fancy-indexed array. > Out[124]: True > > In [125]: a_fancy2 = a[:, [0,1]] > > In [126]: a_fancy2.flags.owndata # expected to be True, a_fancy2 is a > fancy-indexed array. > Out[126]: False > > So when I query an array's flags.owndata, what is it telling me? What > I want to know is whether an array is a view or not. If flags.owndata > has nothing to do with the 'viewness' of an array, how would I > determine if an array is a view? > > In the previous example, a_fancy2 does not own its data, as indicated > by 'owndata' being False. But when I modify a_fancy2, 'a' is not > modified, as expected, but contrary to what 'owndata' would seem to > indicate. > > The numpybook's documentation of owndata could perhaps be a bit > clearer on the subject: > > """ > OWNDATA (O) the array owns the memory it uses or if it borrows it from > another object (if this is False, the base attribute retrieves a > reference to the object this array obtained its data from) > """ > > >From my reading, owndata has nothing to do with the 'viewness' of an > array. Is this correct? What is the intent of this flag, then? Its > wording could perhaps be improved: owndata is True iff (1) the array > owns the memory it uses or (2) the array borrows it from another > object. The second clause seems to indicate that the array **does > not** own its data since it is borrowing it from another object. > However, flags.owndata will be true in this case. > > If I cannot use flags.owndata, what is a reliable way to determine > whether or not an array is a view? Your original intuition was correct. It's just that sometimes an operation will make a copy of the input data, but then make a view on that copy. The output object is a view, just not a view on the input object. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers
On Wed, Feb 29, 2012 at 15:11, Erin Sheldon wrote: > Excerpts from Nathaniel Smith's message of Tue Feb 28 17:22:16 -0500 2012: >> > Even for binary, there are pathological cases, e.g. 1) reading a random >> > subset of nearly all rows. 2) reading a single column when rows are >> > small. In case 2 you will only go this route in the first place if you >> > need to save memory. The user should be aware of these issues. >> >> FWIW, this route actually doesn't save any memory as compared to np.memmap. > > Actually, for numpy.memmap you will read the whole file if you try to > grab a single column and read a large fraction of the rows. Here is an > example that will end up pulling the entire file into memory > > mm=numpy.memmap(fname, dtype=dtype) > rows=numpy.arange(mm.size) > x=mm['x'][rows] > > I just tested this on a 3G binary file and I'm sitting at 3G memory > usage. I believe this is because numpy.memmap only understands rows. I > don't fully understand the reason for that, but I suspect it is related > to the fact that the ndarray really only has a concept of itemsize, and > the fields are really just a reinterpretation of those bytes. It may be > that one could tweak the ndarray code to get around this. But I would > appreciate enlightenment on this subject. Each record (I would avoid the word "row" in this context) is contiguous in memory whether that memory is mapped to disk or not. Additionally, the way that virtual memory (i.e. mapped memory) works is that when you request the data at a given virtual address, the OS will go look up the page it resides in (typically 4-8k in size) and pull the whole page into main memory. Since you are requesting most of the records, you are probably pulling all of the file into main memory. Memory mapping works best when you pull out contiguous chunks at a time rather than pulling out stripes. numpy structured arrays do not rearrange your data to put all of the 'x' data contiguous with each other. You can arrange that yourself, if you like (use a structured scalar with a dtype such that each field is an array of the appropriate length and dtype). Then pulling out all of the 'x' field values will only touch a smaller fraction of the file. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Floating point "close" function?
On Sat, Mar 3, 2012 at 13:59, Ralf Gommers wrote: > > > On Thu, Mar 1, 2012 at 11:44 PM, Joe Kington wrote: >> >> Is there a numpy function for testing floating point equality that returns >> a boolean array? >> >> I'm aware of np.allclose, but I need a boolean array. Properly handling >> NaN's and Inf's (as allclose does) would be a nice bonus. >> >> I wrote the function below to do this, but I suspect there's a method in >> numpy that I missed. > > > I don't think such a function exists, would be nice to have. How about just > adding a keyword "return_array" to allclose to do so? As a general design principle, adding a boolean flag that changes the return type is worse than making a new function. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Floating point "close" function?
On Sat, Mar 3, 2012 at 14:31, Ralf Gommers wrote: > > > On Sat, Mar 3, 2012 at 3:05 PM, Robert Kern wrote: >> >> On Sat, Mar 3, 2012 at 13:59, Ralf Gommers >> wrote: >> > >> > >> > On Thu, Mar 1, 2012 at 11:44 PM, Joe Kington wrote: >> >> >> >> Is there a numpy function for testing floating point equality that >> >> returns >> >> a boolean array? >> >> >> >> I'm aware of np.allclose, but I need a boolean array. Properly >> >> handling >> >> NaN's and Inf's (as allclose does) would be a nice bonus. >> >> >> >> I wrote the function below to do this, but I suspect there's a method >> >> in >> >> numpy that I missed. >> > >> > >> > I don't think such a function exists, would be nice to have. How about >> > just >> > adding a keyword "return_array" to allclose to do so? >> >> As a general design principle, adding a boolean flag that changes the >> return type is worse than making a new function. > > > That's certainly true as a general principle. Do you have a concrete > suggestion in this case though? np.close() > Because this is also bad: >>>> np. > Display all 561 possibilities? (y or n) Not as bad as overloading np.allclose(x,y,return_array=True). Or deprecating np.allclose() in favor of np.close().all(). -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Floating point "close" function?
On Sat, Mar 3, 2012 at 15:22, Benjamin Root wrote: > > > On Saturday, March 3, 2012, Robert Kern wrote: >> On Sat, Mar 3, 2012 at 14:31, Ralf Gommers >> wrote: >>> >>> >>> On Sat, Mar 3, 2012 at 3:05 PM, Robert Kern >>> wrote: >>>> >>>> On Sat, Mar 3, 2012 at 13:59, Ralf Gommers >>>> wrote: >>>> > >>>> > >>>> > On Thu, Mar 1, 2012 at 11:44 PM, Joe Kington >>>> > wrote: >>>> >> >>>> >> Is there a numpy function for testing floating point equality that >>>> >> returns >>>> >> a boolean array? >>>> >> >>>> >> I'm aware of np.allclose, but I need a boolean array. Properly >>>> >> handling >>>> >> NaN's and Inf's (as allclose does) would be a nice bonus. >>>> >> >>>> >> I wrote the function below to do this, but I suspect there's a method >>>> >> in >>>> >> numpy that I missed. >>>> > >>>> > >>>> > I don't think such a function exists, would be nice to have. How about >>>> > just >>>> > adding a keyword "return_array" to allclose to do so? >>>> >>>> As a general design principle, adding a boolean flag that changes the >>>> return type is worse than making a new function. >>> >>> >>> That's certainly true as a general principle. Do you have a concrete >>> suggestion in this case though? >> >> np.close() >> > > When I read that, I mentally think of "close" as in closing a file. I think > we need a synonym. np.isclose() -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Floating point "close" function?
On Sat, Mar 3, 2012 at 14:34, Robert Kern wrote: > On Sat, Mar 3, 2012 at 14:31, Ralf Gommers > wrote: >> Because this is also bad: >>>>> np. >> Display all 561 possibilities? (y or n) > > Not as bad as overloading np.allclose(x,y,return_array=True). Or > deprecating np.allclose() in favor of np.close().all(). I screwed up this paragraph. I meant that as "Another alternative would be to deprecate ...". -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Floating point "close" function?
On Sat, Mar 3, 2012 at 15:51, Olivier Delalleau wrote: > Le 3 mars 2012 10:27, Robert Kern a écrit : >> >> On Sat, Mar 3, 2012 at 14:34, Robert Kern wrote: >> > On Sat, Mar 3, 2012 at 14:31, Ralf Gommers >> > wrote: >> >> >> Because this is also bad: >> >>>>> np. >> >> Display all 561 possibilities? (y or n) >> > >> > Not as bad as overloading np.allclose(x,y,return_array=True). Or >> > deprecating np.allclose() in favor of np.close().all(). >> >> I screwed up this paragraph. I meant that as "Another alternative >> would be to deprecate ...". > > > np.close().all() would probably be a lot less efficient in terms of CPU / > memory though, wouldn't it? No. np.allclose() is essentially doing exactly this already. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Floating point "close" function?
On Sat, Mar 3, 2012 at 16:06, Olivier Delalleau wrote: > Le 3 mars 2012 11:03, Robert Kern a écrit : >> >> On Sat, Mar 3, 2012 at 15:51, Olivier Delalleau wrote: >> > Le 3 mars 2012 10:27, Robert Kern a écrit : >> >> >> >> On Sat, Mar 3, 2012 at 14:34, Robert Kern >> >> wrote: >> >> > On Sat, Mar 3, 2012 at 14:31, Ralf Gommers >> >> > >> >> > wrote: >> >> >> >> >> Because this is also bad: >> >> >>>>> np. >> >> >> Display all 561 possibilities? (y or n) >> >> > >> >> > Not as bad as overloading np.allclose(x,y,return_array=True). Or >> >> > deprecating np.allclose() in favor of np.close().all(). >> >> >> >> I screwed up this paragraph. I meant that as "Another alternative >> >> would be to deprecate ...". >> > >> > >> > np.close().all() would probably be a lot less efficient in terms of CPU >> > / >> > memory though, wouldn't it? >> >> No. np.allclose() is essentially doing exactly this already. > > > Ok. What about then, np.allclose() could theoretically be a lot more > efficient in terms of CPU / memory? ;) True, but so could a hypothetical np.allequal(), np.allless_equal(), etc. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fixing PyArray_Descr flags member size, ABI vs pickling issue
On Tue, Mar 6, 2012 at 03:53, David Cournapeau wrote: > Hi, > > This is following the discussion on bug > http://projects.scipy.org/numpy/ticket/2017 > > Essentially, there is a discrepency between the actual type of the > flags member in the dtype C structure (char) and how it is declared in > the descriptor table (T_INT). The problem is that we are damned if we > fix it, damned if we are not: > - fixing T_INT to T_BYTE. flag in python is now fixed, but it breaks > pickled numpy arrays Is the problem that T_BYTE returns a single-item string? My handwrapping skills are rusty (Cython has blissfully eradicated this from my memory), but aren't these T_INT/T_BYTE things just convenient shortcuts for exposing C struct members as Python attributes? Couldn't we just write a full getter function for returning the correct value, just returned as a Python int instead of a str. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fixing PyArray_Descr flags member size, ABI vs pickling issue
On Tue, Mar 6, 2012 at 18:25, Travis Oliphant wrote: > Why do we want to return a single string char instead of an int? I suspect just to ensure that any provided value fits in the range 0..255. But that's easily done explicitly. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] [enhancement] sum_angle() and sum_polar()
Hi everyone, I am proposing to add the the two following functions to numpy/lib/twodim_base.py: sum_angle() computes the sum of a 2-d array along an angled axis sum_polar() computes the sum of a 2-d array along radial lines or along azimuthal circles https://github.com/numpy/numpy/pull/230 Comments? When I was looking for a solution to these problems of calculating special sums of 2-d arrays I could not find anything and it took me a while to figure out a (hopefully) useful and consistent algorithm. I can see how one would extend these to higher dimensions but that would preclude using bincount() to do the heavy lifting. Looking at some other functions, the doctests might need to be split into real examples and unittests. Best, -- Robert Jordens. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to check type of array?
On Thu, Mar 29, 2012 at 13:47, Derek Homeier
wrote:
> On 29 Mar 2012, at 13:54, Chao YUE wrote:
>
>> how can I check type of array in if condition expression?
>>
>> In [75]: type(a)
>> Out[75]:
>>
>> In [76]: a.dtype
>> Out[76]: dtype('int32')
>>
>> a.dtype=='int32'?
>
> this and
>
> a.dtype=='i4'
> a.dtype==np.int32
>
> all work. For a more general check (e.g. if it is any type of integer), you
> can do
>
> np.issubclass_(a.dtype.type, np.integer)
I don't recommend using that. Use np.issubdtype(a.dtype, np.integer) instead.
--
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Getting C-function pointers from Python to C
On Tue, Apr 10, 2012 at 01:11, Travis Oliphant wrote:
> 1) Create an API for such Ctypes function pointers in NumPy and use
> the ctypes object structure. If ctypes were to ever change it's object
> structure we would have to adapt this API.
>
> Something like this is what is envisioned here:
>
> typedef struct {
> PyObject_HEAD
> char *b_ptr;
> } _cfuncptr_object;
Why not just use PyCapsules?
http://docs.python.org/release/2.7/c-api/capsule.html
--
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] datetime dtype possible regression
On Fri, Apr 27, 2012 at 21:52, Travis Vaught wrote:
> With NumPy 1.6.1 (from EPD 7.2-2) I get this behavior:
>
>
> ~
>
> In [1]: import numpy as np
>
> In [2]: schema = np.dtype({'names':['symbol', 'date', 'open', 'high', 'low',
> ...: 'close', 'volume', 'adjclose'],
> ...: 'formats':['S8', 'M8', float, float, float,
> float,
> ...: float, float]})
>
> In [3]: data = [("AAPL", "2012-04-12", 600.0, 605.0, 598.0, 602.0, 5000,
> 602.0),]
>
> In [4]: recdata = np.array(data, dtype=schema)
>
> In [5]: recdata
> Out[5]:
> array([ ('AAPL', datetime.datetime(2012, 4, 12, 0, 0), 600.0, 605.0, 598.0,
> 602.0, 5000.0, 602.0)],
> dtype=[('symbol', '|S8'), ('date', (' ('high', ' ('adjclose', '
>
>
>
>
> With numpy-1.7.0.dev_3cb783e I get this:
>
>>>> import numpy as np
>
>>>> schema =
>>>> np.dtype({'names':['symbol','data','open','high','low','close','volume','adjclose'],
>>>> 'formats':['S8','M8',float,float,float,float,float,float]})
>
>>>> data = [("AAPL", "2012-04-12", 600.0, 605.0, 598.0, 602.0, 5000,
>>>> 602.0),]
>
>>>> recdata = np.array(data, dtype=schema)
> Traceback (most recent call last):
> File "", line 1, in
> ValueError: Cannot create a NumPy datetime other than NaT with generic units
>
>>>> np.version.version
> '1.7.0.dev-3cb783e'
>
>
>
> Any hints about a regression I can check for? Or perhaps I missed an api
> change for specifying datetime dtypes?
Judging from the error message, it looks like an intentional API change.
--
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Alternative to R phyper
On Mon, Apr 30, 2012 at 12:09, Bruno Santos wrote: > Hello everyone, > > I have a bit of code where I am using rpy2 to import R phyper so I can > perform an hypergeometric test. Unfortunately our cluster does not have a > functional installation of rpy2 working. So I am wondering if I could > translate to scipy which would make the code completly independent of R. The > python code I have is as following: > > > def lphyper(self,q,m,n,k): > """ > self.phyper(self,q,m,n,k) > Calculate p-value using R function phyper from rpy2 low-level > interface. > "R Documentation > phyper(q, m, n, k, lower.tail = TRUE, log.p = FALSE) > q: vector of quantiles representing the number of white balls > drawn without replacement from an urn which contains both > black and white balls. > m: the number of white balls in the urn. > n: the number of black balls in the urn. > k: the number of balls drawn from the urn. > log.p: logical; if TRUE, probabilities p are given as log(p). > lower.tail: logical; if TRUE (default), probabilities are P[X <= x], > otherwise, P[X > x]. > """ > phyper_q = SexpVector([q,], rinterface.INTSXP) > phyper_m = SexpVector([m,], rinterface.INTSXP) > phyper_n = SexpVector([n,], rinterface.INTSXP) > phyper_k = SexpVector([k,], rinterface.INTSXP) > return phyper(phyper_q,phyper_m,phyper_n,phyper_k,**myparams)[0] > > I have looked to scipy.stats.hypergeom but it is giving me a different > result which is also negative. >> 1-phyper(45, 92, 7518, 1329) > [1] 6.92113e-13 > > In [24]: stats.hypergeom.sf(45,(92+7518),92,1329) > Out[24]: -8.4343643180773142e-12 The CDF (CMF? whatever) for stats.hypergeom is not implemented explicitly, so it falls back to the default implementation of just summing up the PMF. You are falling victim to floating point error accumulation such that the sum exceeds 1.0, so the survival function 1-CDF ends up negative. I don't think we have a better implementation of the CDF anywhere in scipy, sorry. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Status of np.bincount
Hello Everybody, is there any news on the status of np.bincount with respect to "big" numbers? It seems I have just been bitten by #225. Is there an efficient way around? I found the np.histogram function painfully slow. Below a simple script, that demonstrates bincount failing with a memory error on big numbers import numpy as np x = np.array((30e9,)).astype(int) np.bincount(x) Any good idea how to work around it. My arrays contain somewhat 50M entries in the range from 0 to 30e9. And I would like to have them bincounted... cheers Robert ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Status of np.bincount
On Thu, May 3, 2012 at 2:24 PM, Robert Elsner wrote: > Hello Everybody, > > is there any news on the status of np.bincount with respect to "big" > numbers? It seems I have just been bitten by #225. Is there an efficient > way around? I found the np.histogram function painfully slow. > > Below a simple script, that demonstrates bincount failing with a memory > error on big numbers > > import numpy as np > > x = np.array((30e9,)).astype(int) > np.bincount(x) > > > Any good idea how to work around it. My arrays contain somewhat 50M > entries in the range from 0 to 30e9. And I would like to have them > bincounted... You need a sparse data structure, then. Are you sure you even have duplicates? Anyways, I won't work out all of the details, but let me sketch something that might get you your answers. First, sort your array. Then use np.not_equal(x[:-1], x[1:]) as a mask on np.arange(1,len(x)) to find the indices where each sorted value changes over to the next. The np.diff() of that should give you the size of each. Use np.unique to get the sorted unique values to match up with those sizes. Fixing all of the off-by-one errors and dealing with the boundary conditions correctly is left as an exercise for the reader. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Status of np.bincount
Am 03.05.2012 15:45, schrieb Robert Kern: > On Thu, May 3, 2012 at 2:24 PM, Robert Elsner wrote: >> Hello Everybody, >> >> is there any news on the status of np.bincount with respect to "big" >> numbers? It seems I have just been bitten by #225. Is there an efficient >> way around? I found the np.histogram function painfully slow. >> >> Below a simple script, that demonstrates bincount failing with a memory >> error on big numbers >> >> import numpy as np >> >> x = np.array((30e9,)).astype(int) >> np.bincount(x) >> >> >> Any good idea how to work around it. My arrays contain somewhat 50M >> entries in the range from 0 to 30e9. And I would like to have them >> bincounted... > > You need a sparse data structure, then. Are you sure you even have duplicates? > > Anyways, I won't work out all of the details, but let me sketch > something that might get you your answers. First, sort your array. > Then use np.not_equal(x[:-1], x[1:]) as a mask on np.arange(1,len(x)) > to find the indices where each sorted value changes over to the next. > The np.diff() of that should give you the size of each. Use np.unique > to get the sorted unique values to match up with those sizes. > > Fixing all of the off-by-one errors and dealing with the boundary > conditions correctly is left as an exercise for the reader. > ?? I suspect that this mail was meant to end up in the thread about sparse array data? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Status of np.bincount
On Thu, May 3, 2012 at 2:50 PM, Robert Elsner wrote: > > Am 03.05.2012 15:45, schrieb Robert Kern: >> On Thu, May 3, 2012 at 2:24 PM, Robert Elsner wrote: >>> Hello Everybody, >>> >>> is there any news on the status of np.bincount with respect to "big" >>> numbers? It seems I have just been bitten by #225. Is there an efficient >>> way around? I found the np.histogram function painfully slow. >>> >>> Below a simple script, that demonstrates bincount failing with a memory >>> error on big numbers >>> >>> import numpy as np >>> >>> x = np.array((30e9,)).astype(int) >>> np.bincount(x) >>> >>> >>> Any good idea how to work around it. My arrays contain somewhat 50M >>> entries in the range from 0 to 30e9. And I would like to have them >>> bincounted... >> >> You need a sparse data structure, then. Are you sure you even have >> duplicates? >> >> Anyways, I won't work out all of the details, but let me sketch >> something that might get you your answers. First, sort your array. >> Then use np.not_equal(x[:-1], x[1:]) as a mask on np.arange(1,len(x)) >> to find the indices where each sorted value changes over to the next. >> The np.diff() of that should give you the size of each. Use np.unique >> to get the sorted unique values to match up with those sizes. >> >> Fixing all of the off-by-one errors and dealing with the boundary >> conditions correctly is left as an exercise for the reader. >> > > ?? I suspect that this mail was meant to end up in the thread about > sparse array data? No, I am responding to you. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Should arr.diagonal() return a copy or a view? (1.7 compatibility issue)
On Wed, May 16, 2012 at 3:10 PM, Nathaniel Smith wrote: > On Wed, May 16, 2012 at 3:04 PM, Benjamin Root wrote: >> Just as a sanity check, do the scipy tests run without producing any such >> messages? > > I tried checking this before, actually, but can't figure out how to > build scipy against a copy of numpy that is installed in either a > virtualenv or just on PYTHONPATH. (Basically, I just don't want to > install some random development numpy into my system python.) Any > suggestions? scipy will build against whatever numpy the python executable that runs the setup.py manages to import. So if you are using virtualenv, just make sure that the virtualenv is activated and "python" refers to the virtualenv's python executable. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Scipy build can't find BLAS when using numpy master? (Was: Should arr.diagonal() return a copy or a view? (1.7 compatibility issue))
On Wed, May 16, 2012 at 4:21 PM, Nathaniel Smith wrote: > I built some pristine python 2.7 installs from scratch (no virtualenv, > no distro tweaks, etc.). Then I installed some version of numpy in > each, then tried building scipy. With numpy 1.6.1 (built from git), > everything seems fine - it finds atlas in /usr/lib64/atlas. > > With current numpy master (3d416a0a), numpy itself builds fine, > but scipy's setup.py can't find atlas (even though it's looking in the > right places). Log attached. The log says it's looking in /usr/lib64/atlas-base/, not /usr/lib64/atlas/. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Scipy build can't find BLAS when using numpy master? (Was: Should arr.diagonal() return a copy or a view? (1.7 compatibility issue))
On Wed, May 16, 2012 at 4:35 PM, Nathaniel Smith wrote:
> On Wed, May 16, 2012 at 4:24 PM, Robert Kern wrote:
>> On Wed, May 16, 2012 at 4:21 PM, Nathaniel Smith wrote:
>>
>>> I built some pristine python 2.7 installs from scratch (no virtualenv,
>>> no distro tweaks, etc.). Then I installed some version of numpy in
>>> each, then tried building scipy. With numpy 1.6.1 (built from git),
>>> everything seems fine - it finds atlas in /usr/lib64/atlas.
>>>
>>> With current numpy master (3d416a0a), numpy itself builds fine,
>>> but scipy's setup.py can't find atlas (even though it's looking in the
>>> right places). Log attached.
>>
>> The log says it's looking in /usr/lib64/atlas-base/, not /usr/lib64/atlas/.
>
> Sorry, that was an error in my email. The libraries are actually in
> /usr/lib64/atlas-base, and when I build against numpy 1.6.1, the build
> log says:
>
> Setting PTATLAS=ATLAS
> FOUND:
> libraries = ['ptf77blas', 'ptcblas', 'atlas']
> library_dirs = ['/usr/lib64/atlas-base']
> language = c
> define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
> include_dirs = ['/usr/include/atlas']
>
> FOUND:
> libraries = ['ptf77blas', 'ptcblas', 'atlas']
> library_dirs = ['/usr/lib64/atlas-base']
> language = c
> define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
> include_dirs = ['/usr/include/atlas']
Hmm. I would throw in some print statements into the "_check_libs()"
and "_lib_list()" methods in numpy/distutils/system_info.py and see
what it's checking for.
--
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] pre-PEP for making creative forking of NumPy less destructive
On Fri, May 18, 2012 at 6:59 PM, Dag Sverre Seljebotn wrote: > On 05/18/2012 05:00 PM, Henry Gomersall wrote: >> On Fri, 2012-05-18 at 14:45 +0200, Dag Sverre Seljebotn wrote: >>> I would focus on the 'polymorphic C API' spin. PyObject_GetItem is >>> polymorphic, but there is no standard way for 3rd party libraries to >>> make such functions. >>> >>> So let's find a C API that's NOT about arrays at all and show how some >>> polymorphism may help. >>> >> so memory mapped IO or something? > > A C API towards a Python extension that's in wide use; an analog to the > NumPy C API in a different domain. Something like a web server or > database Python extension module which also exports a C API for writing > other Python extension modules against it. > > I'm not even sure if such a thing exists, in which case NumPy would > indeed be a special case. numpy *is* pretty unique in this regard. The need for this style of polymorphism at this level is even rarer. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] un-silencing Numpy's deprecation warnings
On Tue, May 22, 2012 at 9:27 AM, Nathaniel Smith wrote: > So starting in Python 2.7 and 3.2, the Python developers have made > DeprecationWarnings invisible by default: > http://docs.python.org/whatsnew/2.7.html#the-future-for-python-2-x > http://mail.python.org/pipermail/stdlib-sig/2009-November/000789.html > http://bugs.python.org/issue7319 > The only way to see them is to explicitly request them by running > Python with -Wd. > > The logic seems to be that between the end-of-development for 2.7 and > the moratorium on 3.2 changes, there were a *lot* of added > deprecations that were annoying people, and deprecations in the Python > stdlib mean "this code is probably sub-optimal but it will still > continue to work indefinitely". That's not quite it, I think, since this change was also made in Python 3.2 and will remain for all future versions of Python. DeprecationWarning *is* used for things that will definitely be going away, not just things that are no longer recommended but will continue to live. Note that the 3.2 moratorium was for changes to the language proper. The point was to encourage stdlib development, including the removal of deprecated code. It was not a moratorium on removing deprecated things. The silencing discussion just came up first in a discussion on the moratorium. The main problem they were running into was that the people who saw these warnings the most were not people directly using the deprecated features; they were users of packages written by third parties that used the deprecated features. Those people can't do anything to fix the problem, and many of them think that something is broken when they see the warning (I don't know why people do this, but they do). This problem is exacerbated by the standard library's position as a standard library. It's at the base of everyone's stack so these indirect effects are quite frequent, quite possibly the majority case. Users would use a newer version of Python library than the third party developer tested on and see these errors instead of the developer. I think this concern is fairly general and applies to numpy nearly as much as it does the standard library. It is at the bottom of many people's stacks. Someone calling matplotlib.pyplot.plot() should not see a DeprecationWarning from numpy. > So they consider that deprecation > warnings are like a lint tool for conscientious developers who > remember to test their code with -Wd, but not something to bother > users with. > > In Numpy, the majority of our users are actually (relatively > unsophisticated) developers, Whether they sometimes wear a developer hat or not isn't the relevant distinction. The question to ask is, "Are they the ones writing the code that directly uses the deprecated features?" > and we don't plan to support deprecated > features indefinitely. Again, this is not relevant. The silencing of DeprecationWarnings was not driven by this. > Our deprecations seem to better match what > Python calls a "FutureWarning": "warnings about constructs that will > change semantically in the future." > http://docs.python.org/library/warnings.html#warning-categories > FutureWarning is displayed by default, and available in all versions of > Python. > > So maybe we should change all our DeprecationWarnings into > FutureWarnings (or at least the ones that we actually plan to follow > through on). Thoughts? Using FutureWarning for deprecated functions (i.e. functions that will disappear in future releases) is an abuse of the semantics. FutureWarning is for things like the numpy.histogram() changes from a few years ago: changes in default arguments that will change the semantics of a given function call. Some of our DeprecationWarnings possibly should be FutureWarnings, but most shouldn't I don't think. I can see a case being made for using a custom non-silenced exception for some cases that really probably show up mostly in true end-user scenarios, e.g. genfromtxt(). But there are many other cases where we should continue to use DeprecationWarning, e.g. _array2string(). But on the whole, I would just leave the DeprecationWarnings as they are. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] un-silencing Numpy's deprecation warnings
On Tue, May 22, 2012 at 11:14 AM, Dag Sverre Seljebotn wrote: > On 05/22/2012 12:06 PM, Robert Kern wrote: >> On Tue, May 22, 2012 at 9:27 AM, Nathaniel Smith wrote: >>> So maybe we should change all our DeprecationWarnings into >>> FutureWarnings (or at least the ones that we actually plan to follow >>> through on). Thoughts? >> >> Using FutureWarning for deprecated functions (i.e. functions that will >> disappear in future releases) is an abuse of the semantics. >> FutureWarning is for things like the numpy.histogram() changes from a >> few years ago: changes in default arguments that will change the >> semantics of a given function call. Some of our DeprecationWarnings >> possibly should be FutureWarnings, but most shouldn't I don't think. > > I guess the diagonal() change would at least be a FutureWarning then? > (When you write to the result?) Sure. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] un-silencing Numpy's deprecation warnings
On Tue, May 22, 2012 at 2:45 PM, Nathaniel Smith wrote: > On Tue, May 22, 2012 at 11:06 AM, Robert Kern wrote: >> On Tue, May 22, 2012 at 9:27 AM, Nathaniel Smith wrote: >>> So starting in Python 2.7 and 3.2, the Python developers have made >>> DeprecationWarnings invisible by default: >>> http://docs.python.org/whatsnew/2.7.html#the-future-for-python-2-x >>> http://mail.python.org/pipermail/stdlib-sig/2009-November/000789.html >>> http://bugs.python.org/issue7319 >>> The only way to see them is to explicitly request them by running >>> Python with -Wd. >>> >>> The logic seems to be that between the end-of-development for 2.7 and >>> the moratorium on 3.2 changes, there were a *lot* of added >>> deprecations that were annoying people, and deprecations in the Python >>> stdlib mean "this code is probably sub-optimal but it will still >>> continue to work indefinitely". >> >> That's not quite it, I think, since this change was also made in >> Python 3.2 and will remain for all future versions of Python. >> DeprecationWarning *is* used for things that will definitely be going >> away, not just things that are no longer recommended but will continue >> to live. Note that the 3.2 moratorium was for changes to the language >> proper. The point was to encourage stdlib development, including the >> removal of deprecated code. It was not a moratorium on removing >> deprecated things. The silencing discussion just came up first in a >> discussion on the moratorium. >> >> The main problem they were running into was that the people who saw >> these warnings the most were not people directly using the deprecated >> features; they were users of packages written by third parties that >> used the deprecated features. Those people can't do anything to fix >> the problem, and many of them think that something is broken when they >> see the warning (I don't know why people do this, but they do). This >> problem is exacerbated by the standard library's position as a >> standard library. It's at the base of everyone's stack so these >> indirect effects are quite frequent, quite possibly the majority case. >> Users would use a newer version of Python library than the third party >> developer tested on and see these errors instead of the developer. >> >> I think this concern is fairly general and applies to numpy nearly as >> much as it does the standard library. It is at the bottom of many >> people's stacks. Someone calling matplotlib.pyplot.plot() should not >> see a DeprecationWarning from numpy. > > Yes, good points -- though I think there is a also real cost/benefit > trade-off that depends on the details of how often these warnings are > issued, the specific user base, etc. > > Compared to stdlib, a *much* higher proportion of numpy-using code > consists of scripts whose only users are their authors, who didn't > think very carefully about error handling, and who will continue to > use these scripts for long periods of time (i.e. over multiple > releases). So I feel like we should have a higher threshold for making > warnings silent by default. > > OTOH, the distinction you suggest does make sense. I would summarize it as: > > - If a function or similar will just disappear in a future release, > causing obvious failures in any code that depends on it, then > DeprecationWarning is fine. People's code will unexpectedly break from > time to time, but in safe ways, and anyway downgrading is easy. > - Otherwise FutureWarning is preferred > > Does that sound like a reasonable rule of thumb? Sure. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] question about in-place operations
On Tue, May 22, 2012 at 3:47 PM, Massimo DiPierro wrote: > Thank you. I will look into numexpr. > > Anyway, I do not need arbitrary expressions. If there were something like > > numpy.add_scaled(a,scale,b) > > with support for scale in int, float, complex, this would be sufficient for > me. BLAS has the xAXPY functions, which will do this for float and complex. import numpy as np from scipy.linalg import fblas def add_scaled_inplace(a, scale, b): if np.issubdtype(a.dtype, complex): fblas.zaxpy(b, a, a=scale) else: fblas.daxpy(b, a, a=scale) -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to avoid re-shaping
On Tue, May 22, 2012 at 4:09 PM, Massimo DiPierro wrote: > One more questions (since this list is very useful. ;-) > > If I have a numpy array of arbitrary shape, is there are a way to > sequentially loop over its elements without reshaping it into a 1D array? > > I am trying to simplify this: > > n=product(data.shape) > oldshape = data.shape > newshape = (n,) > data.reshape(newshape) Note that the .reshape() method does not work in-place. It just returns a new ndarray object viewing the same data using the different shape. That said, just iterate over data.flat, if you must iterate manually. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Checking for views (was: Should arr.diagonal() return a copy or aview?)
On Thu, May 24, 2012 at 4:10 PM, Nathaniel Smith wrote: > On Thu, May 24, 2012 at 3:56 PM, Jonathan T. Niehof wrote: >> On 05/23/2012 05:31 PM, T J wrote: >> >>> It seems that there are a number of ways to check if an array is a view. >>> Do we have a preferred way in the API that is guaranteed to stay >>> available? Or are all of the various methods "here to stay"? >> >> We've settled on checking array.base, which I think was the outcome of a >> stackoverflow thread that I can't dig up. (I'll check with the guy who >> wrote the code.) > > The problem is that "is a view" isn't a very meaningful concept... > checking .base will tell you whether writes to an array are likely to > affect some object that existed before that array was created. But it > doesn't tell you whether writes to that array can affect any > *particular* other object (at least without a fair amount of groveling > around the innards of both objects), and it can happen that an object > has base == None yet writes to it will affect another object, and it > can happen that an object has base != None and yet writes to it won't > affect any object that was ever accessible to your code. AFAICT it's > really these other questions that one would like to answer, and > checking .base won't answer them. numpy.may_share_memory() gets closer, but it can be defeated by certain striding patterns. At least, it is conservative and reports false positives but not false negatives. Implementing numpy.does_share_memory() correctly involves some number theory and hairy edge cases. (Hmm, now that I think about it, the edge cases are when the strides are 0 or negative. 0-stride axes can simply be removed, and I think we should be able to work back to a first item and flip the sign on the negative strides. The typical positive-stride solution can be found in an open source C++ global array code, IIRC. Double-hmmm...) -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Checking for views (was: Should arr.diagonal() return a copy or aview?)
On Thu, May 24, 2012 at 5:52 PM, Robert Kern wrote: > (Hmm, now that I think about it, the edge cases are when the strides > are 0 or negative. 0-stride axes can simply be removed, and I think we > should be able to work back to a first item and flip the sign on the > negative strides. The typical positive-stride solution can be found in > an open source C++ global array code, IIRC. Double-hmmm...) Except that it's still NP-complete. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Checking for views (was: Should arr.diagonal() return a copy or aview?)
On Fri, May 25, 2012 at 3:55 PM, Nathaniel Smith wrote: > On May 25, 2012 2:21 PM, "Robert Kern" wrote: >> >> On Thu, May 24, 2012 at 5:52 PM, Robert Kern wrote: >> >> > (Hmm, now that I think about it, the edge cases are when the strides >> > are 0 or negative. 0-stride axes can simply be removed, and I think we >> > should be able to work back to a first item and flip the sign on the >> > negative strides. The typical positive-stride solution can be found in >> > an open source C++ global array code, IIRC. Double-hmmm...) >> >> Except that it's still NP-complete. > > Huh, is it really? I'm pretty sure checking the existence of a > solution to a linear Diophantine equation is cheap, but I guess > figuring out whether it falls within the "shape" bounds is less > obvious... I believe that's what this is telling me: http://permalink.gmane.org/gmane.comp.gcc.fortran/11797 -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Checking for views (was: Should arr.diagonal() return a copy or aview?)
On Fri, May 25, 2012 at 3:55 PM, Nathaniel Smith wrote:
> On May 25, 2012 2:21 PM, "Robert Kern" wrote:
>>
>> On Thu, May 24, 2012 at 5:52 PM, Robert Kern wrote:
>>
>> > (Hmm, now that I think about it, the edge cases are when the strides
>> > are 0 or negative. 0-stride axes can simply be removed, and I think we
>> > should be able to work back to a first item and flip the sign on the
>> > negative strides. The typical positive-stride solution can be found in
>> > an open source C++ global array code, IIRC. Double-hmmm...)
>>
>> Except that it's still NP-complete.
>
> Huh, is it really? I'm pretty sure checking the existence of a
> solution to a linear Diophantine equation is cheap, but I guess
> figuring out whether it falls within the "shape" bounds is less
> obvious...
If both positive and negative values are allowed, then there is a
polynomial-time algorithm to solve the linear Diophantine equation,
but bounding the possible values renders it NP-complete. When you go
down to {0,1} as the only allowable values, it becomes the SUBSET-SUM
problem.
--
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [enhancement] sum_angle() and sum_polar()
On Tue, May 29, 2012 at 11:03 AM, Stéfan van der Walt wrote: > On Mon, May 28, 2012 at 11:53 AM, Travis Oliphant wrote: >> I could see these functions going into scipy.ndimage but again because they >> are not necessarily just image processing functions, and the fact that they >> are so simple, perhaps they are best put into NumPy itself. > > I'm wondering about the general applicability of these functions. Can > anyone suggest some use cases? An example from solid state physics: If you have a spin chain with some long-range interaction and you have the known, dense, coupling matrix J, sum_angle(J, pi/4) gives you a view at the distance dependence of the interaction. -- Robert Jordens. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [enhancement] sum_angle() and sum_polar()
On Tue, May 29, 2012 at 12:06 PM, Charles R Harris wrote: > I'd like to see these functions is scipy somewhere. The function names > aren't very descriptive and the one line summaries don't give a very good > idea of what they do, so I think those bits could use improvement. Mention > of the Hough/Radon transform would help, I had to pull out that connection > by reading the code... I'll fix the descriptions. What more descriptive names did you have in mind? -- Robert Jordens. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy code in GPL package
On Wed, May 30, 2012 at 4:13 PM, Henry Gomersall wrote: > I'd like to include the _cook_nd_args() function from fftpack in my GPL > code. Is this possible? Yes. The numpy license is compatible with the GPL license, so code from numpy may be incorporated into GPL programs. > How should I modify my license file to satisfy the Numpy license > requirements, but so it's clear which function it applies to? The clearest thing to do is to keep the _cook_nd_args() function implemented in its own file which has the numpy license text placed in a comment at the top. You should have a LICENSE.txt file that says that your program is GPLed (and point to the COPYING file that contains the GPL text) and also mentions that the _cook_nd_args file (which you should mention by name) has the numpy license. Since the numpy license is relatively small, you can just copy the full numpy license into your LICENSE.txt file. Something like this: Copyright (C) 2012 Henry Gomersall This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA 3rd Party Addendum == The file cook_nd_args.c is derived from NumPy, and is available under the Modified BSD License: Copyright (c) 2005-2011, NumPy Developers. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the NumPy Developers nor the names of any contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] How to remove any row or column of a numpy matrix whose sum is 3?
On Mon, Jun 4, 2012 at 5:21 PM, bob tnur wrote: > Hello every body. I am new to python. > How to remove any row or column of a numpy matrix whose sum is 3. > To obtain and save new matrix P with (sum(anyrow)!=3 and sum(anycolumn)!=3 > elements. > > I tried like this: > > P = M[np.logical_not( (M[n,:].sum()==3) & (M[:,n].sum()==3))] > or > P = M[np.logical_not( (np.sum(M[n,:])==3) & (np.sum(M[:,n])==3))] > > > M is the nxn numpy matrix. > But I got indexerror. So can anyone correct this or any other elegant way of > doing this? If M is 5x5 matrix, then M[5,:] and M[:,5] don't work. You can't index past the last element. Python sequences in general and numpy arrays in particular use 0-based indexing. I'm not entirely sure what you intended with those expressions anyways. Here is how I would do it. # Get the integer indices of the rows that sum up to 3 # and the columns that sum up to 3. bad_rows = np.nonzero(M.sum(axis=1) == 3) bad_cols = np.nonzero(M.sum(axis=0) == 3) # Now use the numpy.delete() function to get the matrix # with those rows and columns removed from the original matrix. P = np.delete(M, bad_rows, axis=0) P = np.delete(P, bad_cols, axis=1) -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] varargs for logical_or, etc
On Tue, Jun 5, 2012 at 2:54 PM, Neal Becker wrote: > I think it's unfortunate that functions like logical_or are limited to binary. > > As a workaround, I've been using this: > > def apply_binary (func, *args): > if len (args) == 1: > return args[0] > elif len (args) == 2: > return func (*args) > else: > return func ( > apply_binary (func, *args[:len(args)/2]), > apply_binary (func, *args[(len(args))/2:])) > > Then for example: > > punc2 = np.logical_and (u % 5 == 4, > apply_binary (np.logical_or, u/5 == 3, u/5 == 8, u/5 == > 13)) reduce(np.logical_and, args) -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Incrementing with advanced indexing: why don't repeated indexes repeatedly increment?
On 06/06/2012 05:06 PM, Nathaniel Smith wrote: > On Wed, Jun 6, 2012 at 9:48 AM, John Salvatier > wrote: >> Hello, >> >> I've noticed that If you try to increment elements of an array with advanced >> indexing, repeated indexes don't get repeatedly incremented. For example: >> >> In [30]: x = zeros(5) >> >> In [31]: idx = array([1,1,1,3,4]) >> >> In [32]: x[idx] += [2,4,8,10,30] >> >> In [33]: x >> Out[33]: array([ 0., 8., 0., 10., 30.]) >> >> I would intuitively expect the output to be array([0,14, 0,10,30]) since >> index 1 is incremented by 2+4+8=14, but instead it seems to only increment >> by 8. What is numpy actually doing here? >> >> The authors of Theano noticed this behavior a while ago so they python loop >> through the values in idx (this kind of calculation is necessary for >> calculating gradients), but this is a bit slow for my purposes, so I'd like >> to figure out how to get the behavior I expected, but faster. >> >> I'm also not sure how to navigate the numpy codebase, where would I look for >> the code responsible for this behavior? > > Strictly speaking, it isn't actually in the numpy codebase at all -- > what's happening is that the Python interpreter sees this code: > >x[idx] += vals > > and then it translates it into this code before running it: > >tmp = x.__getitem__(idx) >tmp = tmp.__iadd__(vals) >x.__setitem__(idx, tmp) > > So you can find the implementations of the ndarray methods > __getitem__, __iadd__, __setitem__ (they're called > array_subscript_nice, array_inplace_add, and array_ass_sub in the C > code), but there's no way to fix them so that this works the way you > want it to, because there's no way for __iadd__ to know that the > temporary values that it's working with are really duplicate copies of > "the same" value in the original array. > > It would be nice if numpy had some sort of standard API for doing what > you want, but not sure what a good API would look like, and someone > would have to implement it. This operation is also heavily used for the finite element assembling, and a similar question has been raised already several times (e.g. http://old.nabble.com/How-to-assemble-large-sparse-matrices-effectively-td33833855.html). So why not adding a function np.assemble()? r. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Incrementing with advanced indexing: why don't repeated indexes repeatedly increment?
On 06/06/2012 05:34 PM, Nathaniel Smith wrote: > On Wed, Jun 6, 2012 at 4:30 PM, Robert Cimrman wrote: >> On 06/06/2012 05:06 PM, Nathaniel Smith wrote: >>> On Wed, Jun 6, 2012 at 9:48 AM, John Salvatier >>> wrote: >>>> Hello, >>>> >>>> I've noticed that If you try to increment elements of an array with >>>> advanced >>>> indexing, repeated indexes don't get repeatedly incremented. For example: >>>> >>>> In [30]: x = zeros(5) >>>> >>>> In [31]: idx = array([1,1,1,3,4]) >>>> >>>> In [32]: x[idx] += [2,4,8,10,30] >>>> >>>> In [33]: x >>>> Out[33]: array([ 0., 8., 0., 10., 30.]) >>>> >>>> I would intuitively expect the output to be array([0,14, 0,10,30]) since >>>> index 1 is incremented by 2+4+8=14, but instead it seems to only increment >>>> by 8. What is numpy actually doing here? >>>> >>>> The authors of Theano noticed this behavior a while ago so they python loop >>>> through the values in idx (this kind of calculation is necessary for >>>> calculating gradients), but this is a bit slow for my purposes, so I'd like >>>> to figure out how to get the behavior I expected, but faster. >>>> >>>> I'm also not sure how to navigate the numpy codebase, where would I look >>>> for >>>> the code responsible for this behavior? >>> >>> Strictly speaking, it isn't actually in the numpy codebase at all -- >>> what's happening is that the Python interpreter sees this code: >>> >>> x[idx] += vals >>> >>> and then it translates it into this code before running it: >>> >>> tmp = x.__getitem__(idx) >>> tmp = tmp.__iadd__(vals) >>> x.__setitem__(idx, tmp) >>> >>> So you can find the implementations of the ndarray methods >>> __getitem__, __iadd__, __setitem__ (they're called >>> array_subscript_nice, array_inplace_add, and array_ass_sub in the C >>> code), but there's no way to fix them so that this works the way you >>> want it to, because there's no way for __iadd__ to know that the >>> temporary values that it's working with are really duplicate copies of >>> "the same" value in the original array. >>> >>> It would be nice if numpy had some sort of standard API for doing what >>> you want, but not sure what a good API would look like, and someone >>> would have to implement it. >> >> This operation is also heavily used for the finite element assembling, and a >> similar question has been raised already several times (e.g. >> http://old.nabble.com/How-to-assemble-large-sparse-matrices-effectively-td33833855.html). >> So why not adding a function np.assemble()? > > I read that message, but I don't see what it has to do with this > discussion? It seemed to be about fast ways to assign dense matrices > into sparse matrices, not fast ways of applying in-place arithmetic to > specific spots in a dense matrix. Yes (in that thread), but it applies also adding/assembling vectors into a global vector - this is just x[idx] += vals. I linked that discussion as that was recent enough for me to recall it, but there were other. Anyway, my point was that a having a function with the "adding" semantics in NumPy would be handy. r. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Incrementing with advanced indexing: why don't repeated indexes repeatedly increment?
On Wed, Jun 6, 2012 at 4:52 PM, Robert Cimrman wrote: > Yes (in that thread), but it applies also adding/assembling vectors into a > global vector - this is just x[idx] += vals. I linked that discussion as that > was recent enough for me to recall it, but there were other. > > Anyway, my point was that a having a function with the "adding" semantics in > NumPy would be handy. x += numpy.bincount(idx, vals, minlength=len(x)) -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Incrementing with advanced indexing: why don't repeated indexes repeatedly increment?
On 06/06/2012 06:35 PM, Robert Kern wrote: > On Wed, Jun 6, 2012 at 4:52 PM, Robert Cimrman wrote: > >> Yes (in that thread), but it applies also adding/assembling vectors into a >> global vector - this is just x[idx] += vals. I linked that discussion as that >> was recent enough for me to recall it, but there were other. >> >> Anyway, my point was that a having a function with the "adding" semantics in >> NumPy would be handy. > > x += numpy.bincount(idx, vals, minlength=len(x)) > Nice! Looking at the C source, it seems it should be pretty efficient for this task. r. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Are "min", "max" documented for scalars?
On Thu, Jun 7, 2012 at 2:25 PM, Edward C. Jones wrote: > Silly mistakes. > > If a and b are Python ints, Python floats, or non-complex > numpy.number's, "max" returns, unchanged, the largrt of the two > objects. There is no coercion to a common type. This useful behavior > needs to be documented. Suggestions for improving the standard Python documentation (which documents these functions) can be sent here: http://docs.python.org/bugs.html#documentation-bugs -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] possible enhancement to getitem?
On Thu, Jun 7, 2012 at 7:55 PM, Neal Becker wrote: > In [3]: u = np.arange(10) > > In [4]: u > Out[4]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > > In [5]: u[-2:] > Out[5]: array([8, 9]) > > In [6]: u[-2:2] > Out[6]: array([], dtype=int64) > > I would argue for consistency it would be desirable for this to return > > [8, 9, 0, 1] Unfortunately, this would be inconsistent with Python semantics: [~] |1> u = range(10) [~] |2> u[-2:2] [] -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] trivial question?
On Wed, Jun 20, 2012 at 3:58 PM, Neal Becker wrote: > Maybe I'm being slow, but is there any convenient function to calculate, > for 2 vectors: > > \sum_i \sum_j x_i y_j > > (I had a matrix once, but it vanished without a trace) np.multiply.outer(x, y).sum() -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Good way to develop numpy as popular choice!
On Thu, Jun 21, 2012 at 3:59 PM, bob tnur wrote: > Hi all numpy fun;) > This question is already posted in stackoverflow by some people, I am just > thinking that numpy python will do this with trick;) I guess numpy will be > every ones choice as its popularity increases. The question is herein: > http://stackoverflow.com/questions/10074270/how-can-i-find-the-minimum-number-of-lines-needed-to-cover-all-the-zeros-in-a-2 My "numpy solution" for this is just $ pip install munkres http://pypi.python.org/pypi/munkres -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Good way to develop numpy as popular choice!
On Thu, Jun 21, 2012 at 7:33 PM, eat wrote: > Heh, > > On Thu, Jun 21, 2012 at 6:03 PM, Robert Kern wrote: >> >> On Thu, Jun 21, 2012 at 3:59 PM, bob tnur wrote: >> > Hi all numpy fun;) >> > This question is already posted in stackoverflow by some people, I am >> > just >> > thinking that numpy python will do this with trick;) I guess numpy will >> > be >> > every ones choice as its popularity increases. The question is herein: >> > >> > http://stackoverflow.com/questions/10074270/how-can-i-find-the-minimum-number-of-lines-needed-to-cover-all-the-zeros-in-a-2 >> >> My "numpy solution" for this is just >> >> $ pip install munkres > > munkres seems to be a pure python implementation ;-). Oops! I could have sworn that I once tried one named munkres that used numpy. But that was several years ago. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Bug in pickling an ndarray?
On Sat, Jun 30, 2012 at 11:33 PM, Daniel Hyams wrote: > Hmmm, I wouldn't think that it is correct behavior; I would think that *any* > ndarray arising from pickling would have its .base attribute set to None. > If not, then who is really the one that owns the data? > > It was my understanding that .base should hold a reference to another > ndarray that the data is really coming from, or it's None. It certainly > shouldn't be some random string, should it? It can be any object that will keep the data memory alive while the object is kept alive. It does not have to be an ndarray. In this case, the numpy unpickling constructor takes the string object that the underlying pickling machinery has just created and views its memory directly. In order to keep Python from freeing that memory, the string object needs to be kept alive via a reference, so it gets assigned to the .base. > And yes, it is causing a problem for me, which is why I noticed it. In my > application, ndarrays can come from various sources, pickling being one of > them. Later in the app, I was wanting to resize the array, which you cannot > do if the data is not really owned by that array... You also can't resize an array if any *other* array has a view on that array too, so checking for ownership isn't going to help. .resize() will raise an exception if it can't do this; it's better to just attempt it and catch the exception than to look before you leap. > I had explicit check for > myarray.base==None, which it is not when I get the ndarray from a pickle. That is not the way to check if an ndarray owns its data. Instead, check a.flags['OWNDATA'] -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] "import numpy" performance
On Mon, Jul 2, 2012 at 9:43 PM, Benjamin Root wrote:
>
> On Mon, Jul 2, 2012 at 4:34 PM, Nathaniel Smith wrote:
>> I think this ship has sailed, but it'd be worth looking into lazy
>> importing, where 'numpy.fft' isn't actually imported until someone
>> starts using it. There are a bunch of libraries that do this, and one
>> would have to fiddle to get compatibility with all the different
>> python versions and make sure you're not killing performance (might
>> have to be in C) but something along the lines of
>>
>> class _FFTModule(object):
>> def __getattribute__(self, name):
>> mod = importlib.import_module("numpy.fft")
>> _FFTModule.__getattribute__ = mod.__getattribute__
>> return getattr(mod, name)
>> fft = _FFTModule()
>
> Not sure how this would impact projects like ipython that does
> tab-completion support, but I know that that would drive me nuts in my basic
> tab-completion setup I have for my regular python terminal. Of course, in
> the grand scheme of things, that really isn't all that important, I don't
> think.
We used to do it for scipy. It did interfere with tab completion. It
did drive many people nuts.
--
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Type specific sorts: objects, structured arrays, and all that.
On Tue, Jul 10, 2012 at 4:32 AM, Charles R Harris wrote: > Hi All, > > I've been adding type specific sorts for object and structured arrays. It > seems that datetime64 and timedelta64 are also not supported. Is there any > reason why those types should not be sorted as int64? You need special handling for NaTs to be consistent with how we deal with NaNs in floats. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] use slicing as argument values?
On Thu, Jul 12, 2012 at 9:46 PM, Chao YUE wrote: > Hi Ben, > > it helps a lot. I am nearly finishing a function in a way I think pythonic. > Just one more question, I have: > > In [24]: b=np.arange(1,11) > > In [25]: b > Out[25]: array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]) > > In [26]: b[slice(1)] > Out[26]: array([1]) > > In [27]: b[slice(4)] > Out[27]: array([1, 2, 3, 4]) > > In [28]: b[slice(None,4)] > Out[28]: array([1, 2, 3, 4]) > > so slice(4) is actually slice(None,4), how can I exactly want retrieve a[4] > using slice object? You don't. You use 4. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] use slicing as argument values?
On Thu, Jul 12, 2012 at 10:32 PM, Chao YUE wrote: > Thanks all for the discussion. Actually I am trying to use something like > numpy ndarray indexing in the function. Like when I call: > > func(a,'1:3,:,2:4'), it knows I want to retrieve a[1:3,:,2:4], and > func(a,'1:3,:,4') for a[1:3,:,4] ect. > I am very close now. [~] |1> from numpy import index_exp [~] |2> index_exp[1:3,:,2:4] (slice(1, 3, None), slice(None, None, None), slice(2, 4, None)) -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] building a numpy egg for centos
On Tue, Jul 24, 2012 at 3:00 PM, Jason Stelzer wrote: > I'm building numpy 1.6.2 for python 2.5 on centos 5.8. > > I ran into a problem where bdist_egg was not working. It seems there's > a minor bug in numpy/distutils/core.py > > Under python 2.5 the check for setuptools does not work, so the bdist > target for eggs is not available. > > I've attached a patch that works around the issue for me. It is my > understanding that python 2.5 should still be a valid target for > building this release. If not, ignore this. This was an explicit design choice. numpy.distutils will never import setuptools for you even if you have it installed. It will simply integrate with it if you have run the setup.py script from something that has setuptools imported, like the setupegg.py script. What exactly is failing to work under Python 2.5 on your system? "python setup.py bdist_egg" should never work, but "python setupegg.py bdist_egg" should. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Synonym standards
On Fri, Jul 27, 2012 at 12:05 AM, Colin J. Williams wrote: > On 26/07/2012 4:57 PM, Benjamin Root wrote: > > > On Thu, Jul 26, 2012 at 4:45 PM, Colin J. Williams wrote: >> >> It seems that these standards have been adopted, which is good: >> >> The following import conventions are used throughout the NumPy source and >> documentation: >> >> import numpy as np >> import matplotlib as mpl >> import matplotlib.pyplot as plt >> >> Source: >> https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt >> >> Is there some similar standard for PyLab? >> >> Thanks, >> >> Colin W. >> > > > Colin, > > Typically, with pylab mode of matplotlib, you do: > > from pylab import * > > This is essentially equivalent to: > > from numpy import * > from matplotlib.pyplot import * > > Note that the pylab "module" is actually a part of matplotlib and is a > shortcut to provide an environment that is very familiar to Matlab users. > Converts are then encouraged to use the imports you mentioned in order to > properly utilize python namespaces. > > I hope that helps! > Ben Root > > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > Thanks Ben, > > I would prefer not to use: from xxx import *, > > because of the name pollution. > > The name convention that I copied above facilitates avoiding the pollution. > > In the same spirit, I've used: > import pylab as plb But in that same spirit, using np and plt separately is preferred. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] bug in numpy.where?
On Mon, Jul 30, 2012 at 2:30 PM, Phil Hodge wrote: > On 07/27/2012 03:58 PM, Andreas Mueller wrote: >> Hi Everybody. >> The bug is that no error is raised, right? >> The docs say >> >> where(condition, [x, y]) >> >> x, y : array_like, optional >> Values from which to choose. `x` and `y` need to have the same >> shape as `condition` >> >> In the example you gave, x was a scalar. > > net.max() returns an array: > > >>> print type(net.max()) > No, that's a scalar. The type would be numpy.ndarray if it were an array. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] load of custom .npy file fails with numpy 2.0.0
On Thu, Aug 2, 2012 at 8:46 PM, Geoffrey Irving wrote:
> Hello,
>
> The attached .npy file was written from custom C++ code. It loads
> fine in Numpy 1.6.2 with Python 2.6 installed through MacPorts, but
> fails on a different machine with Numpy 2.0.0 installed via Superpack:
>
> box:array% which python
> /usr/bin/python
> box:array% which python
> box:array% python
> Python 2.6.1 (r261:67515, Aug 2 2010, 20:10:18)
> [GCC 4.2.1 (Apple Inc. build 5646)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import numpy
>>>> numpy.load('blah.npy')
> Traceback (most recent call last):
> File "", line 1, in
> File
> "/Library/Python/2.6/site-packages/numpy-2.0.0.dev_b5cdaee_20110710-py2.6-macosx-10.6-universal.egg/numpy/lib/npyio.py",
> line 351, in load
> return format.read_array(fid)
> File
> "/Library/Python/2.6/site-packages/numpy-2.0.0.dev_b5cdaee_20110710-py2.6-macosx-10.6-universal.egg/numpy/lib/format.py",
> line 440, in read_array
> shape, fortran_order, dtype = read_array_header_1_0(fp)
> File
> "/Library/Python/2.6/site-packages/numpy-2.0.0.dev_b5cdaee_20110710-py2.6-macosx-10.6-universal.egg/numpy/lib/format.py",
> line 361, in read_array_header_1_0
> raise ValueError(msg % (d['descr'],))
> ValueError: descr is not a valid dtype descriptor: 'd8'
>>>> numpy.__version__
> '2.0.0.dev-b5cdaee'
>>>> numpy.__file__
> '/Library/Python/2.6/site-packages/numpy-2.0.0.dev_b5cdaee_20110710-py2.6-macosx-10.6-universal.egg/numpy/__init__.pyc'
>
> It seems Numpy 2.0.0 no longer accepts dtype('d8'):
>
>>>> dtype('d8')
> Traceback (most recent call last):
> File "", line 1, in
> TypeError: data type "d8" not understood
>
> Was that intentional? An API change isn't too much of a problem, but
> it's unfortunate if old data files are no longer easily readable.
As far as I can tell, numpy has never described an array using 'd8'.
That would be a really old compatibility typecode from Numeric, if I
remember correctly. The intention of the NPY format standard was that
it would accept what numpy spits out for the descr, not that it would
accept absolutely anything that numpy.dtype() can consume, even
deprecated aliases (though I will admit that that is almost what the
NEP says). In particular, endianness really should be included or else
your files will be misread on big-endian machines.
My suspicion is that only your code has ever made .npy files with this
descr. I feel your pain, Geoff, and I apologize that my lax
specification led you down this path, but I think you need to fix your
code anyways.
--
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] load of custom .npy file fails with numpy 2.0.0
On Thu, Aug 2, 2012 at 11:41 PM, Geoffrey Irving wrote:
> On Thu, Aug 2, 2012 at 1:26 PM, Robert Kern wrote:
>> On Thu, Aug 2, 2012 at 8:46 PM, Geoffrey Irving wrote:
>>> Hello,
>>>
>>> The attached .npy file was written from custom C++ code. It loads
>>> fine in Numpy 1.6.2 with Python 2.6 installed through MacPorts, but
>>> fails on a different machine with Numpy 2.0.0 installed via Superpack:
>>>
>>> box:array% which python
>>> /usr/bin/python
>>> box:array% which python
>>> box:array% python
>>> Python 2.6.1 (r261:67515, Aug 2 2010, 20:10:18)
>>> [GCC 4.2.1 (Apple Inc. build 5646)] on darwin
>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>> import numpy
>>>>>> numpy.load('blah.npy')
>>> Traceback (most recent call last):
>>> File "", line 1, in
>>> File
>>> "/Library/Python/2.6/site-packages/numpy-2.0.0.dev_b5cdaee_20110710-py2.6-macosx-10.6-universal.egg/numpy/lib/npyio.py",
>>> line 351, in load
>>> return format.read_array(fid)
>>> File
>>> "/Library/Python/2.6/site-packages/numpy-2.0.0.dev_b5cdaee_20110710-py2.6-macosx-10.6-universal.egg/numpy/lib/format.py",
>>> line 440, in read_array
>>> shape, fortran_order, dtype = read_array_header_1_0(fp)
>>> File
>>> "/Library/Python/2.6/site-packages/numpy-2.0.0.dev_b5cdaee_20110710-py2.6-macosx-10.6-universal.egg/numpy/lib/format.py",
>>> line 361, in read_array_header_1_0
>>> raise ValueError(msg % (d['descr'],))
>>> ValueError: descr is not a valid dtype descriptor: 'd8'
>>>>>> numpy.__version__
>>> '2.0.0.dev-b5cdaee'
>>>>>> numpy.__file__
>>> '/Library/Python/2.6/site-packages/numpy-2.0.0.dev_b5cdaee_20110710-py2.6-macosx-10.6-universal.egg/numpy/__init__.pyc'
>>>
>>> It seems Numpy 2.0.0 no longer accepts dtype('d8'):
>>>
>>>>>> dtype('d8')
>>> Traceback (most recent call last):
>>> File "", line 1, in
>>> TypeError: data type "d8" not understood
>>>
>>> Was that intentional? An API change isn't too much of a problem, but
>>> it's unfortunate if old data files are no longer easily readable.
>>
>> As far as I can tell, numpy has never described an array using 'd8'.
>> That would be a really old compatibility typecode from Numeric, if I
>> remember correctly. The intention of the NPY format standard was that
>> it would accept what numpy spits out for the descr, not that it would
>> accept absolutely anything that numpy.dtype() can consume, even
>> deprecated aliases (though I will admit that that is almost what the
>> NEP says). In particular, endianness really should be included or else
>> your files will be misread on big-endian machines.
>>
>> My suspicion is that only your code has ever made .npy files with this
>> descr. I feel your pain, Geoff, and I apologize that my lax
>> specification led you down this path, but I think you need to fix your
>> code anyways.
>
> Sounds good. Both 1.6.2 and 2.0.0 write out ' I'll certainly add the '<' bit to signify endianness, but how should I
> go about determining the letter? My current code looks like
>
> // Get dtype info
> int bits;char letter;
> switch(type_num){
> #define CASE(T) case
> NPY_##T:bits=NPY_BITSOF_##T;letter=NPY_##T##LTR;break;
> #define NPY_BITSOF_BYTE 8
> #define NPY_BITSOF_UBYTE 8
> #define NPY_BITSOF_USHORT NPY_BITSOF_SHORT
> #define NPY_BITSOF_UINT NPY_BITSOF_INT
> #define NPY_BITSOF_ULONG NPY_BITSOF_LONG
> #define NPY_BITSOF_ULONGLONG NPY_BITSOF_LONGLONG
> CASE(BOOL)
> CASE(BYTE)
> CASE(UBYTE)
> CASE(SHORT)
> CASE(USHORT)
> CASE(INT)
> CASE(UINT)
> CASE(LONG)
> CASE(ULONG)
> CASE(LONGLONG)
> CASE(ULONGLONG)
> CASE(FLOAT)
> CASE(DOUBLE)
> CASE(LONGDOUBLE)
> #undef CASE
> default: throw ValueError("Unknown dtype");}
> int bytes = bits/8;
> ...
> len += sprintf(base+len,"{'descr': '%c%d', 'fortran_order': False,
> 'shape': (",letter,bytes);
>
> The code incorrectly assumes that the ...LTR constants are safe ways
> to describe dtypes. Is there a clean, correct way to do this that
> doesn't require special casing for each type? I can use numpy headers
> but can't call any numpy functions, since Python might not be
> initialized (e.g., if I'm writing out files through MPI IO collectives
> on a Cray).
These characters plus the byte-size and endian-ness:
/*
* These are for dtype 'kinds', not dtype 'typecodes'
* as the above are for.
*/
NPY_GENBOOLLTR ='b',
NPY_SIGNEDLTR = 'i',
NPY_UNSIGNEDLTR = 'u',
NPY_FLOATINGLTR = 'f',
NPY_COMPLEXLTR = 'c'
Less amenable to macro magic, certainly, but workable. To
double-check, see what numpy outputs for each of these cases.
--
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Licensing question
Those are not the original Fortran sources. The original Fortran sources are in the public domain as work done by a US federal employee. http://www.netlib.org/fftpack/ Never trust the license of any code on John Burkardt's site. Track it down to the original sources. On Monday, August 6, 2012, Sturla Molden wrote: > But the Fortran FFTPACK is GPL, or has the licence been changed? > > http://people.sc.fsu.edu/~jburkardt/f77_src/fftpack5.1/fftpack5.1.html > > Sturla > > Sendt fra min iPad > > Den 3. aug. 2012 kl. 07:52 skrev Travis Oliphant > > >: > > > This should be completely fine.The fftpack.h file indicates that > fftpack code came from Tela originally anyway and was translated from the > Fortran code FFTPACK. > > > > Good luck with your project. > > > > -Travis > > > > > > On Aug 2, 2012, at 3:44 PM, Damon McDougall wrote: > > > >> Hi, > >> > >> I have a question about the licence for NumPy's codebase. I am currently > >> writing a library and I'd like to release under some BSD-type licence. > >> Unfortunately, my choice to link against MIT's FFTW library (released > >> under the GPL) means that, in its current state, this is not possible. > >> I'm an avid NumPy user and thought to myself that, since NumPy's licence > >> is BSD, I'd be able to use some of the source code (with due credit, of > >> course) instead of FFTW. Is this possible? I mean, can I redistribute > >> *PART* of NumPy's codebase? Namely, the fftpack.c file? I was under the > >> impression that I could only redistribute BSD source code as a whole and > >> then I read the licence more carefully and it states that I can modify > >> the source to suit my needs. I consider 'redistributing a single file > >> and ignoring the other files' as a 'modification' under the BSD > >> definition, but maybe I'm thinking too wishfully here. > >> > >> Any information on this matter would be greatly appreciated since I am a > >> total code licence noob. > >> > >> Thank you. > >> > >> P.S. Yes, I know I could just release under the GPL, but I don't want to > >> turn people off of packaging my work into a useful product licensed > >> under BSD, or even make money from it. > >> > >> -- > >> Damon McDougall > >> http://damon-is-a-geek.com > >> B2.39 > >> Mathematics Institute > >> University of Warwick > >> Coventry > >> West Midlands > >> CV4 7AL > >> United Kingdom > >> ___ > >> NumPy-Discussion mailing list > >> NumPy-Discussion@scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > ___ > > NumPy-Discussion mailing list > > NumPy-Discussion@scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Licensing question
On Wed, Aug 8, 2012 at 10:34 AM, David Cournapeau wrote: > On Wed, Aug 8, 2012 at 12:55 AM, Nathaniel Smith wrote: >> On Mon, Aug 6, 2012 at 8:31 PM, Robert Kern wrote: >>> Those are not the original Fortran sources. The original Fortran sources are >>> in the public domain as work done by a US federal employee. >>> >>> http://www.netlib.org/fftpack/ >>> >>> Never trust the license of any code on John Burkardt's site. Track it down >>> to the original sources. >> >> Taken together, what those websites seem to be claiming is that you >> have a choice of buggy BSD code or fixed GPL code? I assume someone >> has already taken the appropriate measures for numpy, but it seems >> like an unfortunate situation... > > If the code on John Burkardt website is based on the netlib codebase, > he is not entitled to make it GPL unless he is the sole copyright > holder of the original code. He can certainly incorporate the public domain code and rerelease it under whatever restrictions he likes, especially if he adds to it, which appears to be the case. The original sources are legitimately public domain, not just released under a liberal copyright license. He can't "remove" the original code from the public domain, but that's not what he claims to have done. > I think the 'real' solution is to have a separate package linking to > FFTW for people with 'advanced' needs for FFT. None of the other > library I have looked at so far are usable, fast and precise enough > when you go far from the simple case of double precision and 'well > factored' size. http://pypi.python.org/pypi/pyFFTW -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] broadcasting question
On Thu, Aug 30, 2012 at 12:49 PM, Neal Becker wrote: > I think this should be simple, but I'm drawing a blank > > I have 2 2d matrixes > > Matrix A has indexes (i, symbol) > Matrix B has indexes (state, symbol) > > I combined them into a 3d matrix: > > C = A[:,newaxis,:] + B[newaxis,:,:] > where C has indexes (i, state, symbol) > > That works fine. > > Now suppose I want to omit B (for debug), like: > > C = A[:,newaxis,:] > > In other words, all I want is to add a dimension into A and force it to > broadcast along that axis. How do I do that? C, dummy = numpy.broadcast_arrays(A[:,newaxis,:], numpy.empty([1,state,1])) -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy.distutils, lapack, and cython
On Fri, Sep 28, 2012 at 9:45 AM, Nathaniel Smith wrote: > Hi all, > > I've gotten a pull request for scikits-sparse to switch it to using > numpy.distutils: > https://github.com/njsmith/scikits-sparse/pull/2 > > Overall this seems fair enough, finding libraries is a pain and > numpy.distutils has that knowledge. > > 1) What's the proper way to find lapack using numpy.distutils? The > patch tries lapack_mkl_info and lapack_info, but this is 2 of the 6 > variants that my numpy.distutils.system_info contains. Really what I > probably want is a way to ask numpy how it was built (including any > custom paths the user used) and to default to doing the same? Is that > possible? You can get some of that information from the generated numpy.__config__ module (see its show() and get_info() functions). I'm not sure if that includes custom paths that were hacked in by the user by editing the setup.py files, but I think it includes custom paths specified in the site.cfg that the builder used. Of course, if one installed numpy using a binary built on another machine, it is possible for that information to be not applicable to the current build. I believe that you want to use 'lapack_opt' as the most generic optimized LAPACK build information. That should dispatch to the vendor-specific ones if they are present, I think. It's what numpy.linalg and scipy.linalg do. I have rather blissfully forgotten such details, so you may want to do some digging of your own. > 2) Is there a better way to build Cython files than this weird > monkey-patching thing they propose? (It's still better than the horror > that setuptools/distribute require, but I guess I have higher > expectations...) Sadly, probably not. numpy.distutils is not much less horrifying than setuptools. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] distutils & C++ & Fortran
On Tue, Oct 16, 2012 at 9:56 PM, Pauli Virtanen wrote: > Charles R Harris gmail.com> writes: >> On Sun, Oct 14, 2012 at 11:53 AM, Pauli Virtanen iki.fi> wrote: >> Hi, >> I'd like to link both C++ and Fortran code into a single >> Python extension, using numpy.distutils (for scipy.special). >> But it seems the distutils-based tools actually cannot do this? >> Does someone know if there's a way to work around this? >> >> Looks like the answer is "no" ;) Is this a linking problem? > > Pretty much. Knowing the Fortran or C++ runtime names / link flags > on different compilers would be enough, but apparently this > information is not written down anywhere in the build system. With some platforms/compilers, we use the compiler itself as the linker, because it knows the right runtime library and link flags for the language, often because they are more variable with respect to specific compiler versions. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] matrix norm
On Mon, Oct 22, 2012 at 9:03 PM, Pauli Virtanen wrote: > Jason Grout creativetrax.com> writes: > [clip] >> I think we've established that the other software mentioned does indeed >> use the spectral norm by default. I'm still curious: what was the >> reason for breaking with the norm (pun intended :)? Any chance that in >> a (probably far distant) future release, the norm default could be >> changed to conform with matlab/octave/mathematica's view of the world? >> It's not a huge deal to me now that I know to watch out for it, but it >> did just bite me and a student a few times. > > The trail leads to here: > http://projects.scipy.org/numpy/attachment/ticket/36/numpy-6-norm-change- > default.diff > > Seems like the chances of learning the reason why this change was done > are pretty slim. http://mail.scipy.org/pipermail/numpy-discussion/2006-March/019194.html -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Is there a way to reset an accumulate function?
On Tue, Oct 23, 2012 at 6:11 PM, Cera, Tim wrote: > I have an array that is peppered throughout in random spots with 'nan'. I > would like to use 'cumsum', but I want it to reset the accumulation to 0 > whenever a 'nan' is encountered. Is there a way to do this? Aside from a > loop - which is what I am going to setup here in a moment. How about this? def nancumsum(x): nans = np.isnan(x) x = np.array(x) x[nans] = 0 reset_idx = np.zeros(len(x), dtype=int) reset_idx[nans] = np.arange(len(x))[nans] reset_idx = np.maximum.accumulate(reset_idx) cumsum = np.cumsum(x) cumsum = cumsum - cumsum[reset_idx] return cumsum -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] RE : 1.7.0 release
Congratulation. De : numpy-discussion-boun...@scipy.org [numpy-discussion-boun...@scipy.org] de la part de Ondřej Čertík [ondrej.cer...@gmail.com] Date d'envoi : 12 novembre 2012 17:27 À : Discussion of Numerical Python Objet : Re: [Numpy-discussion] 1.7.0 release Hi, On Mon, Nov 5, 2012 at 11:33 PM, Travis Oliphant wrote: > Hey all, > > Ondrej has been tied up finishing his PhD for the past several weeks. He is > defending his work shortly and should be available to continue to help with > the 1.7.0 release around the first of December.He and I have been in > contact during this process, and I've been helping where I can. > Fortunately, other NumPy developers have been active closing tickets and > reviewing pull requests which has helped the process substantially. > > The release has taken us longer than we expected, but I'm really glad that > we've received the bug-reports and issues that we have seen because it will > help the 1.7.0 release be a more stable series. Also, the merging of the > Trac issues with Git has exposed over-looked problems as well and will > hopefully encourage more Git-focused participation by users. > > We are targeting getting the final release of 1.7.0 out by mid December > (based on Ondrej's availability). But, I would like to find out which > issues are seen as blockers by people on this list. I think most of the > issues that I had as blockers have been resolved.If there are no more > remaining blockers, then we may be able to accelerate the final release of > 1.7.0 to just after Thanksgiving. I successfully defended my Ph.D. thesis last Thursday, I just need to do some changes to it and submit it and I am done. So I started to work on the release again (my apologies that it got delayed, I had to devote my full attention to finishing my school first). Here is a list of issues that need to be fixed before the release: https://github.com/numpy/numpy/issues?milestone=3&state=open If anyone wants to help, we just need to get through them and submit a PR for each, or close it if it doesn't apply anymore. This is what I am doing now. Ondrej ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy.power vs pylab.power
On Thu, Nov 15, 2012 at 4:02 PM, Will Furnass wrote: > On my machine these are rather confusingly different functions, with the > latter corresponding to numpy.random.power. I appreciate that pylab > imports everything from both the numpy and numpy.random modules but > wouldn't it make sense if pylab.power were the frequently used power > function rather than a means for sampling from the power distribution? Matplotlib may be discussed over here: https://lists.sourceforge.net/lists/listinfo/matplotlib-users -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Simple Loadtxt question
I have a file with thousands of lines like this:
Signal was returned in 204 microseconds
Signal was returned in 184 microseconds
Signal was returned in 199 microseconds
Signal was returned in 4274 microseconds
Signal was returned in 202 microseconds
Signal was returned in 189 microseconds
I try to read it like this:
data = np.loadtxt('dummy.data', dtype={'names':('label','times','musec'),
'fmts':('|S23','i8','|S13')})
It fails, I think, because it wants a string format and field for each of the
words 'Signal' 'was' 'returned' etc.
Can I make it treat that whole string before the number as one string, one
field? All I really care about is the numbers anyway.
Any advice appreciated.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] install numpy 1.6.2 .dmg on macosx 10.7, check for python 2.7
On Thu, Nov 29, 2012 at 12:52 PM, denis wrote: > Trying to install numpy 1.6.2 on a mac osx 10.7.4 from this .dmg > 9323135 numpy-1.6.2-py2.7-python.org-macosx10.3.dmg > gives > "numpy 1.6.2 can't be installed on this disk. > numpy requires python.org Python 2.7 to install." > > But python 2.7.3 *is* installed from python.org > 18761950 python-2.7.3-macosx10.6.dmg > and /usr/bin/python is linked as described in > http://wolfpaulus.com/journal/mac/installing_python_osx Ouch! I'm sorry, but these are wildly *dangerous* instructions. Please disregard them. The author is reckless. You can't undo the damage without reinstalling OS X, but there is a good chance you will not run into the parts of OS X that depended on the /System Python being unmolested. However, you do need to reinstall the python.org Python framework again. Just don't mv it from where it gets installed. Then the numpy-1.6.2-py2.7-python.org-macosx10.3.dmg will recognize it. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np.seterr doesn't work for masked array?
On Fri, Dec 14, 2012 at 1:57 PM, Chao YUE wrote: > Dear all, > > I tried to capture the zero divide error when I divide a masked array by > another. It seems that np.seterr is not working for masked array? > when I do np.divide on two masked array, it directly put the zero divides > part as being masked. The np.seterr works if the two arrays for dividing are > not masked arrays. > could anyone explain? thanks!! numpy.ma uses np.seterr(divide='ignore', invalid='ignore') for most of its operations, so it is overriding your settings. This is usually desirable since many of the masked values will be trip these errors spuriously even though they will be masked out in the result. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np.seterr doesn't work for masked array?
On Fri, Dec 14, 2012 at 2:40 PM, Chao YUE wrote: > Thanks. You mean actually when numpy handle masked array, it will first > treat all the base data, and then apply the mask after the treatment. > and normally the base data of maksed elements will very likely to intrigure > these errors, and you will see a lot errory warning or print in the process, > and will make it impossible to see the error information your want to see > for those elements that are not masked but still can intriguer the error you > would like to see or check. I didn't realize this. It's a good to overwrite > the error setting. Precisely. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
