Re: [Numpy-discussion] Getting Callbacks with arrays to work
Hi, Thanks all works now! The implicit none only didn't work when defining dv as a function now its a subroutine it seems to work. Regards Jon On 12/01/2010 13:44, Pearu Peterson wrote: Hi, The problem is that f2py does not support callbacks that return arrays. There is easy workaround to that: provide returnable arrays as arguments to callback functions. Using your example: SUBROUTINE CallbackTest(dv,v0,Vout,N) IMPLICIT NONE !F2PY intent( hide ):: N INTEGER:: N, ic EXTERNAL:: dv DOUBLE PRECISION, DIMENSION( N ), INTENT(IN):: v0 DOUBLE PRECISION, DIMENSION( N ), INTENT(OUT):: Vout DOUBLE PRECISION, DIMENSION( N ):: Vnow DOUBLE PRECISION, DIMENSION( N ):: temp Vnow = v0 !f2py intent (out) temp call dv(temp, Vnow, N) DO ic = 1, N Vout( ic ) = temp(ic) END DO END SUBROUTINE CallbackTest $ f2py -c test.f90 -m t --fcompiler=gnu95 from numpy import * from t import * arr = array([2.0, 4.0, 6.0, 8.0]) def dV(v): print 'in Python dV: V is: ',v ret = v.copy() ret[1] = 100.0 return ret ... output = callbacktest(dV, arr) in Python dV: V is: [ 2. 4. 6. 8.] output array([ 2., 100.,6.,8.]) What problems do you have with implicit none? It works fine here. Check the format of your source code, if it is free then use `.f90` extension, not `.f`. HTH, Pearu Jon Moore wrote: Hi, I'm trying to build a differential equation integrator and later a stochastic differential equation integrator. I'm having trouble getting f2py to work where the callback itself receives an array from the Fortran routine does some work on it and then passes an array back. For the stoachastic integrator I'll need 2 callbacks both dealing with arrays. The idea is the code that never changes (ie the integrator) will be in Fortran and the code that changes (ie the callbacks defining differential equations) will be different for each problem. To test the idea I've written basic code which should pass an array back and forth between Python and Fortran if it works right. Here is some code which doesn't work properly:- SUBROUTINE CallbackTest(dv,v0,Vout,N) !IMPLICIT NONE cF2PY intent( hide ):: N INTEGER:: N, ic EXTERNAL:: dv DOUBLE PRECISION, DIMENSION( N ), INTENT(IN):: v0 DOUBLE PRECISION, DIMENSION( N ), INTENT(OUT):: Vout DOUBLE PRECISION, DIMENSION( N ):: Vnow DOUBLE PRECISION, DIMENSION( N ):: temp Vnow = v0 temp = dv(Vnow, N) DO ic = 1, N Vout( ic ) = temp(ic) END DO END SUBROUTINE CallbackTest When I test it with this python code I find the code just replicates the first term of the array! from numpy import * import callback as c def dV(v): print 'in Python dV: V is: ',v return v.copy() arr = array([2.0, 4.0, 6.0, 8.0]) print 'Arr is: ', arr output = c.CallbackTest(dV, arr) print 'Out is: ', output Arr is: [ 2. 4. 6. 8.] in Python dV: V is: [ 2. 4. 6. 8.] Out is: [ 2. 2. 2. 2.] Any ideas how I should do this, and also how do I get the code to work with implicit none not commented out? Thanks Jon ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] wrong casting of augmented assignment statements
I've written a self-contained example that shows that numpy indeed
tries to call the __float__ method.
What is buggy is what happens if calling the __float__ method raises
an Exception.
Then numpy assumes (in this case wrongly) that the object should be
casted to the neutral element.
I'd guess that the __float__ method is called somewhere in a try:
statement and if an exception is raised it is casted to the neutral
element.
I've tried to locate the corresponding code in the numpy sources but I
got lost. Could someone be so kind and point me to it?
start code --
import numpy
print 'numpy.__version__ = ',numpy.__version__
class ad1:
def __init__(self,x):
self.x = x
def __mul__(self,other):
if not isinstance(other, self.__class__):
return self.__class__(self.x * other)
return self.__class__(self.x * other.x)
def __rmul__(self,other):
return self * other
def __float__(self):
raise Exception('this is not possible')
def __str__(self):
return str(self.x)
print '\nThis example yields buggy behavior:'
x1 = numpy.array([ad1(1.), ad1(2.), ad1(3.)])
y1 = numpy.random.rand(3)
print 'y1= ',y1
print 'x1= ',x1
z1 = x1 * y1
y1 *= x1# this should call the __float__ method of ad1
which would raise an Exception
print 'z1=x1*y1',z1
print 'y1*=x1 ',y1
class ad2:
def __init__(self,x):
self.x = x
def __mul__(self,other):
if not isinstance(other, self.__class__):
return self.__class__(self.x * other)
return self.__class__(self.x * other.x)
def __rmul__(self,other):
return self * other
def __float__(self):
return float(self.x)
def __str__(self):
return str(self.x)
print '\nThis example works fine:'
x2 = numpy.array([ad2(1.), ad2(2.), ad2(3.)])
y2 = numpy.random.rand(3)
print 'y2= ',y2
print 'x2= ',x2
z2 = x2 * y2
y2 *= x2# this should call the __float__ method of ad1
which would raise an Exception
print 'z2=x2*y2',z2
print 'y2*=x2 ',y2
end code --
output -
wal...@wronski$ python
wrong_casting_object_to_float_of_augmented_assignment_statements.py
numpy.__version__ = 1.3.0
This example yields buggy behavior:
y1= [ 0.15322371 0.47915903 0.81153995]
x1= [1.0 2.0 3.0]
z1=x1*y1 [0.153223711127 0.958318053803 2.43461983729]
y1*=x1 [ 0.15322371 0.47915903 0.81153995]
This example works fine:
y2= [ 0.49377037 0.60908423 0.79772095]
x2= [1.0 2.0 3.0]
z2=x2*y2 [0.493770370747 1.21816846399 2.39316283707]
y2*=x2 [ 0.49377037 1.21816846 2.39316284]
end output -
On Tue, Jan 12, 2010 at 7:38 PM, Robert Kern robert.k...@gmail.com wrote:
On Tue, Jan 12, 2010 at 12:31, Sebastian Walter
sebastian.wal...@gmail.com wrote:
On Tue, Jan 12, 2010 at 7:09 PM, Robert Kern robert.k...@gmail.com wrote:
On Tue, Jan 12, 2010 at 12:05, Sebastian Walter
sebastian.wal...@gmail.com wrote:
Hello,
I have a question about the augmented assignment statements *=, +=, etc.
Apparently, the casting of types is not working correctly. Is this
known resp. intended behavior of numpy?
Augmented assignment modifies numpy arrays in-place, so the usual
casting rules for assignment into an array apply. Namely, the array
being assigned into keeps its dtype.
what are the usual casting rules?
For assignment into an array, the array keeps its dtype and the data
being assigned into it will be cast to that dtype.
How does numpy know how to cast an object to a float?
For a general object, numpy will call its __float__ method.
If you do not want in-place modification, do not use augmented assignment.
Normally, I'd be perfectly fine with that.
However, this particular problem occurs when you try to automatically
differentiate an algorithm by using an Algorithmic Differentiation
(AD) tool.
E.g. given a function
x = numpy.ones(2)
def f(x):
a = numpy.ones(2)
a *= x
return numpy.sum(a)
one would use an AD tool as follows:
x = numpy.array([adouble(1.), adouble(1.)])
y = f(x)
but since the casting from object to float is not possible the
computed gradient \nabla_x f(x) will be wrong.
Sorry, but that's just a limitation of the AD approach. There are all
kinds of numpy constructions that AD can't handle.
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth.
-- Umberto Eco
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Matrix vs array in ma.minimum
Hi, I encountered a problem in matlab which boils down to a surprising behavior of np.ma.minimum: x = np.random.randn(2, 3) mx = np.matrix(x) np.ma.minimum(x) # smallest item of x ret = np.ma.minimum(mx) # flattened version of mx, i.e. ret == mx.flatten() Is this expected ? cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] dtype.isbuiltin changed by .newbyteorder
Hi,
Over on the scipy list, someone pointed out an oddness in the output
of the matlab reader, which revealed this - to me - unexpected
behavior in numpy:
In [20]: dt = np.dtype('f8')
In [21]: dt.isbuiltin
Out[21]: 1
In [22]: ndt = dt.newbyteorder('')
In [23]: ndt.isbuiltin
Out[23]: 0
I was expecting the 'isbuiltin' attribute to be the same (1) after
byte swapping.Does that seem reasonable to y'all? Then, is this a
bug?
Thanks a lot,
Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] dtype.isbuiltin changed by .newbyteorder
On Thu, Jan 14, 2010 at 07:01, Matthew Brett matthew.br...@gmail.com wrote:
Hi,
Over on the scipy list, someone pointed out an oddness in the output
of the matlab reader, which revealed this - to me - unexpected
behavior in numpy:
In [20]: dt = np.dtype('f8')
In [21]: dt.isbuiltin
Out[21]: 1
In [22]: ndt = dt.newbyteorder('')
In [23]: ndt.isbuiltin
Out[23]: 0
I was expecting the 'isbuiltin' attribute to be the same (1) after
byte swapping. Does that seem reasonable to y'all? Then, is this a
bug?
It is at least undesirable. It may not be a bug per se as I don't
think that we guarantee that .isbuiltin is free from false negatives
(though we do guarantee that it is free from false positives). The
reason is that we would have to search the builtin dtypes for a match
every time we create a new dtype object, and that could be more
expensive than we care to do for *every* creation of a dtype object.
It is possible that we can have a cheaper heuristic (native byte order
and the standard typecodes) and that transformations like
.newbyteorder() can have just a teeny bit more intelligent logic about
how it transforms the .isbuiltin flag.
Just for clarity and future googling, the issue is that when a native
dtype has .newbyteorder() called on it to make a new dtype that has
the *same* native byte order, the .isbuiltin flag incorrectly states
that it is not builtin. Using .newbyteorder() to swap the byte order
to the non-native byte order should and does cause the resulting dtype
to not be considered builtin.
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth.
-- Umberto Eco
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] dtype.isbuiltin changed by .newbyteorder
Hi, It is at least undesirable. It may not be a bug per se as I don't think that we guarantee that .isbuiltin is free from false negatives (though we do guarantee that it is free from false positives). The reason is that we would have to search the builtin dtypes for a match every time we create a new dtype object, and that could be more expensive than we care to do for *every* creation of a dtype object. It is possible that we can have a cheaper heuristic (native byte order and the standard typecodes) and that transformations like .newbyteorder() can have just a teeny bit more intelligent logic about how it transforms the .isbuiltin flag. Thanks - that's very clear and helpful, and made me realize I didn't understand the builtin attribute. I suppose something like the following: output_dtype.isbuiltin = input_dtype.isbuiltin and new_byteorder == native would at least reduce the false negatives at little cost. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] hello~
Hi,One of my friends introduce a very good website to me: http://www.flsso.com/. All their products are new and original. They have many brands, such as Sony, HP, Apple, Nokia and so on. Now , they are promoting their products for the coustomers. So their prices are very competitive. By the way, they mainly sell iphones, laptops, tvs, playstation and so on.If you need these products, it will be a good choice.Regards! ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] comparison operators (e.g. ==) on array with dtype object do not work
Dear NumPy People,
First I want to apologize if I misbehaved on NumPy Trac by reopening the
closed ticket
http://projects.scipy.org/numpy/ticket/1362
but I still feel strongly that there is misunderstanding
and the bug/defect is valid. I would appreciate if someone would waste
more of his time to persuade me that I am wrong but please first read
till the end:
The issue, as originally reported, is demonstrated with:
,---
| python -c 'import numpy as N; print N.__version__; a=N.array([1,
(0,1)],dtype=object); print a==1; print a == (0,1), a[1] == (0,1)'
| 1.5.0.dev
| [ True False]
| [False False] True
`---
whenever I expected the last line to be
[False True] True
charris (thanks for all the efforts to enlighten me) summarized it as
the result was correct given that the tuple (0,1) was converted to an
object array with elements 0 and 1. It is *not* converted to an array
containing a tuple.
and I was trying to argue that it is not the case in my example. It is
the case in charris's example though whenever both elements are of
the same length, or there is just a single tuple, i.e.
,---
| In [1]: array((0,1), dtype=object)
| Out[1]: array([0, 1], dtype=object)
|
| In [2]: array((0,1), dtype=object).shape
| Out[2]: (2,)
`---
There I would not expect my comparison to be valid indeed. But lets see what
happens in my case:
,---
| In [2]: array([1, (0,1)],dtype=object)
| Out[2]: array([1, (0, 1)], dtype=object)
|
| *In [3]: array([1, (0,1)],dtype=object).shape
| Out[3]: (2,)
|
| *In [4]: array([1, (0,1)],dtype=object)[1].shape
| ---
| AttributeErrorTraceback (most recent call
| last)
|
| /home/yoh/proj/ipython console in module()
|
| AttributeError: 'tuple' object has no attribute 'shape'
`---
So, as far as I see it, the array does contain an object of type tuple,
which does not get correctly compared upon __eq__ operation. Am I
wrong? Or does numpy internally somehow does convert 1st item (ie
tuple) into an array, but casts it back to tuple upon __repr__ or
__getitem__?
Thanks in advance for feedback
On Thu, 14 Jan 2010, NumPy Trac wrote:
#1362: comparison operators (e.g. ==) on array with dtype object do not work
-+--
Reporter: yarikoptic | Owner: somebody
Type: defect | Status: closed
Priority: normal | Milestone:
Component: Other | Version:
Resolution: invalid |Keywords:
-+--
Changes (by charris):
* status: reopened = closed
* resolution: = invalid
Old description:
You can see this better with the '*' operator:
{{{
In [8]: a * (0,2)
Out[8]: array([0, (0, 1, 0, 1)], dtype=object)
}}}
Note how the tuple is concatenated with itself. The reason the original
instance of a worked was that 1 and (0,1) are of different lengths, so
the decent into the nested sequence types stopped at one level and a
tuple is one of the elements. When you do something like ((0,1),(0,1))
the decent goes down two levels and you end up with a 2x2 array of
integer objects. The rule of thumb for object arrays is that you get an
array with as many indices as possible. Which is why object arrays are
hard to create. Another example:
{{{
In [10]: array([(1,2,3),(1,2)], dtype=object)
Out[10]: array([(1, 2, 3), (1, 2)], dtype=object)
In [11]: array([(1,2),(1,2)], dtype=object)
Out[11]:
array([[1, 2],
[1, 2]], dtype=object)
}}}
New description:
{{{
python -c 'import numpy as N; print N.__version__; a=N.array([1,
(0,1)],dtype=object); print a==1; print a == (0,1), a[1] == (0,1)'
}}}
results in
{{{
1.5.0.dev
[ True False]
[False False] True
}}}
I expected last line to be
{{{
[False True] True
}}}
So, it works for int but doesn't work for tuple... I guess it doesn't try
to compare element by element but does smth else.
--
Yaroslav O. Halchenko
Postdoctoral Fellow, Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] comparison operators (e.g. ==) on array with dtype object do not work
Yaroslav Halchenko wrote:
Dear NumPy People,
First I want to apologize if I misbehaved on NumPy Trac by reopening the
closed ticket
http://projects.scipy.org/numpy/ticket/1362
but I still feel strongly that there is misunderstanding
and the bug/defect is valid. I would appreciate if someone would waste
more of his time to persuade me that I am wrong but please first read
till the end:
The issue, as originally reported, is demonstrated with:
,---
| python -c 'import numpy as N; print N.__version__; a=N.array([1,
(0,1)],dtype=object); print a==1; print a == (0,1), a[1] == (0,1)'
| 1.5.0.dev
| [ True False]
| [False False] True
`---
whenever I expected the last line to be
[False True] True
charris (thanks for all the efforts to enlighten me) summarized it as
the result was correct given that the tuple (0,1) was converted to an
object array with elements 0 and 1. It is *not* converted to an array
containing a tuple.
and I was trying to argue that it is not the case in my example. It is
the case in charris's example though whenever both elements are of
the same length, or there is just a single tuple, i.e.
The problem is that the tuple is converted to an array in the
statement that
does the comparison, not in the construction of the array. Numpy attempts
to convert the right hand side of the == operator into an array. It
then does
the comparison using the two arrays.
One way to get what you want is to create your own array and then do
the comparison:
In [1]: import numpy as np
In [2]: a = np.array([1, (0,1)], dtype='O')
In [3]: t = np.empty(1, dtype='O')
In [4]: t[0] = (0,1)
In [5]: a == t
Out[5]: array([False, True], dtype=bool)
In the above code, a numpy array 't' of objects with shape (1,) is created,
and the single element is assigned the value (0,1). Then the comparison
works as expected.
More food for thought:
In [6]: b = np.array([1, (0,1), foo], dtype='O')
In [7]: b == 1
Out[7]: array([ True, False, False], dtype=bool)
In [8]: b == (0,1)
Out[8]: False
In [9]: b == foo
Out[9]: array([False, False, True], dtype=bool)
Warren
,---
| In [1]: array((0,1), dtype=object)
| Out[1]: array([0, 1], dtype=object)
|
| In [2]: array((0,1), dtype=object).shape
| Out[2]: (2,)
`---
There I would not expect my comparison to be valid indeed. But lets see what
happens in my case:
,---
| In [2]: array([1, (0,1)],dtype=object)
| Out[2]: array([1, (0, 1)], dtype=object)
|
| *In [3]: array([1, (0,1)],dtype=object).shape
| Out[3]: (2,)
|
| *In [4]: array([1, (0,1)],dtype=object)[1].shape
| ---
| AttributeErrorTraceback (most recent call
| last)
|
| /home/yoh/proj/ipython console in module()
|
| AttributeError: 'tuple' object has no attribute 'shape'
`---
So, as far as I see it, the array does contain an object of type tuple,
which does not get correctly compared upon __eq__ operation. Am I
wrong? Or does numpy internally somehow does convert 1st item (ie
tuple) into an array, but casts it back to tuple upon __repr__ or
__getitem__?
Thanks in advance for feedback
On Thu, 14 Jan 2010, NumPy Trac wrote:
#1362: comparison operators (e.g. ==) on array with dtype object do not work
-+--
Reporter: yarikoptic | Owner: somebody
Type: defect | Status: closed
Priority: normal | Milestone:
Component: Other | Version:
Resolution: invalid |Keywords:
-+--
Changes (by charris):
* status: reopened = closed
* resolution: = invalid
Old description:
You can see this better with the '*' operator:
{{{
In [8]: a * (0,2)
Out[8]: array([0, (0, 1, 0, 1)], dtype=object)
}}}
Note how the tuple is concatenated with itself. The reason the original
instance of a worked was that 1 and (0,1) are of different lengths, so
the decent into the nested sequence types stopped at one level and a
tuple is one of the elements. When you do something like ((0,1),(0,1))
the decent goes down two levels and you end up with a 2x2 array of
integer objects. The rule of thumb for object arrays is that you get an
array with as many indices as possible. Which is why object arrays are
hard to create. Another example:
{{{
In [10]: array([(1,2,3),(1,2)], dtype=object)
Out[10]: array([(1, 2, 3), (1, 2)], dtype=object)
In [11]: array([(1,2),(1,2)], dtype=object)
Out[11]:
array([[1, 2],
[1, 2]], dtype=object)
}}}
New description:
{{{
python -c 'import numpy as N; print N.__version__; a=N.array([1,
(0,1)],dtype=object); print a==1; print a == (0,1), a[1] == (0,1)'
}}}
results
Re: [Numpy-discussion] comparison operators (e.g. ==) on array with dtype object do not work
On Thu, Jan 14, 2010 at 5:49 PM, Warren Weckesser
warren.weckes...@enthought.com wrote:
Yaroslav Halchenko wrote:
Dear NumPy People,
First I want to apologize if I misbehaved on NumPy Trac by reopening the
closed ticket
http://projects.scipy.org/numpy/ticket/1362
but I still feel strongly that there is misunderstanding
and the bug/defect is valid. I would appreciate if someone would waste
more of his time to persuade me that I am wrong but please first read
till the end:
The issue, as originally reported, is demonstrated with:
,---
| python -c 'import numpy as N; print N.__version__; a=N.array([1,
(0,1)],dtype=object); print a==1; print a == (0,1), a[1] == (0,1)'
| 1.5.0.dev
| [ True False]
| [False False] True
`---
whenever I expected the last line to be
[False True] True
charris (thanks for all the efforts to enlighten me) summarized it as
the result was correct given that the tuple (0,1) was converted to an
object array with elements 0 and 1. It is *not* converted to an array
containing a tuple.
and I was trying to argue that it is not the case in my example. It is
the case in charris's example though whenever both elements are of
the same length, or there is just a single tuple, i.e.
The problem is that the tuple is converted to an array in the
statement that
does the comparison, not in the construction of the array. Numpy attempts
to convert the right hand side of the == operator into an array. It
then does
the comparison using the two arrays.
One way to get what you want is to create your own array and then do
the comparison:
In [1]: import numpy as np
In [2]: a = np.array([1, (0,1)], dtype='O')
In [3]: t = np.empty(1, dtype='O')
In [4]: t[0] = (0,1)
In [5]: a == t
Out[5]: array([False, True], dtype=bool)
In the above code, a numpy array 't' of objects with shape (1,) is created,
and the single element is assigned the value (0,1). Then the comparison
works as expected.
More food for thought:
In [6]: b = np.array([1, (0,1), foo], dtype='O')
In [7]: b == 1
Out[7]: array([ True, False, False], dtype=bool)
In [8]: b == (0,1)
Out[8]: False
In [9]: b == foo
Out[9]: array([False, False, True], dtype=bool)
It looks difficult to construct an object array with only 1 element,
since a tuple is interpreted as different array elements.
N.array([(0,1)],dtype=object).shape
(1, 2)
N.array([(0,1),()],dtype=object).shape
(2,)
c = N.array([(0,1),()],dtype=object)[:1]
c.shape1,)
a == c
array([False, True], dtype=bool)
It looks like some convention is necessary for interpreting a tuple in
the array construction, but it doesn't look like a problem with the
comparison operator just a consequence.
Josef
Warren
,---
| In [1]: array((0,1), dtype=object)
| Out[1]: array([0, 1], dtype=object)
|
| In [2]: array((0,1), dtype=object).shape
| Out[2]: (2,)
`---
There I would not expect my comparison to be valid indeed. But lets see what
happens in my case:
,---
| In [2]: array([1, (0,1)],dtype=object)
| Out[2]: array([1, (0, 1)], dtype=object)
|
| *In [3]: array([1, (0,1)],dtype=object).shape
| Out[3]: (2,)
|
| *In [4]: array([1, (0,1)],dtype=object)[1].shape
| ---
| AttributeError Traceback (most recent call
| last)
|
| /home/yoh/proj/ipython console in module()
|
| AttributeError: 'tuple' object has no attribute 'shape'
`---
So, as far as I see it, the array does contain an object of type tuple,
which does not get correctly compared upon __eq__ operation. Am I
wrong? Or does numpy internally somehow does convert 1st item (ie
tuple) into an array, but casts it back to tuple upon __repr__ or
__getitem__?
Thanks in advance for feedback
On Thu, 14 Jan 2010, NumPy Trac wrote:
#1362: comparison operators (e.g. ==) on array with dtype object do not work
-+--
Reporter: yarikoptic | Owner: somebody
Type: defect | Status: closed
Priority: normal | Milestone:
Component: Other | Version:
Resolution: invalid | Keywords:
-+--
Changes (by charris):
* status: reopened = closed
* resolution: = invalid
Old description:
You can see this better with the '*' operator:
{{{
In [8]: a * (0,2)
Out[8]: array([0, (0, 1, 0, 1)], dtype=object)
}}}
Note how the tuple is concatenated with itself. The reason the original
instance of a worked was that 1 and (0,1) are of different lengths, so
the decent into the nested sequence types stopped at one level and a
tuple is one of the elements. When you do something like ((0,1),(0,1))
the decent goes down two levels and you end up with a 2x2 array of
integer objects. The rule of thumb for object arrays is that you get an
array with
Re: [Numpy-discussion] comparison operators (e.g. ==) on array with dtype object do not work
On Thu, 14 Jan 2010, josef.p...@gmail.com wrote: It looks difficult to construct an object array with only 1 element, since a tuple is interpreted as different array elements. yeap It looks like some convention is necessary for interpreting a tuple in the array construction, but it doesn't look like a problem with the comparison operator just a consequence. Well -- there is a reason why we use tuples -- they are immutable ... as well as strings actually ;) Thus, imho, it would be a logical API if immutable datatypes are not coerced magically into mutable arrays at least whenever I am already requesting dtype='O'. Such generic treatment of immutable dtypes would address special treatment of strings but it is too much of a change and debatable anyways ;-) -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] comparison operators (e.g. ==) on array with dtype object do not work
Hi Warren,
The problem is that the tuple is converted to an array in the
statement that does the comparison, not in the construction of the
array. Numpy attempts
to convert the right hand side of the == operator into an array.
It then does the comparison using the two arrays.
Thanks for the description! It kinda makes sense now, although, in
general, I am not pleased with the API, I would take it as a documented
feature from now on ;)
One way to get what you want is to create your own array and then do
the comparison:
yeah... I might like to check if lhs has dtype==dtype('object') and then
convert that rhs item into object array before comparison (for now I
just did list comprehension ;))
In [8]: b == (0,1)
Out[8]: False
yeah -- lengths are different now ;)
In [9]: b == foo
Out[9]: array([False, False, True], dtype=bool)
yeah -- strings are getting special treatment despite being iterables
;) but that is ok I guess anyways
The main confusion seems to come from the feature of numpy in doing
smart things -- like deciding either it thinks it needs to do
element-wise comparison across lhs and rhs (if lengths match) or mapping
comparison across all items. That behavior is quite different from basic
Python iterable containers suchas tuples and lists, where it does just
global comparison:
,---
| *In [33]: [1,2] == [1,3]
| Out[33]: False
|
| *In [34]: array([1,2]) == array([1,3])
| Out[34]: array([ True, False], dtype=bool)
`---
I guess I just need to remember that and what you have described
thanks again
--
Yaroslav O. Halchenko
Postdoctoral Fellow, Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Matrix vs array in ma.minimum
Pierre GM wrote: Er, no. np.ma.minimum(a, b) returns the lowest value of a and b element-wsie, or the the lowest element of a is b is None. The behavior is inherited from the very first implementation of maskedarray in numeric. This itself is unexpected, since np.minimum requires at least 2 input arguments. As you observed, the current function breaks down w/ np.matrix objects when only one argument is given (and when the axis is None): we call umath.minimum.reduce on the ravelled matirx, which returns the ravelled matrix. One would expect a scalar, so yes, this behavior is also unexpected. Now, which way should we go ? Keep np.ma.minimum as it is (fixing the bug so that a scalar is returned if the function is called with only 1 argument and an axis None) ? Adapt it to match np.minimum ? I am not a user of Masked Array, so I don't know what is the most desirable behavior. The problem appears when using pylab.imshow on matrices, because matplotlib (and not matlab :) ) uses masked arrays when normalizing the values. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Matrix vs array in ma.minimum
On Jan 14, 2010, at 8:52 PM, David Cournapeau wrote: Pierre GM wrote: Er, no. np.ma.minimum(a, b) returns the lowest value of a and b element-wsie, or the the lowest element of a is b is None. The behavior is inherited from the very first implementation of maskedarray in numeric. This itself is unexpected, since np.minimum requires at least 2 input arguments. As you observed, the current function breaks down w/ np.matrix objects when only one argument is given (and when the axis is None): we call umath.minimum.reduce on the ravelled matirx, which returns the ravelled matrix. One would expect a scalar, so yes, this behavior is also unexpected. Now, which way should we go ? Keep np.ma.minimum as it is (fixing the bug so that a scalar is returned if the function is called with only 1 argument and an axis None) ? Adapt it to match np.minimum ? I am not a user of Masked Array, so I don't know what is the most desirable behavior. I'm not a regular user of np.minimum. The problem appears when using pylab.imshow on matrices, because matplotlib (and not matlab :) ) uses masked arrays when normalizing the values. David, you mind pointing me to the relevan part of the code and/or give me an example ? In any case, I'd appreciate more feedback on the behavior of np.ma.minimum. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] comparison operators (e.g. ==) on array with dtype object do not work
On Thu, Jan 14, 2010 at 3:49 PM, Warren Weckesser warren.weckes...@enthought.com wrote: Yaroslav Halchenko wrote: Dear NumPy People, First I want to apologize if I misbehaved on NumPy Trac by reopening the closed ticket http://projects.scipy.org/numpy/ticket/1362 but I still feel strongly that there is misunderstanding and the bug/defect is valid. I would appreciate if someone would waste more of his time to persuade me that I am wrong but please first read till the end: The issue, as originally reported, is demonstrated with: ,--- | python -c 'import numpy as N; print N.__version__; a=N.array([1, (0,1)],dtype=object); print a==1; print a == (0,1), a[1] == (0,1)' | 1.5.0.dev | [ True False] | [False False] True `--- whenever I expected the last line to be [False True] True charris (thanks for all the efforts to enlighten me) summarized it as the result was correct given that the tuple (0,1) was converted to an object array with elements 0 and 1. It is *not* converted to an array containing a tuple. and I was trying to argue that it is not the case in my example. It is the case in charris's example though whenever both elements are of the same length, or there is just a single tuple, i.e. The problem is that the tuple is converted to an array in the statement that does the comparison, not in the construction of the array. Numpy attempts to convert the right hand side of the == operator into an array. It then does the comparison using the two arrays. One way to get what you want is to create your own array and then do the comparison: In [1]: import numpy as np In [2]: a = np.array([1, (0,1)], dtype='O') In [3]: t = np.empty(1, dtype='O') In [4]: t[0] = (0,1) In [5]: a == t Out[5]: array([False, True], dtype=bool) In the above code, a numpy array 't' of objects with shape (1,) is created, and the single element is assigned the value (0,1). Then the comparison works as expected. More food for thought: In [6]: b = np.array([1, (0,1), foo], dtype='O') In [7]: b == 1 Out[7]: array([ True, False, False], dtype=bool) In [8]: b == (0,1) Out[8]: False Oooh, that last one is strange. Also In [6]: arange(2) == arange(3) Out[6]: False So the comparison isn't element-wise. I rather think a shape mismatch error should be raised in this case. snip Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Matrix vs array in ma.minimum
On Fri, Jan 15, 2010 at 11:59 AM, Pierre GM pgmdevl...@gmail.com wrote: On Jan 14, 2010, at 8:52 PM, David Cournapeau wrote: Pierre GM wrote: Er, no. np.ma.minimum(a, b) returns the lowest value of a and b element-wsie, or the the lowest element of a is b is None. The behavior is inherited from the very first implementation of maskedarray in numeric. This itself is unexpected, since np.minimum requires at least 2 input arguments. As you observed, the current function breaks down w/ np.matrix objects when only one argument is given (and when the axis is None): we call umath.minimum.reduce on the ravelled matirx, which returns the ravelled matrix. One would expect a scalar, so yes, this behavior is also unexpected. Now, which way should we go ? Keep np.ma.minimum as it is (fixing the bug so that a scalar is returned if the function is called with only 1 argument and an axis None) ? Adapt it to match np.minimum ? I am not a user of Masked Array, so I don't know what is the most desirable behavior. I'm not a regular user of np.minimum. Damn, I thought I coul The problem appears when using pylab.imshow on matrices, because matplotlib (and not matlab :) ) uses masked arrays when normalizing the values. David, you mind pointing me to the relevan part of the code and/or give me an example ? Here is a self-contained example reproducing the matplotlib pb: import numpy as np from numpy import ma import matplotlib.cbook as cbook class Normalize: Normalize a given value to the 0-1 range def __init__(self, vmin=None, vmax=None, clip=False): If *vmin* or *vmax* is not given, they are taken from the input's minimum and maximum value respectively. If *clip* is *True* and the given value falls outside the range, the returned value will be 0 or 1, whichever is closer. Returns 0 if:: vmin==vmax Works with scalars or arrays, including masked arrays. If *clip* is *True*, masked values are set to 1; otherwise they remain masked. Clipping silently defeats the purpose of setting the over, under, and masked colors in the colormap, so it is likely to lead to surprises; therefore the default is *clip* = *False*. self.vmin = vmin self.vmax = vmax self.clip = clip def __call__(self, value, clip=None): if clip is None: clip = self.clip if cbook.iterable(value): vtype = 'array' val = ma.asarray(value).astype(np.float) else: vtype = 'scalar' val = ma.array([value]).astype(np.float) self.autoscale_None(val) vmin, vmax = self.vmin, self.vmax if vmin vmax: raise ValueError(minvalue must be less than or equal to maxvalue) elif vmin==vmax: return 0.0 * val else: if clip: mask = ma.getmask(val) val = ma.array(np.clip(val.filled(vmax), vmin, vmax), mask=mask) result = (val-vmin) * (1.0/(vmax-vmin)) if vtype == 'scalar': result = result[0] return result def inverse(self, value): if not self.scaled(): raise ValueError(Not invertible until scaled) vmin, vmax = self.vmin, self.vmax if cbook.iterable(value): val = ma.asarray(value) return vmin + val * (vmax - vmin) else: return vmin + value * (vmax - vmin) def autoscale(self, A): ''' Set *vmin*, *vmax* to min, max of *A*. ''' self.vmin = ma.minimum(A) self.vmax = ma.maximum(A) def autoscale_None(self, A): ' autoscale only None-valued vmin or vmax' if self.vmin is None: self.vmin = ma.minimum(A) if self.vmax is None: self.vmax = ma.maximum(A) def scaled(self): 'return true if vmin and vmax set' return (self.vmin is not None and self.vmax is not None) if __name__ == __main__: x = np.random.randn(10, 10) mx = np.matrix(x) print Normalize()(x) print Normalize()(mx) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Wanted: new release manager for 1.5 and above
On Wed, Jan 13, 2010 at 11:34 PM, David Cournapeau da...@silveregg.co.jpwrote: Charles R Harris wrote: What is the setup one needs to build the installers? It might be well to document that, the dependencies, and the process. Right. The top script is: http://projects.scipy.org/numpy/browser/trunk/release.sh the bulk of the work is in : http://projects.scipy.org/numpy/browser/trunk/pavement.py which describes what is needed to build installers. On mac os x, the release script may be used as is to build every installer + the release notes. Umm, I think it needs some more explanation. There are virtual environments, c compilers, wine, paver, etc. All/ of these might require some installation, version numbers, and setup. This might all seem clear to you, but a newbie coming on to build the packages probably needs more instruction. What sort of setup do you run, what hardware, etc. If code needs to be compiled for the PCC I assume the compiler needs to be to do that. What about c libraries (for numpy) and c++ libraries (for scipy)? Does one need a MAC? etc. I'm probably just ignorant, but I think a careful step by step procedure would be helpful. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Wanted: new release manager for 1.5 and above
Charles R Harris wrote: On Wed, Jan 13, 2010 at 11:34 PM, David Cournapeau da...@silveregg.co.jp mailto:da...@silveregg.co.jp wrote: Charles R Harris wrote: What is the setup one needs to build the installers? It might be well to document that, the dependencies, and the process. Right. The top script is: http://projects.scipy.org/numpy/browser/trunk/release.sh the bulk of the work is in : http://projects.scipy.org/numpy/browser/trunk/pavement.py which describes what is needed to build installers. On mac os x, the release script may be used as is to build every installer + the release notes. Umm, I think it needs some more explanation. There are virtual environments, c compilers, wine, paver, etc. All/ of these might require some installation, version numbers, and setup. This might all seem clear to you, but a newbie coming on to build the packages probably needs more instruction. I think it is a waste of time to document all this very precisely, because it is continuously changing. Documenting everything would boil down to rewrite the paver script in English (and most likely would be much more verbose). That's exactly why I was suggesting to have some volunteers to do 1.4.1 to do the release together, as a way to pass the knowledge around. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
