[Numpy-discussion] subclassing ndarray in python3

[Darren Dale]

Now that the trunk has some support for python3, I am working on
making Quantities work with python3 as well. I'm running into some
problems related to subclassing ndarray that can be illustrated with a
simple script, reproduced below. It looks like there is a problem with
the reflected operations, I see problems with __rmul__ and __radd__,
but not with __mul__ and __add__:

import numpy as np


class A(np.ndarray):
def __new__(cls, *args, **kwargs):
return np.ndarray.__new__(cls, *args, **kwargs)

class B(A):
def __mul__(self, other):
return self.view(A).__mul__(other)
def __rmul__(self, other):
return self.view(A).__rmul__(other)
def __add__(self, other):
return self.view(A).__add__(other)
def __radd__(self, other):
return self.view(A).__radd__(other)

a = A((10,))
b = B((10,))

print('A __mul__:')
print(a.__mul__(2))
# ok
print(a.view(np.ndarray).__mul__(2))
# ok
print(a*2)
# ok

print('A __rmul__:')
print(a.__rmul__(2))
# yields NotImplemented
print(a.view(np.ndarray).__rmul__(2))
# yields NotImplemented
print(2*a)
# ok !!??

print('B __mul__:')
print(b.__mul__(2))
# ok
print(b.view(A).__mul__(2))
# ok
print(b.view(np.ndarray).__mul__(2))
# ok
print(b*2)
# ok

print('B __add__:')
print(b.__add__(2))
# ok
print(b.view(A).__add__(2))
# ok
print(b.view(np.ndarray).__add__(2))
# ok
print(b+2)
# ok

print('B __rmul__:')
print(b.__rmul__(2))
# yields NotImplemented
print(b.view(A).__rmul__(2))
# yields NotImplemented
print(b.view(np.ndarray).__rmul__(2))
# yields NotImplemented
print(2*b)
# yields: TypeError: unsupported operand type(s) for *: 'int' and 'B'

print('B __radd__:')
print(b.__radd__(2))
# yields NotImplemented
print(b.view(A).__radd__(2))
# yields NotImplemented
print(b.view(np.ndarray).__radd__(2))
# yields NotImplemented
print(2+b)
# yields: TypeError: unsupported operand type(s) for +: 'int' and 'B'
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] subclassing ndarray in python3

[Pauli Virtanen]

Hi Darren,

to, 2010-03-11 kello 11:11 -0500, Darren Dale kirjoitti:
 Now that the trunk has some support for python3, I am working on
 making Quantities work with python3 as well. I'm running into some
 problems related to subclassing ndarray that can be illustrated with a
 simple script, reproduced below. It looks like there is a problem with
 the reflected operations, I see problems with __rmul__ and __radd__,
 but not with __mul__ and __add__:

Thanks for testing. I wish the test suite was more complete (hint!
hint! :)

Yes, Python 3 introduced some semantic changes in how subclasses of
builtin classes (= written in C) inherit the __r*__ operations.

Below I'll try to explain what is going on. We probably need to change
some things to make things work better on Py3, within the bounds we are
able to.

Suggestions are welcome. The most obvious one could be to explicitly
implement __rmul__ etc. on Python 3.

[clip]
 class A(np.ndarray):
 def __new__(cls, *args, **kwargs):
 return np.ndarray.__new__(cls, *args, **kwargs)
 
 class B(A):
 def __mul__(self, other):
 return self.view(A).__mul__(other)
 def __rmul__(self, other):
 return self.view(A).__rmul__(other)
 def __add__(self, other):
 return self.view(A).__add__(other)
 def __radd__(self, other):
 return self.view(A).__radd__(other)
[clip]
 print('A __rmul__:')
 print(a.__rmul__(2))
 # yields NotImplemented
 print(a.view(np.ndarray).__rmul__(2))
 # yields NotImplemented

Correct. ndarray does not implement __rmul__, but relies on an automatic
wrapper generated by Python.

The automatic wrapper (wrap_binaryfunc_r) does the following:

1. Is `type(other)` a subclass of `type(self)`?
   If yes, call __mul__ with swapped arguments.
2. If not, bail out with NotImplemented.

So it bails out.

Previously, the ndarray type had a flag that made Python to skip the
subclass check. That does not exist any more on Python 3, and is the
root of this issue.

 print(2*a)
 # ok !!??

Here, Python checks

1. Does nb_multiply from the left op succeed? Nope, since floats don't
   know how to multiply ndarrays.

2. Does nb_multiply from the right op succeed? Here the execution
   passes *directly* to array_multiply, completely skipping the __rmul__
   wrapper.

   Note also that in the C-level number protocol there is only a single
   multiplication function for both left and right multiplication.

[clip]
 print('B __rmul__:')
 print(b.__rmul__(2))
 # yields NotImplemented
 print(b.view(A).__rmul__(2))
 # yields NotImplemented
 print(b.view(np.ndarray).__rmul__(2))
 # yields NotImplemented
 print(2*b)
 # yields: TypeError: unsupported operand type(s) for *: 'int' and 'B'

But here, the subclass calls the wrapper ndarray.__rmul__, which wants
to be careful with types, and hence fails.

Yes, probably explicitly defining __rmul__ for ndarray could be the
right solution. Please file a bug report on this.

Cheers,
Pauli



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] subclassing ndarray in python3

[Darren Dale]

Hi Pauli,

On Thu, Mar 11, 2010 at 3:38 PM, Pauli Virtanen p...@iki.fi wrote:
 Thanks for testing. I wish the test suite was more complete (hint!
 hint! :)

I'll be happy to contribute, but lately I get a few 15-30 minute
blocks a week for this kind of work (hence the short attempt to work
on Quantities this morning), and its not likely to let up for about 3
weeks.

 Yes, probably explicitly defining __rmul__ for ndarray could be the
 right solution. Please file a bug report on this.

Done: http://projects.scipy.org/numpy/ticket/1426

Cheers, and *thank you* for all you have already done to support python-3,
Darren
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion