Re: sometype.__new__ and C subclasses
On May 1, 11:03 pm, James Porter port...@alum.rit.edu wrote: I've been trying to write a Python C extension module that uses NumPy and has a subtype of numpy.ndarray written in C. However, I've run into a snag: calling numpy.ndarray.__new__(mysubtype, ...) triggers an exception in the bowels of Python (this is necessary for a handful of NumPy features). I'm posting to this list to try to figure out why this exception exists in the first place, and what (if anything) I can do to work around it. The exception in question happens in Objects/typeobject.c in tp_new_wrapper. Here's the comment for the block: /* Check that the use doesn't do something silly and unsafe like object.__new__(dict). To do this, we check that the most derived base that's not a heap type is this type. */ The code has the end effect that basetype.__new__(subtype, ...) fails whenever subtype is a statically-defined type (i.e. a normal C extension type object). Why is this necessary in general? I can see why it might be bad for a limited number of core Python types, but it seems unnecessarily limiting for Python C extensions. Why don't you use mysubtype.__new__(mysubtype,...)? If you wrote mysubtype in C, and defined a different tp_new than ndarray, then this exception will trigger. And it ought to; you don't want to use ndarray's tp_new to create an object of your subclass, if you've defined a different tp_new. On a more practical note, is there a way (short of rewriting the subtype in Python) to work around this? It seems that I could call the type metaclass to create a heap type in C, but I'm not sure of all the implications of that. It should work if you use mysubtype.__new__(mysubtype,...). If it doesn't do what you want, then there's probably something wrong with the way you subclassed ndarray. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: sometype.__new__ and C subclasses
On 5/2/2010 4:34 AM, Carl Banks wrote: Why don't you use mysubtype.__new__(mysubtype,...)? If you wrote mysubtype in C, and defined a different tp_new than ndarray, then this exception will trigger. And it ought to; you don't want to use ndarray's tp_new to create an object of your subclass, if you've defined a different tp_new. Unfortunately, I can't do that, since that call is in NumPy itself and it's part of their standard way of making instances of subclasses of ndarray. Functions like numpy.zeros_like use ndarray.__new__(subtype, ...) to create new arrays based on the shape of other arrays. The Python version of the subclass is shown here: http://docs.scipy.org/doc/numpy/user/basics.subclassing.html#slightly-more-realistic-example-attribute-added-to-existing-array, and I'm trying to write something pretty similar in C. I'm trying to stay in C since everything else is in C, so it's easier to stay in C then to jump back and forth all the time. Maybe the real answer to this question is NumPy is doing it wrong and I should be on their list; still, it seems strange that the behavior is different between Python and C. - Jim -- http://mail.python.org/mailman/listinfo/python-list
Re: sometype.__new__ and C subclasses
On 2010-05-02 12:48 , James Porter wrote: On 5/2/2010 4:34 AM, Carl Banks wrote: Why don't you use mysubtype.__new__(mysubtype,...)? If you wrote mysubtype in C, and defined a different tp_new than ndarray, then this exception will trigger. And it ought to; you don't want to use ndarray's tp_new to create an object of your subclass, if you've defined a different tp_new. Unfortunately, I can't do that, since that call is in NumPy itself and it's part of their standard way of making instances of subclasses of ndarray. Functions like numpy.zeros_like use ndarray.__new__(subtype, ...) to create new arrays based on the shape of other arrays. The Python version of the subclass is shown here: http://docs.scipy.org/doc/numpy/user/basics.subclassing.html#slightly-more-realistic-example-attribute-added-to-existing-array, and I'm trying to write something pretty similar in C. I'm trying to stay in C since everything else is in C, so it's easier to stay in C then to jump back and forth all the time. Maybe the real answer to this question is NumPy is doing it wrong and I should be on their list; still, it seems strange that the behavior is different between Python and C. Perhaps things would be clearer if you could post the C code that you've written that fails. So far, you've only alluded at what you are doing using Python-syntax examples. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: sometype.__new__ and C subclasses
On 5/2/2010 1:43 PM, Robert Kern wrote:
Perhaps things would be clearer if you could post the C code that you've
written that fails. So far, you've only alluded at what you are doing
using Python-syntax examples.
I'm not sure how much this will help, but here you go. The actual C code
probably doesn't matter except for where I set tp_flags, tp_new, and
register the type, but I included it for completeness. The full C source
is available here if you need it, but be warned that other strangeness
abounds in the code:
http://trac.mcs.anl.gov/projects/ITAPS/browser/python/trunk/iMesh_array.inl?rev=3831.
Obviously, this is kind of a bizarre case, so I'm not entirely sure what
the best route is here.
Thanks,
Jim
static PyObject*
iMeshArrObj_new(PyTypeObject *cls,PyObject *args,PyObject *kw)
{
static char *kwlist[] = {object,instance,0};
PyObject *obj;
iMesh_Object *instance = NULL;
PyObject *arr = NULL;
iMeshArr_Object *self;
if(!PyArg_ParseTupleAndKeywords(args,kw,O|O!,kwlist,obj,
iMesh_Type,instance))
return NULL;
arr = PyArray_FROM_O(obj);
if(arr == NULL)
return NULL;
self = (iMeshArr_Object*)PyObject_CallMethod(arr,view,O,cls);
Py_DECREF(arr);
if(self == NULL)
return NULL;
/* some boring stuff to set |instance| */
return self;
}
static void
iMeshArrObj_dealloc(iMeshArr_Object *self)
{
Py_XDECREF(self-instance);
self-array.ob_type-tp_free((PyObject*)self);
}
static PyObject*
iMeshArrObj_finalize(iMeshArr_Object *self,PyObject *args)
{
iMeshArr_Object *context;
if(PyArg_ParseTuple(args,O!,iMeshArr_Type,context))
{
self-instance = context-instance;
Py_XINCREF(self-instance);
}
PyErr_Clear();
Py_RETURN_NONE;
}
static PyMethodDef iMeshArrObj_methods[] = {
{ __array_finalize__, (PyCFunction)iMeshArrObj_finalize,
METH_VARARGS,
},
{0}
};
static PyMemberDef iMeshArrObj_members[] = {
{instance, T_OBJECT_EX, offsetof(iMeshArr_Object, instance),
READONLY, base iMesh instance},
{0}
};
static PyTypeObject iMeshArr_Type = {
PyObject_HEAD_INIT(NULL)
/* ... */
(destructor)iMeshArrObj_dealloc, /* tp_dealloc */
/* ... */
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
iMesh array objects,/* tp_doc */
/* ... */
iMeshArrObj_methods, /* tp_methods */
iMeshArrObj_members, /* tp_members */
/* ... */
iMeshArrObj_new, /* tp_new */
};
PyMODINITFUNC initiMesh(void)
{
PyObject *m;
m = Py_InitModule(iMesh,module_methods);
import_array();
iMeshArr_Type.tp_base = PyArray_Type;
if(PyType_Ready(iMeshArr_Type) 0)
return;
Py_INCREF(iMeshArr_Type);
PyModule_AddObject(m,Array,(PyObject *)iMeshArr_Type);
}
/* End C code */
And then in Python:
A = iMesh.Array(numpy.array([1,2,3,4,5]), instance=mesh)
numpy.zeros_like(A) # fails here
Inside NumPy, zeros_like looks like this (there's a bit more than this,
but it's irrelevant to this problem):
def zeros_like(a):
if isinstance(a, ndarray):
res = ndarray.__new__(type(a), a.shape, a.dtype,
order=a.flags.fnc)
res.fill(0)
return res
--
http://mail.python.org/mailman/listinfo/python-list
Re: sometype.__new__ and C subclasses
On May 2, 10:48 am, James Porter port...@alum.rit.edu wrote: On 5/2/2010 4:34 AM, Carl Banks wrote: Why don't you use mysubtype.__new__(mysubtype,...)? If you wrote mysubtype in C, and defined a different tp_new than ndarray, then this exception will trigger. And it ought to; you don't want to use ndarray's tp_new to create an object of your subclass, if you've defined a different tp_new. Unfortunately, I can't do that, since that call is in NumPy itself and it's part of their standard way of making instances of subclasses of ndarray. Functions like numpy.zeros_like use ndarray.__new__(subtype, ...) to create new arrays based on the shape of other arrays. The Python version of the subclass is shown here: http://docs.scipy.org/doc/numpy/user/basics.subclassing.html#slightly..., and I'm trying to write something pretty similar in C. I'm trying to stay in C since everything else is in C, so it's easier to stay in C then to jump back and forth all the time. Maybe the real answer to this question is NumPy is doing it wrong and I should be on their list; still, it seems strange that the behavior is different between Python and C. I would say numpy is wrong here, so I suggest filing a bug report. In fact I can't think of any benefit to EVER calling X.__new__(Y) where X is not Y. Maybe old-style classes? Someone who wants to ensure they're getting an instance of a certain type can check issubclass(Y,X) then call Y.__new__(Y). Unfortunately, you just can't get rid of the test in tp_new_wrapper. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: sometype.__new__ and C subclasses
On 2010-05-02 15:28 , Carl Banks wrote: On May 2, 10:48 am, James Porterport...@alum.rit.edu wrote: On 5/2/2010 4:34 AM, Carl Banks wrote: Why don't you use mysubtype.__new__(mysubtype,...)? If you wrote mysubtype in C, and defined a different tp_new than ndarray, then this exception will trigger. And it ought to; you don't want to use ndarray's tp_new to create an object of your subclass, if you've defined a different tp_new. Unfortunately, I can't do that, since that call is in NumPy itself and it's part of their standard way of making instances of subclasses of ndarray. Functions like numpy.zeros_like use ndarray.__new__(subtype, ...) to create new arrays based on the shape of other arrays. The Python version of the subclass is shown here: http://docs.scipy.org/doc/numpy/user/basics.subclassing.html#slightly..., and I'm trying to write something pretty similar in C. I'm trying to stay in C since everything else is in C, so it's easier to stay in C then to jump back and forth all the time. Maybe the real answer to this question is NumPy is doing it wrong and I should be on their list; still, it seems strange that the behavior is different between Python and C. I would say numpy is wrong here, so I suggest filing a bug report. In fact I can't think of any benefit to EVER calling X.__new__(Y) where X is not Y. Maybe old-style classes? Someone who wants to ensure they're getting an instance of a certain type can check issubclass(Y,X) then call Y.__new__(Y). Well, the Y.__new__(Y) may call X.__new__(Y) (and we certainly do this successfully in other Python subclasses of ndarray; this also appears in the Python regression tests). I'm not sure why this would be permitted there and not in a regular function (numpy.zeros_like() seems to be the function that does this and fails for the OP). The reason we do it there instead of calling the subclass's constructor is because the subclass's constructor may have different arguments. I'm happy to concede that this might be a bug in numpy, but I don't understand why this is allowed for Python subclasses but not C subtypes. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: sometype.__new__ and C subclasses
On 2010-05-02 15:03 , James Porter wrote: And then in Python: A = iMesh.Array(numpy.array([1,2,3,4,5]), instance=mesh) numpy.zeros_like(A) # fails here Inside NumPy, zeros_like looks like this (there's a bit more than this, but it's irrelevant to this problem): def zeros_like(a): if isinstance(a, ndarray): res = ndarray.__new__(type(a), a.shape, a.dtype, order=a.flags.fnc) res.fill(0) return res Well, I think we can change zeros_like() and the rest to work around this issue. Can you bring it up on the numpy mailing list? def zeros_like(a): if isinstance(a, ndarray): res = numpy.empty(a.shape, a.dtype, order=a.flags.fnc) res.fill(0) res = res.view(type(a)) return res ... -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: sometype.__new__ and C subclasses
On May 2, 1:51 pm, Robert Kern robert.k...@gmail.com wrote: On 2010-05-02 15:28 , Carl Banks wrote: On May 2, 10:48 am, James Porterport...@alum.rit.edu wrote: On 5/2/2010 4:34 AM, Carl Banks wrote: Why don't you use mysubtype.__new__(mysubtype,...)? If you wrote mysubtype in C, and defined a different tp_new than ndarray, then this exception will trigger. And it ought to; you don't want to use ndarray's tp_new to create an object of your subclass, if you've defined a different tp_new. Unfortunately, I can't do that, since that call is in NumPy itself and it's part of their standard way of making instances of subclasses of ndarray. Functions like numpy.zeros_like use ndarray.__new__(subtype, ...) to create new arrays based on the shape of other arrays. The Python version of the subclass is shown here: http://docs.scipy.org/doc/numpy/user/basics.subclassing.html#slightly..., and I'm trying to write something pretty similar in C. I'm trying to stay in C since everything else is in C, so it's easier to stay in C then to jump back and forth all the time. Maybe the real answer to this question is NumPy is doing it wrong and I should be on their list; still, it seems strange that the behavior is different between Python and C. I would say numpy is wrong here, so I suggest filing a bug report. In fact I can't think of any benefit to EVER calling X.__new__(Y) where X is not Y. Maybe old-style classes? Someone who wants to ensure they're getting an instance of a certain type can check issubclass(Y,X) then call Y.__new__(Y). Well, the Y.__new__(Y) may call X.__new__(Y) (and we certainly do this successfully in other Python subclasses of ndarray; this also appears in the Python regression tests). I'm not sure why this would be permitted there and not in a regular function (numpy.zeros_like() seems to be the function that does this and fails for the OP). The reason we do it there instead of calling the subclass's constructor is because the subclass's constructor may have different arguments. I'm happy to concede that this might be a bug in numpy, but I don't understand why this is allowed for Python subclasses but not C subtypes. Because Python subclasses (i.e., heap types) all invoke tp_new_wrapper, which is guaranteed to call the tp_new of the most derived base. C subtypes can, and often have to, replace tp_new with their own version. Calling a base type's tp_new when you've defined your own tp_new at the C level is dangerous. As for the issue with a subclass's arguments being different, I'm shocked that anyone at numpy could possibly think bypassing the subtype's constructor is good idea. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: sometype.__new__ and C subclasses
On 5/2/2010 3:58 PM, Robert Kern wrote: Well, I think we can change zeros_like() and the rest to work around this issue. Can you bring it up on the numpy mailing list? def zeros_like(a): if isinstance(a, ndarray): res = numpy.empty(a.shape, a.dtype, order=a.flags.fnc) res.fill(0) res = res.view(type(a)) return res ... I'm having difficulty posting to the NumPy list (both via gmane and email) so I'm just going to put this here so it doesn't get lost. zeros_like probably needs to call __array_finalize__ for this to work properly (it'll cause a segfault for me otherwise): def zeros_like(a): if isinstance(a, ndarray): res = numpy.zeros(a.shape, a.dtype, order=a.flags.fnc) res = res.view(type(a)) res.__array_finalize__(a) return res ... - Jim -- http://mail.python.org/mailman/listinfo/python-list
Re: sometype.__new__ and C subclasses
James Porter wrote: Functions like numpy.zeros_like use ndarray.__new__(subtype, ...) to create new arrays based on the shape of other arrays. Maybe the real answer to this question is NumPy is doing it wrong Yes, I think NumPy is doing it wrong, even for subclasses written in Python. If the subtype has overridden ndarray's __new__ method, the way NumPy is doing it will skip the subclass's version of the method. -- Greg -- http://mail.python.org/mailman/listinfo/python-list
