Re: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail
Hi, I made a PR with my fix: https://github.com/numpy/numpy/pull/2709 Frédéric On Tue, Oct 2, 2012 at 6:18 PM, Charles R Harris charlesr.har...@gmail.comwrote: On Tue, Oct 2, 2012 at 1:44 PM, Frédéric Bastien no...@nouiz.org wrote: With numpy 1.6.2, it is working. So this is an interface change. Are you sure you want this? This break existing code. I do not understand what you mean by slot? Pythonese for structure member ;) I'm not sure is the PyArray_SWAP is a good long term idea. I would not make it if it is only for temporarily. The C++ stdlib provides something similar for std::vector. One common use case would be to pass in a vector by reference that gets swapped with one on the stack. When the function exits the one on the stack is cleaned up and the vector that was passed in has the new data, but it has to be the same type. For PyArray_SWAP I was thinking of swapping everything: type, dims, strides, data, etc. That is what f2py does. To set the base ptr, there is PyArray_SetBaseObject() fct that is new in 1.7. Is a similar fct useful in the long term for numpy? In the case where we implement differently the ndarray object, I think it won't be useful. We will also need to know how the memory is laid out by numpy for performance critical code. We we will need an attribute that tell the intern structure used. So do you want to force this interface change in numpy 1.7 so that I need to write codes now or can I wait to do it when you force the new interface? Well, no we don't want to force you to use the new interface. If you don't define NPY_NO_DEPRECATED_API things should still work. Although if it is defined the function returns an rvalue, so some other method needs to be provided for what you are doing. Currently the used code for PyArray_BYTES is: #define PyArray_BYTES(obj) ((char *)(((PyArrayObject_fields *)(obj))-data)) if I change it to #define PyArray_BYTES(obj) PyArrayObject_fields *)(obj))-data)) it work! I don't understand why removing the case make it work. the data field is already an (char*), so this should not make a difference to my underderstanding. But I'm missing something here, do someone know? What I find strange is that it is the same macro in 1.7 and 1.6, only the name of the structure was changed. Hmm... This looks almost like some compiler subtlety, I wonder if the compiler version/optimization flags have changed? In any case, I think the second form would be more correct for the lvalue since the structure member is, as you say, already a char*. We want things to work for you as they should, so we need to understand this and fix it. snip Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail
Hi, I don't know if that was raised or not, but in np1.7b2 doing this fail with this error message: PyArray_BYTES(obj)=ptr file:lne_number:offset: error: lvalue required as left operatnd of assignment. I tried with PyArray_DATA(obj)=ptr and this also fail. Do you want to remove this feature now? I would think this change will be done at the same time as the one related to the macro NPY_NO_DEPRECATED_API. If I missed the discussion about this, tell me. thanks Fred ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail
On Tue, Oct 2, 2012 at 8:34 AM, Frédéric Bastien no...@nouiz.org wrote: Hi, I don't know if that was raised or not, but in np1.7b2 doing this fail with this error message: PyArray_BYTES(obj)=ptr file:lne_number:offset: error: lvalue required as left operatnd of assignment. I tried with PyArray_DATA(obj)=ptr and this also fail. Do you want to remove this feature now? I would think this change will be done at the same time as the one related to the macro NPY_NO_DEPRECATED_API. If I missed the discussion about this, tell me. f2py wants to do the same thing, i.e., change the data pointer of an existing array, which is why NPY_NO_DEPRECATED_API is not defined in that module. I had some discussion off list with Pearu about that but it petered out. I think for this sort of thing is a new function needs to be implemented. What precisely is your application? IIRC, Pearu was using it to exchange pointers between two arrays to avoid a copy. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail
We implement our own subtensor(x[...], where ... can be index or slice) c code due to the way Theano work. I can probably change the logic, but if you plan to revert this interface changes, I prefer to wait for this change we someone else is doing other changes that would conflict. Also, I did a Theano release candidate and I really would like the final version to work with the next release of NumPy. thanks. Fred On Tue, Oct 2, 2012 at 11:33 AM, Charles R Harris charlesr.har...@gmail.com wrote: On Tue, Oct 2, 2012 at 8:34 AM, Frédéric Bastien no...@nouiz.org wrote: Hi, I don't know if that was raised or not, but in np1.7b2 doing this fail with this error message: PyArray_BYTES(obj)=ptr file:lne_number:offset: error: lvalue required as left operatnd of assignment. I tried with PyArray_DATA(obj)=ptr and this also fail. Do you want to remove this feature now? I would think this change will be done at the same time as the one related to the macro NPY_NO_DEPRECATED_API. If I missed the discussion about this, tell me. f2py wants to do the same thing, i.e., change the data pointer of an existing array, which is why NPY_NO_DEPRECATED_API is not defined in that module. I had some discussion off list with Pearu about that but it petered out. I think for this sort of thing is a new function needs to be implemented. What precisely is your application? IIRC, Pearu was using it to exchange pointers between two arrays to avoid a copy. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail
On Tue, Oct 2, 2012 at 9:45 AM, Frédéric Bastien no...@nouiz.org wrote: We implement our own subtensor(x[...], where ... can be index or slice) c code due to the way Theano work. I can probably change the logic, but if you plan to revert this interface changes, I prefer to wait for this change we someone else is doing other changes that would conflict. Also, I did a Theano release candidate and I really would like the final version to work with the next release of NumPy. Well, you don't *have* to define NPY_NO_DEPRECATED_API. If you don't you can access the array as before using the macros even for future versions of numpy. The only way that could cause a problems is if the array structure is rearranged and I don't think that will happen anytime soon. On that account there has not been any discussion of reverting the changes. However, I'd like f2py generated modules to use the new functions at some point and in order to do that numpy needs to supply some extra functionality, I'm just not sure of the best way to do it at the moment. If I had a good idea of what you want to do it would help in deciding what numpy should provide. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail
On Tue, Oct 2, 2012 at 1:18 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Tue, Oct 2, 2012 at 9:45 AM, Frédéric Bastien no...@nouiz.org wrote: We implement our own subtensor(x[...], where ... can be index or slice) c code due to the way Theano work. I can probably change the logic, but if you plan to revert this interface changes, I prefer to wait for this change we someone else is doing other changes that would conflict. Also, I did a Theano release candidate and I really would like the final version to work with the next release of NumPy. Well, you don't *have* to define NPY_NO_DEPRECATED_API. If you don't you can access the array as before using the macros even for future versions of numpy. The only way that could cause a problems is if the array structure is rearranged and I don't think that will happen anytime soon. On that account there has not been any discussion of reverting the changes. However, I'd like f2py generated modules to use the new functions at some point and in order to do that numpy needs to supply some extra functionality, I'm just not sure of the best way to do it at the moment. If I had a good idea of what you want to do it would help in deciding what numpy should provide. I do not define NPY_NO_DEPRECATED_API, so I get g++ warning about that. That is the problem I have. What I do is close to that: alloc a new ndarray that point to the start of the ndarray I want to view with the right number of output dimensions. Compute the new dimensions/strides and data ptr. Then set the data ptr to what I computed. Then set the base ptr. I can reverse this and create the ndarray only at the end, but as this change break existing code here, it can break more code. That is why I wrote to the list. doing PyArray_BASE(xview) = ptr work when I don't define NPY_NO_DEPRECATED_API, but do not work when I define NPY_NO_DEPRECATED_API. I would have expected the same for PyArray_BYTES/DATA. Do this explain clearly the problem I saw? Fred ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail
On Tue, Oct 2, 2012 at 11:30 AM, Frédéric Bastien no...@nouiz.org wrote: On Tue, Oct 2, 2012 at 1:18 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Tue, Oct 2, 2012 at 9:45 AM, Frédéric Bastien no...@nouiz.org wrote: We implement our own subtensor(x[...], where ... can be index or slice) c code due to the way Theano work. I can probably change the logic, but if you plan to revert this interface changes, I prefer to wait for this change we someone else is doing other changes that would conflict. Also, I did a Theano release candidate and I really would like the final version to work with the next release of NumPy. Well, you don't *have* to define NPY_NO_DEPRECATED_API. If you don't you can access the array as before using the macros even for future versions of numpy. The only way that could cause a problems is if the array structure is rearranged and I don't think that will happen anytime soon. On that account there has not been any discussion of reverting the changes. However, I'd like f2py generated modules to use the new functions at some point and in order to do that numpy needs to supply some extra functionality, I'm just not sure of the best way to do it at the moment. If I had a good idea of what you want to do it would help in deciding what numpy should provide. I do not define NPY_NO_DEPRECATED_API, so I get g++ warning about that. That is the problem I have. What I do is close to that: alloc a new ndarray that point to the start of the ndarray I want to view with the right number of output dimensions. Compute the new dimensions/strides and data ptr. Then set the data ptr to what I computed. Then set the base ptr. I can reverse this and create the ndarray only at the end, but as this change break existing code here, it can break more code. That is why I wrote to the list. doing PyArray_BASE(xview) = ptr work when I don't define NPY_NO_DEPRECATED_API, but do not work when I define NPY_NO_DEPRECATED_API. I would have expected the same for PyArray_BYTES/DATA. Do this explain clearly the problem I saw? Yes, thanks. I see in ndarraytypes.h #define PyArray_DATA(obj) ((void *)(((PyArrayObject_fields *)(obj))-data)) I wonder if the cast to void* is causing a problem? Could you try char* instead? Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail
On Tue, Oct 2, 2012 at 11:43 AM, Charles R Harris charlesr.har...@gmail.com wrote: On Tue, Oct 2, 2012 at 11:30 AM, Frédéric Bastien no...@nouiz.org wrote: On Tue, Oct 2, 2012 at 1:18 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Tue, Oct 2, 2012 at 9:45 AM, Frédéric Bastien no...@nouiz.org wrote: We implement our own subtensor(x[...], where ... can be index or slice) c code due to the way Theano work. I can probably change the logic, but if you plan to revert this interface changes, I prefer to wait for this change we someone else is doing other changes that would conflict. Also, I did a Theano release candidate and I really would like the final version to work with the next release of NumPy. Well, you don't *have* to define NPY_NO_DEPRECATED_API. If you don't you can access the array as before using the macros even for future versions of numpy. The only way that could cause a problems is if the array structure is rearranged and I don't think that will happen anytime soon. On that account there has not been any discussion of reverting the changes. However, I'd like f2py generated modules to use the new functions at some point and in order to do that numpy needs to supply some extra functionality, I'm just not sure of the best way to do it at the moment. If I had a good idea of what you want to do it would help in deciding what numpy should provide. I do not define NPY_NO_DEPRECATED_API, so I get g++ warning about that. That is the problem I have. What I do is close to that: alloc a new ndarray that point to the start of the ndarray I want to view with the right number of output dimensions. Compute the new dimensions/strides and data ptr. Then set the data ptr to what I computed. Then set the base ptr. I can reverse this and create the ndarray only at the end, but as this change break existing code here, it can break more code. That is why I wrote to the list. doing PyArray_BASE(xview) = ptr work when I don't define NPY_NO_DEPRECATED_API, but do not work when I define NPY_NO_DEPRECATED_API. I would have expected the same for PyArray_BYTES/DATA. Do this explain clearly the problem I saw? Yes, thanks. I see in ndarraytypes.h #define PyArray_DATA(obj) ((void *)(((PyArrayObject_fields *)(obj))-data)) I wonder if the cast to void* is causing a problem? Could you try char* instead? Oops, the problem is that you need a pointer to the slot, not the pointer in the slot. That is, you need a different macro/function. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail
On Tue, Oct 2, 2012 at 11:47 AM, Charles R Harris charlesr.har...@gmail.com wrote: On Tue, Oct 2, 2012 at 11:43 AM, Charles R Harris charlesr.har...@gmail.com wrote: On Tue, Oct 2, 2012 at 11:30 AM, Frédéric Bastien no...@nouiz.orgwrote: On Tue, Oct 2, 2012 at 1:18 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Tue, Oct 2, 2012 at 9:45 AM, Frédéric Bastien no...@nouiz.org wrote: We implement our own subtensor(x[...], where ... can be index or slice) c code due to the way Theano work. I can probably change the logic, but if you plan to revert this interface changes, I prefer to wait for this change we someone else is doing other changes that would conflict. Also, I did a Theano release candidate and I really would like the final version to work with the next release of NumPy. Well, you don't *have* to define NPY_NO_DEPRECATED_API. If you don't you can access the array as before using the macros even for future versions of numpy. The only way that could cause a problems is if the array structure is rearranged and I don't think that will happen anytime soon. On that account there has not been any discussion of reverting the changes. However, I'd like f2py generated modules to use the new functions at some point and in order to do that numpy needs to supply some extra functionality, I'm just not sure of the best way to do it at the moment. If I had a good idea of what you want to do it would help in deciding what numpy should provide. I do not define NPY_NO_DEPRECATED_API, so I get g++ warning about that. That is the problem I have. What I do is close to that: alloc a new ndarray that point to the start of the ndarray I want to view with the right number of output dimensions. Compute the new dimensions/strides and data ptr. Then set the data ptr to what I computed. Then set the base ptr. I can reverse this and create the ndarray only at the end, but as this change break existing code here, it can break more code. That is why I wrote to the list. doing PyArray_BASE(xview) = ptr work when I don't define NPY_NO_DEPRECATED_API, but do not work when I define NPY_NO_DEPRECATED_API. I would have expected the same for PyArray_BYTES/DATA. Do this explain clearly the problem I saw? Yes, thanks. I see in ndarraytypes.h #define PyArray_DATA(obj) ((void *)(((PyArrayObject_fields *)(obj))-data)) I wonder if the cast to void* is causing a problem? Could you try char* instead? Oops, the problem is that you need a pointer to the slot, not the pointer in the slot. That is, you need a different macro/function. And what I was thinking of for this was something like a PyArray_SWAP, that simply swaps the contents of two array structures. However, that needs to be analysed closely for nasty side effects beyond the surprise of an array unexpectedly changing all of its characteristics underneath its pointer. It's inherently dangerous but probably essential for some things. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail
With numpy 1.6.2, it is working. So this is an interface change. Are you sure you want this? This break existing code. I do not understand what you mean by slot? I'm not sure is the PyArray_SWAP is a good long term idea. I would not make it if it is only for temporarily. To set the base ptr, there is PyArray_SetBaseObject() fct that is new in 1.7. Is a similar fct useful in the long term for numpy? In the case where we implement differently the ndarray object, I think it won't be useful. We will also need to know how the memory is laid out by numpy for performance critical code. We we will need an attribute that tell the intern structure used. So do you want to force this interface change in numpy 1.7 so that I need to write codes now or can I wait to do it when you force the new interface? Currently the used code for PyArray_BYTES is: #define PyArray_BYTES(obj) ((char *)(((PyArrayObject_fields *)(obj))-data)) if I change it to #define PyArray_BYTES(obj) PyArrayObject_fields *)(obj))-data)) it work! I don't understand why removing the case make it work. the data field is already an (char*), so this should not make a difference to my underderstanding. But I'm missing something here, do someone know? Fred On Tue, Oct 2, 2012 at 1:47 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Tue, Oct 2, 2012 at 11:43 AM, Charles R Harris charlesr.har...@gmail.com wrote: On Tue, Oct 2, 2012 at 11:30 AM, Frédéric Bastien no...@nouiz.org wrote: On Tue, Oct 2, 2012 at 1:18 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Tue, Oct 2, 2012 at 9:45 AM, Frédéric Bastien no...@nouiz.org wrote: We implement our own subtensor(x[...], where ... can be index or slice) c code due to the way Theano work. I can probably change the logic, but if you plan to revert this interface changes, I prefer to wait for this change we someone else is doing other changes that would conflict. Also, I did a Theano release candidate and I really would like the final version to work with the next release of NumPy. Well, you don't *have* to define NPY_NO_DEPRECATED_API. If you don't you can access the array as before using the macros even for future versions of numpy. The only way that could cause a problems is if the array structure is rearranged and I don't think that will happen anytime soon. On that account there has not been any discussion of reverting the changes. However, I'd like f2py generated modules to use the new functions at some point and in order to do that numpy needs to supply some extra functionality, I'm just not sure of the best way to do it at the moment. If I had a good idea of what you want to do it would help in deciding what numpy should provide. I do not define NPY_NO_DEPRECATED_API, so I get g++ warning about that. That is the problem I have. What I do is close to that: alloc a new ndarray that point to the start of the ndarray I want to view with the right number of output dimensions. Compute the new dimensions/strides and data ptr. Then set the data ptr to what I computed. Then set the base ptr. I can reverse this and create the ndarray only at the end, but as this change break existing code here, it can break more code. That is why I wrote to the list. doing PyArray_BASE(xview) = ptr work when I don't define NPY_NO_DEPRECATED_API, but do not work when I define NPY_NO_DEPRECATED_API. I would have expected the same for PyArray_BYTES/DATA. Do this explain clearly the problem I saw? Yes, thanks. I see in ndarraytypes.h #define PyArray_DATA(obj) ((void *)(((PyArrayObject_fields *)(obj))-data)) I wonder if the cast to void* is causing a problem? Could you try char* instead? Oops, the problem is that you need a pointer to the slot, not the pointer in the slot. That is, you need a different macro/function. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np 1.7b2 PyArray_BYTES(obj)=ptr fail
On Tue, Oct 2, 2012 at 1:44 PM, Frédéric Bastien no...@nouiz.org wrote: With numpy 1.6.2, it is working. So this is an interface change. Are you sure you want this? This break existing code. I do not understand what you mean by slot? Pythonese for structure member ;) I'm not sure is the PyArray_SWAP is a good long term idea. I would not make it if it is only for temporarily. The C++ stdlib provides something similar for std::vector. One common use case would be to pass in a vector by reference that gets swapped with one on the stack. When the function exits the one on the stack is cleaned up and the vector that was passed in has the new data, but it has to be the same type. For PyArray_SWAP I was thinking of swapping everything: type, dims, strides, data, etc. That is what f2py does. To set the base ptr, there is PyArray_SetBaseObject() fct that is new in 1.7. Is a similar fct useful in the long term for numpy? In the case where we implement differently the ndarray object, I think it won't be useful. We will also need to know how the memory is laid out by numpy for performance critical code. We we will need an attribute that tell the intern structure used. So do you want to force this interface change in numpy 1.7 so that I need to write codes now or can I wait to do it when you force the new interface? Well, no we don't want to force you to use the new interface. If you don't define NPY_NO_DEPRECATED_API things should still work. Although if it is defined the function returns an rvalue, so some other method needs to be provided for what you are doing. Currently the used code for PyArray_BYTES is: #define PyArray_BYTES(obj) ((char *)(((PyArrayObject_fields *)(obj))-data)) if I change it to #define PyArray_BYTES(obj) PyArrayObject_fields *)(obj))-data)) it work! I don't understand why removing the case make it work. the data field is already an (char*), so this should not make a difference to my underderstanding. But I'm missing something here, do someone know? What I find strange is that it is the same macro in 1.7 and 1.6, only the name of the structure was changed. Hmm... This looks almost like some compiler subtlety, I wonder if the compiler version/optimization flags have changed? In any case, I think the second form would be more correct for the lvalue since the structure member is, as you say, already a char*. We want things to work for you as they should, so we need to understand this and fix it. snip Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
