Re: [Numpy-discussion] Numpy on Python3
On Mon, Nov 23, 2009 at 10:08 AM, Pauli Virtanen wrote: > Setup.py runs 2to3 automatically for all changed files. Of course, if it's > possible to cater for24 and 3 at the same time,that's good. How do you work > around the relative imports andthe changed exception catching syntax? > hi, see my tree for how I did these things. http://github.com/illume/numpy3k I'm not sure I like how I handled the changed relative imports. However the exception handling is fairly easy/clean to do. cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
awesome work Pauli! ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
On Mon, Nov 23, 2009 at 7:19 AM, David Cournapeau wrote: > On Mon, Nov 23, 2009 at 2:35 PM, Pauli Virtanen wrote: > >> It might be nice to have this merged in at some point after 1.4.0 (after >> the most obvious glaring bugs have been fixed), so that we could perhaps >> start aiming for Python3 compatibility in Numpy 1.5.0. > > One thing I have on my end is a numpy.distutils which runs under both > python 2 and 3, so that you don't have to run 2to3 everytime you make > a change. > > I did not put it in the trunk because I did not want to modify > numpy.distutils for 1.4.0 at this point, but I will include the > changes as soon as I branch the trunk into 1.4.0, > hi, my tree has this fairly far along (2 and 3 compatible). cheers, ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
Pauli Virtanen wrote: > Mon, 23 Nov 2009 08:58:47 +0100, Sturla Molden wrote: > >> Pauli Virtanen skrev: >> >>> XXX: 3K: numpy.random is disabled for now, uses PyString_* XXX: 3K: >>> numpy.ma is disabled for now -- some issues >>> >>> >> I thought numpy.random uses Cython? Is it just a matter of recompiling >> the pyx-file? >> > > The Cython file uses the C-api directly, so we'll need a .h file with the > necessary compile-time conditionals. > > >>> I remember Dag was working on this a bit: how far did it go? >>> >> Cython's include file numpy.pxd has an ndarray class that extend >> PyArrayObject with PEP 3118 buffer compatibility. >> > > Great! I believe I will just steal whatever I can and rewrite it in C -- > for now, it seems possible to keep Numpy's core in plain C. > I did sit down with David to learn enough to do this and had a brief start on doing it properly for NumPy on SciPy 2009 (with seperate testcases and the buffer format string stored directly in the NumPy dtype structs on creation). I meant to come back to it in November but due to becoming sick etc. etc. that's no longer possible. If nothing happens by the mid/end of January I still hope to be able to do this then. Feel free to ask any questions about the buffer PEP if you do go forward with this as I've used it a lot in Cython (and wrote the "implementation on behalf of NumPy" there). The Cython numpy.pxd does: - Temporarily allocate buffers for the format string, this is inefficient as NumPy can store them directly in the dtype when the dtype is constructed - Not support non-native endian (and some other relevant packing formats I believe) - Not support the string types - Not support dtypes with nested sub-arrays within records What is done: David added some code (at least in some branch) that ensures that sizeof(npy_intp) == sizeof(Py_ssize_t) on Python 2.6+. (I.e. if that assumption is violated NumPy won't compile, so we're free to assume it until the issue of using npy_intp for indices is fixed on a more fundamental level in NumPy). This means that the shapes/strides in Py_buffer can be directed directly to the dimensions/strides in the NumPy array struct (whereas Cython's numpy.pxd has to make a copy on some platforms for Python 2.4). Dag Sverre ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
On Nov 23, 2009, at 4:36 AM, Pauli Virtanen wrote: > Mon, 23 Nov 2009 01:40:00 -0500, Pierre GM wrote: > [clip] >>> XXX: 3K: numpy.ma is disabled for now -- some issues >> >> What are the issues ? > > Something resolving which would have taken more than 5 minutes :) > Possibly because something that ma depends on is currently broken in > numpy.core. Fair enough, fair enough... But y'all, please let me know potential issues (not that I'll be able to work on it anytime soon, but just in case...). P. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
Mon, 23 Nov 2009 08:58:47 +0100, Sturla Molden wrote: > Pauli Virtanen skrev: >> XXX: 3K: numpy.random is disabled for now, uses PyString_* XXX: 3K: >> numpy.ma is disabled for now -- some issues >> > I thought numpy.random uses Cython? Is it just a matter of recompiling > the pyx-file? The Cython file uses the C-api directly, so we'll need a .h file with the necessary compile-time conditionals. >> I remember Dag was working on this a bit: how far did it go? > > Cython's include file numpy.pxd has an ndarray class that extend > PyArrayObject with PEP 3118 buffer compatibility. Great! I believe I will just steal whatever I can and rewrite it in C -- for now, it seems possible to keep Numpy's core in plain C. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
Mon, 23 Nov 2009 01:40:00 -0500, Pierre GM wrote: [clip] >> XXX: 3K: numpy.ma is disabled for now -- some issues > > What are the issues ? Something resolving which would have taken more than 5 minutes :) Possibly because something that ma depends on is currently broken in numpy.core. I just wanted to breeze through and arrive as fast as possible at something that can be imported and works for simple things, so I didn't stop at anything that would take longer. I'll take a look at the rest later on. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
Pauli Virtanen wrote: > Setup.py runs 2to3 automatically for all changed files. Yes, but I think it is more practical to have the build process to be 2 and 3-compatible. > Of course, if it's possible to cater for24 and 3 at the same time,that's > good. How do you work around the relative imports andthe changed exception > catching syntax? > For the exception catching, one can have a few utilities to walk through the stack - I don't remember how I did it for the relative import, but this was not a pb IIRC. I am quite disappointed that numscons will not be usable in the foreseable future for py3k, I hoped it would have been simpler than numpy.distutils, but porting scons to py3k is too big of a task. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
The issue with longs is that we wouldn't want array([1,2,3]) create object arrays -- so we need to decide on casting rules for longs. Currently, I think they're treated like python2 ints. -- alkuper. viesti -- Aihe: Re: [Numpy-discussion] Numpy on Python3 Lähettäjä: Charles R Harris Päivämäärä: 23.11.2009 08:08 On Sun, Nov 22, 2009 at 10:35 PM, Pauli Virtanen wrote: > http://github.com/pv/numpy-work/tree/py3k > > $ mkdir -p $PWD/dist/lib/python3.1/site-packages > $ python3 setup.py install --prefix=$PWD/dist > $ cd $PWD/dist/lib/python3.1/site-packages && python3 > Python 3.1.1+ (r311:74480, Oct 11 2009, 20:22:16) > [GCC 4.4.1] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > XXX: 3K: numpy.random is disabled for now, uses PyString_* > XXX: 3K: numpy.ma is disabled for now -- some issues > >>> numpy.array([1., 2, 3, 4]) > array([ 1., 2., 3., 4.]) > >>> _ + 10 > array([ 11., 12., 13., 14.]) > >>> numpy.ones((4,), dtype=complex)/4 > array([ 0.25+0.j, 0.25+0.j, 0.25+0.j, 0.25+0.j]) > >>> numpy.array([object(), object()]) > array([, ], > dtype=b'object') > >*** > > Things were fairly straightforward this far, just many tiny changes. > What's left is then sorting out the bigger problems :) > > This is still far from being complete: > > - Most use of PyString_* needs auditing (note e.g. the b'object' in the > dtype print above...). > > I simply added convenience wrappers for PyString -> PyBytes, > but this is not the correct choice at all points. > > - Also, should dtype='S' be Bytes or Unicode? I chose Bytes for now. > > - Whether to inherit Numpy ints from PyLong_* needs some thinking, > as they are quite different objects. Now, I dropped the inheritance, > but I wonder if this will break something. > > Maybe. But it was always a hassle because it behaved differently than the other integer types. Now onto float ;) > - PyFile_AsFile has disappeared, and AsFileDescriptor+fdopen doesn't > seem to cut it -- don't know exactly what's wrong here. > > - Integer -> String formatting does not seem to work > > - Larger-than-long-long Python ints probably cause problems > > We used a python call which would raise an error if the number was too large. If that call is still valid, things should work. > - The new buffer interface needs to be implemented -- currently there > are just error-raising stubs. > > I remember Dag was working on this a bit: how far did it go? > > - Relative imports + 2to3 is a bit of a pain. A pity we can't have > them in the mainline code because of python2.4. > > - I didn't check for semantic changes in tp_* interface functions. > This we need still to do. > > I left some notes in the src folder. If you discover anything new put it in there. > - And probably many other issues lurking. > > We do need to look at the initialization of the type math stuff in the ufuncobject module. Yeah, its a bit of a circular dependency, one reason it would be nice to have ufuncs operate on buffer objects instead of ndarrays would be to break the mutual dependence. > > Also, I didn't yet try checking how far the test suite passes on > Python3. (It still passes completely on Python2, so at least I didn't > break that part.) > > It might be nice to have this merged in at some point after 1.4.0 (after > the most obvious glaring bugs have been fixed), so that we could perhaps > start aiming for Python3 compatibility in Numpy 1.5.0. > > If you want to see real suffering, look at the programmer notes in the hacked CRU files. Some poor sucker was stuck with fixing up the g*dawful code while also needing to determine what data was in undocumented binary files, some with the same names but containing different data. Folks, don't let that happen to you. The conversion to Py3k is going to be a breeze by comparison. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
Setup.py runs 2to3 automatically for all changed files. Of course, if it's possible to cater for24 and 3 at the same time,that's good. How do you work around the relative imports andthe changed exception catching syntax? -- alkuper. viesti -- Aihe: Re: [Numpy-discussion] Numpy on Python3 Lähettäjä: David Cournapeau Päivämäärä: 23.11.2009 08:19 On Mon, Nov 23, 2009 at 2:35 PM, Pauli Virtanen wrote: > It might be nice to have this merged in at some point after 1.4.0 (after > the most obvious glaring bugs have been fixed), so that we could perhaps > start aiming for Python3 compatibility in Numpy 1.5.0. One thing I have on my end is a numpy.distutils which runs under both python 2 and 3, so that you don't have to run 2to3 everytime you make a change. I did not put it in the trunk because I did not want to modify numpy.distutils for 1.4.0 at this point, but I will include the changes as soon as I branch the trunk into 1.4.0, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
Pauli Virtanen skrev: > XXX: 3K: numpy.random is disabled for now, uses PyString_* > XXX: 3K: numpy.ma is disabled for now -- some issues > I thought numpy.random uses Cython? Is it just a matter of recompiling the pyx-file? > I remember Dag was working on this a bit: how far did it go? > > Cython's include file numpy.pxd has an ndarray class that extend PyArrayObject with PEP 3118 buffer compatibility. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
On Nov 23, 2009, at 12:35 AM, Pauli Virtanen wrote: > http://github.com/pv/numpy-work/tree/py3k > > $ mkdir -p $PWD/dist/lib/python3.1/site-packages > $ python3 setup.py install --prefix=$PWD/dist > $ cd $PWD/dist/lib/python3.1/site-packages && python3 > Python 3.1.1+ (r311:74480, Oct 11 2009, 20:22:16) > [GCC 4.4.1] on linux2 > Type "help", "copyright", "credits" or "license" for more information. import numpy > XXX: 3K: numpy.random is disabled for now, uses PyString_* > XXX: 3K: numpy.ma is disabled for now -- some issues What are the issues ? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
On Sun, Nov 22, 2009 at 10:35 PM, Pauli Virtanen wrote: > http://github.com/pv/numpy-work/tree/py3k > > $ mkdir -p $PWD/dist/lib/python3.1/site-packages > $ python3 setup.py install --prefix=$PWD/dist > $ cd $PWD/dist/lib/python3.1/site-packages && python3 > Python 3.1.1+ (r311:74480, Oct 11 2009, 20:22:16) > [GCC 4.4.1] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > XXX: 3K: numpy.random is disabled for now, uses PyString_* > XXX: 3K: numpy.ma is disabled for now -- some issues > >>> numpy.array([1., 2, 3, 4]) > array([ 1., 2., 3., 4.]) > >>> _ + 10 > array([ 11., 12., 13., 14.]) > >>> numpy.ones((4,), dtype=complex)/4 > array([ 0.25+0.j, 0.25+0.j, 0.25+0.j, 0.25+0.j]) > >>> numpy.array([object(), object()]) > array([, ], > dtype=b'object') > >*** > > Things were fairly straightforward this far, just many tiny changes. > What's left is then sorting out the bigger problems :) > > This is still far from being complete: > > - Most use of PyString_* needs auditing (note e.g. the b'object' in the > dtype print above...). > > I simply added convenience wrappers for PyString -> PyBytes, > but this is not the correct choice at all points. > > - Also, should dtype='S' be Bytes or Unicode? I chose Bytes for now. > > Yeah, this needs deciding. I think compatibility with old files requires Bytes, we can't change file formats and keep compatibility. BTW, I made some changes for py3k that depend on the macro NPY_PY3K being defined, but didn't actually define it anywhere. It needs to go in one of the include files. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
On Mon, Nov 23, 2009 at 2:35 PM, Pauli Virtanen wrote: > It might be nice to have this merged in at some point after 1.4.0 (after > the most obvious glaring bugs have been fixed), so that we could perhaps > start aiming for Python3 compatibility in Numpy 1.5.0. One thing I have on my end is a numpy.distutils which runs under both python 2 and 3, so that you don't have to run 2to3 everytime you make a change. I did not put it in the trunk because I did not want to modify numpy.distutils for 1.4.0 at this point, but I will include the changes as soon as I branch the trunk into 1.4.0, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
On Sun, Nov 22, 2009 at 10:35 PM, Pauli Virtanen wrote: > http://github.com/pv/numpy-work/tree/py3k > > $ mkdir -p $PWD/dist/lib/python3.1/site-packages > $ python3 setup.py install --prefix=$PWD/dist > $ cd $PWD/dist/lib/python3.1/site-packages && python3 > Python 3.1.1+ (r311:74480, Oct 11 2009, 20:22:16) > [GCC 4.4.1] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > XXX: 3K: numpy.random is disabled for now, uses PyString_* > XXX: 3K: numpy.ma is disabled for now -- some issues > >>> numpy.array([1., 2, 3, 4]) > array([ 1., 2., 3., 4.]) > >>> _ + 10 > array([ 11., 12., 13., 14.]) > >>> numpy.ones((4,), dtype=complex)/4 > array([ 0.25+0.j, 0.25+0.j, 0.25+0.j, 0.25+0.j]) > >>> numpy.array([object(), object()]) > array([, ], > dtype=b'object') > >*** > > Things were fairly straightforward this far, just many tiny changes. > What's left is then sorting out the bigger problems :) > > This is still far from being complete: > > - Most use of PyString_* needs auditing (note e.g. the b'object' in the > dtype print above...). > > I simply added convenience wrappers for PyString -> PyBytes, > but this is not the correct choice at all points. > > - Also, should dtype='S' be Bytes or Unicode? I chose Bytes for now. > > - Whether to inherit Numpy ints from PyLong_* needs some thinking, > as they are quite different objects. Now, I dropped the inheritance, > but I wonder if this will break something. > > Maybe. But it was always a hassle because it behaved differently than the other integer types. Now onto float ;) > - PyFile_AsFile has disappeared, and AsFileDescriptor+fdopen doesn't > seem to cut it -- don't know exactly what's wrong here. > > - Integer -> String formatting does not seem to work > > - Larger-than-long-long Python ints probably cause problems > > We used a python call which would raise an error if the number was too large. If that call is still valid, things should work. > - The new buffer interface needs to be implemented -- currently there > are just error-raising stubs. > > I remember Dag was working on this a bit: how far did it go? > > - Relative imports + 2to3 is a bit of a pain. A pity we can't have > them in the mainline code because of python2.4. > > - I didn't check for semantic changes in tp_* interface functions. > This we need still to do. > > I left some notes in the src folder. If you discover anything new put it in there. > - And probably many other issues lurking. > > We do need to look at the initialization of the type math stuff in the ufuncobject module. Yeah, its a bit of a circular dependency, one reason it would be nice to have ufuncs operate on buffer objects instead of ndarrays would be to break the mutual dependence. > > Also, I didn't yet try checking how far the test suite passes on > Python3. (It still passes completely on Python2, so at least I didn't > break that part.) > > It might be nice to have this merged in at some point after 1.4.0 (after > the most obvious glaring bugs have been fixed), so that we could perhaps > start aiming for Python3 compatibility in Numpy 1.5.0. > > If you want to see real suffering, look at the programmer notes in the hacked CRU files. Some poor sucker was stuck with fixing up the g*dawful code while also needing to determine what data was in undocumented binary files, some with the same names but containing different data. Folks, don't let that happen to you. The conversion to Py3k is going to be a breeze by comparison. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Numpy on Python3
http://github.com/pv/numpy-work/tree/py3k $ mkdir -p $PWD/dist/lib/python3.1/site-packages $ python3 setup.py install --prefix=$PWD/dist $ cd $PWD/dist/lib/python3.1/site-packages && python3 Python 3.1.1+ (r311:74480, Oct 11 2009, 20:22:16) [GCC 4.4.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy XXX: 3K: numpy.random is disabled for now, uses PyString_* XXX: 3K: numpy.ma is disabled for now -- some issues >>> numpy.array([1., 2, 3, 4]) array([ 1., 2., 3., 4.]) >>> _ + 10 array([ 11., 12., 13., 14.]) >>> numpy.ones((4,), dtype=complex)/4 array([ 0.25+0.j, 0.25+0.j, 0.25+0.j, 0.25+0.j]) >>> numpy.array([object(), object()]) array([, ], dtype=b'object') *** Things were fairly straightforward this far, just many tiny changes. What's left is then sorting out the bigger problems :) This is still far from being complete: - Most use of PyString_* needs auditing (note e.g. the b'object' in the dtype print above...). I simply added convenience wrappers for PyString -> PyBytes, but this is not the correct choice at all points. - Also, should dtype='S' be Bytes or Unicode? I chose Bytes for now. - Whether to inherit Numpy ints from PyLong_* needs some thinking, as they are quite different objects. Now, I dropped the inheritance, but I wonder if this will break something. - PyFile_AsFile has disappeared, and AsFileDescriptor+fdopen doesn't seem to cut it -- don't know exactly what's wrong here. - Integer -> String formatting does not seem to work - Larger-than-long-long Python ints probably cause problems - The new buffer interface needs to be implemented -- currently there are just error-raising stubs. I remember Dag was working on this a bit: how far did it go? - Relative imports + 2to3 is a bit of a pain. A pity we can't have them in the mainline code because of python2.4. - I didn't check for semantic changes in tp_* interface functions. This we need still to do. - And probably many other issues lurking. Also, I didn't yet try checking how far the test suite passes on Python3. (It still passes completely on Python2, so at least I didn't break that part.) It might be nice to have this merged in at some point after 1.4.0 (after the most obvious glaring bugs have been fixed), so that we could perhaps start aiming for Python3 compatibility in Numpy 1.5.0. Cheers, Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
