On Fri, Feb 5, 2010 at 5:15 AM, Pauli Virtanen <p...@iki.fi> wrote: > to, 2010-02-04 kello 12:09 -0700, Charles R Harris kirjoitti: >> Realistically, I don't think Py3k will be ready by April-May. Fall is >> probably doable and maybe there will be some things for a SOC person >> to work on this summer. > > Well, we have many components of Py3 support in place in the SVN > trunk. 1470 of 1983 unit tests currently pass. (Except that some recent > commit introduced some Py3-breaking Python code.) > > Main things missing: > > - String arrays, and other worms out of the str/unicode/bytes can. > I've audited many parts of the C code -- there are 66 remaining > unchecked points where PyString -> PyBytes is assumed without much > thinking (grep PyString). > > The Python side (= mainly the tests), however, needs more love. > Several tests break at the moment since array(['a']).dtype == 'U' > on Py3. > > - There's also the decision about whether 'S' == bytes or str. > If the former, the tests need fixing, if the latter, the code needs > fixing. > > - Consuming PEP 3118 buffer arrays so that shape and strides are > correctly acquired. This could in the end allow ufuncs to > operate on PEP 3118 arrays. > > The buffer provider support is already done, also for Py2.6. > > - Rewriting fromfile to use the low-level I/O. Py3 does not support > getting FILE* pointers out, and the handle from fdopen is currently > left dangling. > > That's certainly a manageable amount of work. For me, however, I believe > real life(TM) will still intrude a bit on my working on Numpy before the > summer, so I can't promise that by April-May all of the test suite > passes on Py3. With some luck it might be possible, though, and > certainly by fall. > > I hoped to have Py3 support the only bigger change in the next release, > so that the number of moving pieces of code would be kept down. It does > probably not require an ABI break. > > But if an ABI break is in the pipeline, it might make sense to put it > out before "officially" supporting Py3.
Thanks for the thorough report, it gives me a better idea of what is left to be done. > > *** > > So how to proceed? > > I would prefer if new code introduced in the rewrite would compile and > work correctly both on Py2 and Py3. I wouldn't expect this to be a high > overhead, and it would remove the need for a separate Py3 branch. I think a py3k buildbot would help for this, right ? Another thing is that the py3k changes do not work at all with Visual Studio compilers, but that's mostly cosmetic things (like #warning not being supported and things like that). > Most C code that works on Py2 works also on Py3. Py3 mainly means not > using PyString, but choosing between Unicode + Bytes + UString (=Bytes > for Py2 & Unicode for Py3). Also, it may be necessary to avoid FILE* > pointers in the API (on Py3 those are no longer as easily obtained), and > be wary when working with buffers. So once the py3k support is in place, should we deprecate those functions so that people interested in porting to py3k can plan in advance ? Getting rid of FILE* pointers and file descriptor would also helps quite a bit on windows. I know that at some point, there were some discussions to make the python C API safe to multiple C runtimes, but I cannot find any recent discussion on that fact. I should just ask on python-dev, I guess. This would be a great relief if we don't have to care about those issues anymore. > I assume the rewrite will be worked on a separate SVN branch? Also, is > there a plan yet on what needs changing to make Numpy's ABI more > resistant? There are two issues: - What we currently means by ABI, that is the ABI for a given python version. The main issue is the binary layout of the structures (I think the function ordering is pretty solid now, it is difficult to change it inadvertently). The only way to fix this is to hide the content of those structures, and define the structures in the C code instead (opaque pointer, also known as the pimpl idiom). This means a massive break of the C API, both internally and externally, but that's something that is really needed IMO. - Higher goal: ABI across python versions. This is motivated by PEP 384. It means avoiding calls to API which are not "safe". I have no idea whether it is possible, but that's something to keep in mind once we start a major overhaul. cheers, David _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
