Hi,
I pushed a series of code fixes to cython-unstable to support running the
compiler in Python 3.1 and to run the test suite as usual, just by calling
runtests.py. Enabling that made me realise that Cython still isn't Py3
clean, as there are still issues related to byte strings, mostly when
writing them out to the C code stream. A lovely example is that
"some unicode string formatted with %s" % some_byte_string
didn't work, as it results in this:
"some unicode string formatted with b'byte string content'"
being written. These problems appear in the weirdest and best hidden
places. For example, the "compile/c_directives" test fails because a
(defaulted?) buffer option pops up from the middle of nowhere that is (for
whatever reason) encoded as a byte string and gets passed on as a keyword
argument. In Py3, keyword argument names must be unicode, so you get a
beautiful compiler crash.
Another problem is string comparisons. Byte strings and unicode strings
never compare equal in Py3, even when they look exactly the same. So we get
a couple of test failures due to unknown or misinterpreted names, and a
couple of failing doctests because the output contains a 'b' prefix at some
point.
Regarding code generation: given that all generated code gets written as
Unicode strings in Py3, I decided to go the route of fake-decoding byte
strings from ISO-8859-1 to Unicode (which generates character code points
from 0-255 that are identical to the original byte sequence), and then
re-encoding the written C code as ISO-8859-1 on its way into the target
file to make them re-appear as the correct bytes. This is slightly hackish,
but everything else would largely complicate the way we write strings into
the code stream. To work around this, I gave the BytesLiteral class its own
__str__() method that returns a 'decoded' unicode string, so that %
formatting on unicode strings works now.
I'll keep carving my way through the code to find the places to fix.
However, I would already like to ask the other core developers to install
the official Python 3.1 release and to make a test run on that platform
part of your normal test cycle. It's easy to get things wrong, but now that
it's easy to run the test suite, I hope it'll also become easier to catch
unexpected problems before non-portable code gets committed.
It would be great if you could run the test suite on your side and skip
through the failures to check for tests and tested code that you are
familiar with. There are definitely tests where it would take me quite a
while to even finger point the place where things go wrong.
Stefan
_______________________________________________
Cython-dev mailing list
[email protected]
http://codespeak.net/mailman/listinfo/cython-dev
