Nick Coghlan writes: > GvR writes: > > Let's just define a Unicode string to be a sequence of code points and > > let libraries deal with the rest. Ok, methods like lower() should > > consider characters, but indexing/slicing should refer to code points. > > Same for '=='; we can have a library that compares by applying (or > > assuming?) certain normalizations. Tom C tells me that case-less > > comparison cannot use a.lower() == b.lower(); fine, we can add that > > operation to the library too. But this exceeds the scope of PEP 393, > > right? > > Yep, I was agreeing with you on this point - I think you're right that > if we provide a solid code point based core Unicode type (perhaps with > some character based methods), then library support can fill the gap > between handling code points and handling characters.
+1 I don't really see an alternative to this approach. The underlying array has to be exposed because there are too many applications that can take advantage of it, and analysis of decomposed characters requires it. Making that array be an array of code points is a really good idea, and Python already has that in the UCS-4 build. PEP 393 is "just" a space optimization that allows getting rid of the narrow build, with all its wartiness. > something to incorporate directly around the 3.4 timeframe (or > potentially even in 3.3, since it should already be possible to > develop such a wrapper based on UCS4 builds of 3.2) I agree that it's possible, but I estimate that it's not feasible for 3.3 because we don't yet know the requirements. This one really needs to ferment and mature in PyPI for a while because we just don't know how far the scope of user needs is going to extend. Bidi is a mudball[1], confusable character indexes sound like a cool idea for the web and email but is anybody really going to use them?, etc. > However, there may an important distinction to be made on the > Python-the-language vs CPython-the-implementation front: is another > implementation (e.g. PyPy) *allowed* to implement character based > indexing instead of code point based for 2.x unicode/3.x str type? Or > is the code point indexing part of the language spec, and any > character based indexing needs to be provided via a separate type or > module? +1 for language spec. Remember, there are cases in Unicode where you'd like to access base characters and the like. So you need to be able to get at individual code points in an NFD string. You shouldn't need to use different code for that in different implementations of Python. Footnotes: [1] Sure, we can implement the UAX#9 bidi algorithm, but it's not good enough by itself: something as simple as "File name (default {0}): ".format(name) can produce disconcerting results if the whole resulting string is treated by the UBA. Specifically, using the usual convention of uppercase letters being an RTL script, name = "ABCD" will result in the prompt: File name (default :(DCBA _ (where _ denotes the position of the insertion cursor). The Hebrew speakers on emacs-devel agreed that an example using a real Hebrew string didn't look right to them, either. _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com