[issue12568] Add functions to get the width in columns of a character
Tom Christiansen added the comment: I would encourage you to look at the Perl CPAN module Unicode::LineBreak, which fully implements tr11. It includes Unicode::GCString, a class that has a columns() method to determine the print columns. This is very fancy in the case of Asian widths, but of course there are many other cases too. If you'd like, I can show you a program that uses these, a rewrite the standard Unix fmt(1) filter that works properly on Unicode column widths. --tom -- ___ Python tracker <http://bugs.python.org/issue12568> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12568] Add functions to get the width in columns of a character
Tom Christiansen added the comment: >Martin v. L=C3=B6wis added the comment: >> Martin, I think you meant to write "if w =3D=3D 'A':". >> Some very common characters have ambiguous widths though (e.g. the Greek = >alphabet), so you can't just raise an error for them. >That's precisely why I don't think this should be in the library, but >in the application. Application developers who need that also need >to concern themselves with the border cases, and decide on how >they need to resolve them. The column-width of a string is not an application issue. It is well-defined by Unicode. Again, please see how we've done it in Perl, where tr11 is fully implemented. The columns() method from Unicode::GCString always gives the right answer per the Standard for any string, even what you are calling ambiguous ones. This is not an applications issue -- at all. --tom -- ___ Python tracker <http://bugs.python.org/issue12568> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12568] Add functions to get the width in columns of a character
Tom Christiansen added the comment: >Martin v. L=C3=B6wis added the comment: >> I would encourage you to look at the Perl CPAN module Unicode::LineBreak, >> which fully implements tr11. >Thanks for the pointer! >> If you'd like, I can show you a program that uses these, a rewrite the >> standard Unix fmt(1) filter that works properly on Unicode column widths. >I believe there can't be any truly "proper" implementation, as you >can't be certain how the terminal will handle these itself. Hm. I think we may not be talking about the same thing after all. If we're talking about the Curses library, or something similar, this is not the same. I do not think Curses has support for combining characters, right to left text, wide characters, etc. However, Unicode does, and defines the column width for those. I have an illustration of what this looks like in the picture in the very last recipe, #44, in http://training.perl.com/scripts/perlunicook.html That is what I have been talking about by print widths. It's running in a Mac terminal emulator, and unlike the HTML which grabs from too many fonts, the terminal program does the right thing with the widths. Are we talking about different things? --tom -- ___ Python tracker <http://bugs.python.org/issue12568> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12728] Python re lib fails case insensitive matches on Unicode data
New submission from Tom Christiansen : The Python re library is broken in its approach to case-insensitive matches. It erroneously attempts to compare lowercase mappings. This is wrong. You must compare the Unicode casefolds, not the Unicode casemaps. Otherwise you get wrong answers. I include a small test case that illustrates this bug. The bug exists on both 2.7 and 3.2, and on both wide builds and narrow builds. For comparison, I also show results using Matthew Barnett's regex library, which gets all 5 tests correct where re gets all 5 tests wrong. A sample run is: FAIL: repattern Ι isnot the same as string ͅ PASS: regex pattern Ι is indeed the same as string ͅ FAIL: repattern Μ isnot the same as string µ PASS: regex pattern Μ is indeed the same as string µ FAIL: repattern ſ isnot the same as string s PASS: regex pattern ſ is indeed the same as string s FAIL: repattern ΣΤΙΓΜΑΣ isnot the same as string στιγμας PASS: regex pattern ΣΤΙΓΜΑΣ is indeed the same as string στιγμας FAIL: repattern POST isnot the same as string poſt PASS: regex pattern POST is indeed the same as string poſt relib passed 0 of 5 tests regex lib passed 5 of 5 tests -- components: Library (Lib) files: sigmata.python messages: 141916 nosy: tchrist priority: normal severity: normal status: open title: Python re lib fails case insensitive matches on Unicode data versions: Python 2.7 Added file: http://bugs.python.org/file22879/sigmata.python ___ Python tracker <http://bugs.python.org/issue12728> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12729] Python lib re cannot handle Unicode properly due to narrow/wide bug
New submission from Tom Christiansen : Python is in flagrant violation of the very most basic premises of Unicode Technical Report #18 on Regular Expressions, which requires that a regex engine support Unicode characters as "basic logical units independent of serialization like UTF‑*". Because sometimes you must specify ".." to match a single Unicode character -- whenever those code points are above the BMP and you are on a narrow build -- Python regexes cannot be reliably used for Unicode text. % python3.2 Python 3.2 (r32:88445, Jul 21 2011, 14:44:19) [GCC 4.2.1 (Apple Inc. build 5664)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import re >>> g = "\N{GREEK SMALL LETTER ALPHA WITH VARIA AND YPOGEGRAMMENI}" >>> print(g) ᾲ >>> print(re.search(r'\w', g)) <_sre.SRE_Match object at 0x10051f988> >>> p = "\N{MATHEMATICAL SCRIPT CAPITAL P}" >>> print(p) 𝒫 >>> print(re.search(r'\w', p)) None >>> print(re.search(r'..', p)) # ← 𝙏𝙃𝙄𝙎 𝙄𝙎 𝙏𝙃𝙀 𝙑𝙄𝙊𝙇𝘼𝙏𝙄𝙊𝙉 𝙍𝙄𝙂𝙃𝙏 𝙃𝙀𝙍𝙀 <_sre.SRE_Match object at 0x10051f988> >>> print(len(chr(0x1D4AB))) 2 That is illegal in Unicode regular expressions. -- components: Regular Expressions messages: 141917 nosy: tchrist priority: normal severity: normal status: open title: Python lib re cannot handle Unicode properly due to narrow/wide bug type: behavior versions: Python 2.7 ___ Python tracker <http://bugs.python.org/issue12729> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12730] Python's casemapping functions are untrustworthy due to narrow/wide build issues
New submission from Tom Christiansen : You cannot use Python's casemapping functions on Unicode data because they fail on narrow builds. This makes it impossible to write portable code in Python that can cope with full Unicode. I've tried several times to submit this bug, but the file selection widget blows up. I believe it was an Opera bug because I had a write lock on the file. One more time. -- components: Unicode files: casemaps.python messages: 141918 nosy: tchrist priority: normal severity: normal status: open title: Python's casemapping functions are untrustworthy due to narrow/wide build issues type: behavior versions: Python 2.7 Added file: http://bugs.python.org/file22880/casemaps.python ___ Python tracker <http://bugs.python.org/issue12730> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12731] python lib re uses obsolete sense of \w in full violation of UTS#18 RL1.2a
New submission from Tom Christiansen : You cannot use Python's lib re for handling Unicode regular expressions because it violates the standard set out for the same in UTS#18 on Unicode Regular Expressions in RL1.2a on compatibility properties. What \w is allowed to match is clearly explained there, but Python has its own idea. Because it is in clear violation of the standard, it is misleading and wrong for Python to claim that the re.UNICODE flag makes \w and friends match Unicode. Here are the failed test cases when the attached file is run under v3.2; there are further failures when run under v2.7. FAIL lib refound non alphanumeric string café FAIL lib refound non alphanumeric string Ⓚ FAIL lib refound non alphanumeric string ͅ FAIL lib refound non alphanumeric string ְ FAIL lib refound non alphanumeric string 𝟘 FAIL lib refound non alphanumeric string 𐍁 FAIL lib refound non alphanumeric string 𝔘𝔫𝔦𝔠𝔬𝔡𝔢 FAIL lib refound non alphanumeric string 𐐔𐐯𐑅𐐨𐑉𐐯𐐻 FAIL lib refound non alphanumeric string connector‿punctuation FAIL lib refound non alphanumeric string Ὰͅ_Στο_Διάολο FAIL lib refound non alphanumeric string 𐌰𐍄𐍄𐌰‿𐌿𐌽𐍃𐌰𐍂‿𐌸𐌿‿𐌹𐌽‿𐌷𐌹𐌼𐌹𐌽𐌰𐌼 FAIL lib refound all alphanumeric string ¹²³ FAIL lib refound all alphanumeric string ₁₂₃ FAIL lib refound all alphanumeric string ¼½¾ FAIL lib refound all alphanumeric string ⑶ Note that Matthew Barnett's regex lib for Python handles all of these cases in comformance with The Unicode Standard. -- components: Regular Expressions files: alnum.python messages: 141920 nosy: tchrist priority: normal severity: normal status: open title: python lib re uses obsolete sense of \w in full violation of UTS#18 RL1.2a type: behavior versions: Python 2.7 Added file: http://bugs.python.org/file22881/alnum.python ___ Python tracker <http://bugs.python.org/issue12731> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12732] Can't portably use Unicode in Python identifiers
New submission from Tom Christiansen : You cannot reliably use Unicode in Python identifiers because of the narrow/wide build issue. The enclosed file is fine on wide builds but gets compiler errors on narrow ones during compilation. Go, Ruby, Java, and Perl all handle this situation without any problem; only Python has the bug. -- components: Interpreter Core files: badidents.python messages: 141923 nosy: tchrist priority: normal severity: normal status: open title: Can't portably use Unicode in Python identifiers type: behavior versions: Python 3.2 Added file: http://bugs.python.org/file22882/badidents.python ___ Python tracker <http://bugs.python.org/issue12732> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12728] Python re lib fails case insensitive matches on Unicode data
Changes by Tom Christiansen : -- components: +Regular Expressions -Library (Lib) type: -> behavior ___ Python tracker <http://bugs.python.org/issue12728> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12733] Request for grapheme support in Python re lib
New submission from Tom Christiansen : Without proper grapheme support in the regular expression library, it is impossible to correctly process Unicode. And the very least, one needs the \X escape supported, which is an extended grapheme cluster per UTS#18. This escape is supported by many regex libraries, include Perl's own and of course PCRE (and thence PHP, the standard ICU library, and Matthew Barnett's replacement regex library for Python. How do you process a string by graphemes if you cannot split on \X? How can you avoid splitting a grapheme into silly pieces if you cannot match one? How do I match the letter O no matter what diacritics have been applied to it otherwise? A match of (?=O)\X against an NFD string is by far the simplest and best way. This is necessary for a wide variety of reasons. Adding \pM and \PM go a little ways, but not far enough, because that is not how grapheme clusters are defined. You need a proper \X. -- components: Regular Expressions messages: 141924 nosy: tchrist priority: normal severity: normal status: open title: Request for grapheme support in Python re lib type: feature request versions: Python 3.2 ___ Python tracker <http://bugs.python.org/issue12733> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12734] Request for property support in Python re lib
New submission from Tom Christiansen : Python supports no Unicode properties in its re library, making it unsuitable for work with Unicode. This is therefore a formal request for the Python re library to support Unicode properties. The eleven properties required by Unicode Technical Report #18's RL1.2 are the bare minimum which must be added to make it possible to use Python reguyar expressions on Unicode. The proposed RL2.7 on Full Properties is even better. That is found at http://unicode.org/reports/tr18/proposed.html#Full_Properties Although by the time you read this, it will have been made an official part of tr18. Matthew Barnett's replacement library for re, called regex, support 67 Unicode properties at last count, including the strongly recommended loose matching. The standard re library needs to be spiffed up to make it suitable for Unicode processing; it is not currently usable for that due to this missing functionality. I quote from the Level 1 conformance requirement of tr18: "Level 1: This is a minimal level for useful Unicode support. It does not account for end-user expectations for character support, but does satisfy most low-level programmer requirements. The results of regular expression matching at this level are independent of country or language. At this level, the user of the regular expression engine would need to write more complicated regular expressions to do full Unicode processing." pass RL1.1 Hex Notation fail RL1.2 Properties fail RL1.2a Compatibility Properties fail RL1.3 Subtraction and Intersection fail RL1.4 Simple Word Boundaries fail RL1.5 Simple Loose Matches fail RL1.6 Line Boundaries fail RL1.7 Supplementary Code Points (withdrawn) RL2.1 Canonical Equivalents fail RL2.2 Extended Grapheme Clusters fail RL2.3 Default Word Boundaries fail RL2.4 Default Case Conversion pass RL2.5 Name Properties fail RL2.6 Wildcards in Property Values fail RL2.7 Full Properties I won’t even talk about Level 3. ICU, Perl, and Java7 all meet Level One conformance requirements with several Level 2 requirements also met. It is important for Python to meet the Unicode Standard in this so that people can use Python for regex matching Unicode text. They currently cannot usefully do so per the requirements of tr18. -- components: Regular Expressions messages: 141925 nosy: tchrist priority: normal severity: normal status: open title: Request for property support in Python re lib type: feature request versions: Python 3.2 ___ Python tracker <http://bugs.python.org/issue12734> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12735] request full Unicode collation support in std python library
New submission from Tom Christiansen : Python has no standard support for the Unicode Collation Library as explained in UTS #10. This is request that UCA library be added to the standard Python distribution. Collation underlies virtually everything we do with text, not just sorting but any sort of comparison. Furthermore, the UCA is tailorable for locales in a portable way that does not require dodgy vendor support. It is a very important step in making Python suitable for full Unicode text processing. -- components: Library (Lib) messages: 141926 nosy: tchrist priority: normal severity: normal status: open title: request full Unicode collation support in std python library type: feature request versions: Python 3.2 ___ Python tracker <http://bugs.python.org/issue12735> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12736] Request for python casemapping functions to use full not simple casemaps per Unicode's recommendation
New submission from Tom Christiansen : Python's casemapping functions only use what Unicode calls simple casemaps. These are only appropriate for functions that operate on single characters alone, not for those that operate on strings. The reason for this is that you get much better results with full casemapping. Java, Ruby, and Perl all do full casemapping for their equivalent functions that do string mapping, and Python should, too. I include a program that has a much of mappings and foldings both simple and full. Yes, it was machine-generated. -- components: Library (Lib) files: mux.python messages: 141928 nosy: tchrist priority: normal severity: normal status: open title: Request for python casemapping functions to use full not simple casemaps per Unicode's recommendation type: feature request versions: Python 3.2 Added file: http://bugs.python.org/file22883/mux.python ___ Python tracker <http://bugs.python.org/issue12736> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12737] string.title() is overzealous by upcasing combining marks inappropriately
New submission from Tom Christiansen : Python's string.title() function claims it titlecases the first letter in each word and lowercases the rest. However, this is not true. It is not using either of the two word detection algorithms that Unicode provides. One allows you to use a legacy \w+, where \w means any Alphabetic, Mark, Decimal Number, or Connector Punctuation (see UTS#18 Annex C: Compatibility Properties), and the other uses the more sophisticated word-break provided by the Word_Break properties such as Word_Break=MidNumLet Python is using neither of these, so gets the wrong answer. titlecase of déme un café should be Déme Un Café not DéMe Un Café titlecase of i̇stanbul should be İstanbul not İStanbul titlecase of ᾲ στο διάολο should be Ὰͅ Στο Διάολο not ᾺΙ Στο ΔιάΟλο Because those are in NFD form, you get different answers than if they are in NFC. That is not right. You should get the same answer. The bug is you aren't using the right definition for \w, and so get screwed up. This is likely related to issue 12731. In the enclosed tester file, which fails 4 out of its 6 tests, there is also a bug shown with this failed result: titlecase of 𐐼𐐯𐑅𐐨𐑉𐐯𐐻 should be 𐐔𐐯𐑅𐐨𐑉𐐯𐐻 not 𐐼𐐯𐑅𐐨𐑉𐐯𐐻 That one is related to issue 12730. See the attached tester, which was run under Python 3.2. As far as I can tell, these bugs exist in all python versions. -- files: titletest.python messages: 141929 nosy: tchrist priority: normal severity: normal status: open title: string.title() is overzealous by upcasing combining marks inappropriately type: behavior versions: Python 3.2 Added file: http://bugs.python.org/file22884/titletest.python ___ Python tracker <http://bugs.python.org/issue12737> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12734] Request for property support in Python re lib
Tom Christiansen added the comment: I've been a lot of testing of Matthew's regex library against UTS#18 issues, but only somewhat incidentally testing re. To use regex, one has to accept that certain things will work differently than they work in re, because he is following Unicode definitions for things like casefolding. But I doubt that is the sort of difference you are talking about. One of the things that Java, Go, and Perl all do is run regression tests against the whole Unicode Character Database to make sure nothing gets hosed, missed, or otherwise out of sync. That might a sort of regression test you might like to add. -- ___ Python tracker <http://bugs.python.org/issue12734> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12568] Add functions to get the width in columns of a character
Tom Christiansen added the comment: I can attest that being able to get the columns of a grapheme cluster is very important for printing, because you need this to do correct linebreaking. There might be something you can steal from http://search.cpan.org/perldoc?Unicode::GCString http://search.cpan.org/perldoc?Unicode::LineBreak which implements UAX#14 on linebreaking and UAX#11 on East Asian widths. I use this in my own code to help format Unicode strings my columns or lines. The right way would be to build this sort of knowledge into string.format(), but that is much harder, so an intermediary library module seems good enough for now. -- nosy: +tchrist ___ Python tracker <http://bugs.python.org/issue12568> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11230] "Full unicode import system" not in 3.2
Tom Christiansen added the comment: How does this work for modules that have filesystem names different from the one used for import? The issue I'm thinking about is that the Mac HSF+ filesystem keeps its Unicode filenames in (close to) NFD form. That means that a module named "caf\N{LATIN SMALL LETTER E WITH ACUTE}" with 4 graphemes and 4 code points in its name winds up in the filesystem as "cafe\N{COMBINING ACUTE ACCENT}" still with 4 graphemes but now with 5 code points. I believe (well, suspect; I have empirical evidence not proof) Python stores its own identifiers in NFD, so this may not be quite as much of a problem as it might otherwise be. Nonetheless, I have had users complain about what HFS+ does with such filenames, although I am not quite sure why. I think it’s because they access a file with 4 chars but they need a 5-char fileglob to wildcard it, so touch "caf\N{LATIN SMALL LETTER E WITH ACUTE}" and then you need a wildcard of "?" with an extra ? to find it. Kinda weird. -- nosy: +tchrist ___ Python tracker <http://bugs.python.org/issue11230> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2857] add codec for java modified utf-8
Tom Christiansen added the comment: Please do not call this "utf-8-java". It is called "cesu-8" per UTS#18 at: http://unicode.org/reports/tr26/ CESU-8 is *not* a a valid Unicode Transform Format and should not be called UTF-8. It is a real pain in the butt, caused by people who misunderand Unicode mis-encoding UCS-2 into UTF-8, screwing it up. I understand the need to be able to read it, but call it what it is, please. Despite the talk about Lucene, I note that the Perl port of Lucene uses real UTF-8, not CESU-8. -- nosy: +tchrist ___ Python tracker <http://bugs.python.org/issue2857> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11230] "Full unicode import system" not in 3.2
Tom Christiansen added the comment: Whoops, I meant that it appears that Python runs its identifiers through NFC. How that gets along with a filesystem that has quasi-NFD filenames I'm not sure, but it seems like it might be a variant of the case-insensitivity issue in filenames. -- ___ Python tracker <http://bugs.python.org/issue11230> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12728] Python re lib fails case insensitive matches on Unicode data
Tom Christiansen added the comment: > Terry J. Reedy added the comment: > I am not sure that everyone will agree that this is a bug, rather than a fe= > ature request, or that if a bug, that it should be changed in existing rele= > ases and possibly break running code. The doc just says, somewhat vaguely, = > that IGNORECASE "works for Unicode characters as expected". I have added ot= > hers as nosy for their opinions. Working as expected for Unicode characters means it must the Unicode's rules for casefolding. Otherwise you don't have Unicode at all; you just have ISO 10646. Unicode is not merely a larger character repertoire; again, that is merely ISO 10646. Unicode is all about the rules for processing this larger repertoire. This is a very common mistake, so common that it is in the Unicode FAQ: Q: What is the relation between ISO/IEC 10646 and Unicode? A: In 1991, the ISO Working Group responsible for ISO/IEC 10646 (JTC 1/SC 2/WG 2) and the Unicode Consortium decided to create one universal standard for coding multilingual text. Since then, the ISO 10646 Working Group (SC 2/WG 2) and the Unicode Consortium have worked together very closely to extend the standard and to keep their respective versions synchronized. [EH] Q: So are they the same thing? A: No. Although the character codes and encoding forms are synchronized between Unicode and ISO/IEC 10646, the Unicode Standard imposes additional constraints on implementations to ensure that they treat characters uniformly across platforms and applications. To this end, it supplies an extensive set of functional character specifications, character data, algorithms and substantial background material that is *not* in ISO/IEC 10646. http://unicode.org/faq/unicode_iso.html Part of those functional character specifications can be found in the three casefolding fields of the file UnicodeData.txt and also in two auxiliary files of the Unicode distribution, CaseFolding.txt and SpecialCasing.txt. The Unicode Character Database is not optional. If you do not use it, you do not have Unicode; instead you merely have ISO 10646, which is of zero practical use to anyone compared with Unicode. I'm sure that Python would not want to be stuck having something of no use to anyone when everyone else actually supports Unicode. One is not allowed to make up one's own rules that run counter to Unicode's and still make the claim that one is working on Unicode, since that is in fact not what one is doing. Based on all that, Python does not do case insensitive matching on Unicode, a condition contrary to its documented claims. That clearly makes it a bug that needs fixing rather than a feature request to be summarily ignored. > The test file should have omitted the gratuitous and distracting warnings, = > especially the one that effectively scolds Windows users for running Window= > s. With those omitted, the test cases given would form the basis for an add= > ed TestCase. I have absolutely no idea what on earth you could possibly be referring to. Honestly. I ran my tests on both releases (2.7 and 3.2), on both builds (wide and narrow), and on both platforms (Unix and Mac). The warnings are in there so I can make sure I have everything set up correctly to run the tests, and will understand why I get more failures than expected in the event that things are not set up appropriately. Let me make perfectly clear that I have never in my life come anywhere near a Microsoft system, let alone touched one, and that I furthermore never shall. I have not the foggiest notion what in the world you are complaining about. If the problem is that you are for some reason unable to create a Python with full Unicode support under Microsoft, that is hardly my fault. Render unto Caesar that which is Caesar's: complain to Microsoft about Microsoft's bugs, not to me, as I am wholly blameless of their problems. If you don't like my test cases, you know where to find vi. I supposed I could always send you the program that writes these programs for me, but as I knew you won't like it, I withheld it. You already have all that you need to see exactly where the bugs are and how to fix them. --tom -- ___ Python tracker <http://bugs.python.org/issue12728> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12729] Python lib re cannot handle Unicode properly due to narrow/wide bug
Tom Christiansen added the comment: "Terry J. Reedy" wrote on Fri, 12 Aug 2011 22:21:59 -: > Does the regex module handle these particular issues better? No, it currently does not. One would have to ask Matthew directly, but I believe it was because he was trying to stay compatible with re, sometimes apparently even if that means being bug compatible. I have brought it to his attention though, and at last report he was pondering the matter. In contrast to how Python behaves on narrow builds, even though Java uses UTF-16 for its internal representation of strings, its Java Pattern is quite adamant about treating with logical code points alone. Besides running afoul of tr18, it is senseless to do otherwise. A dot is one Unicode code point, no matter whether you have 8-bit code units, 16-bit code units, or 32-bit code units. Similarly, character classes and their negations only match entire code points, never pieces of the same. ICU's regexes work the same way the normal Java Pattern library does. So too do Perl, Ruby, and Go. Python is really the odd man out here. Almost. One interesting counterexample is the vim editor. It has dot match a complete grapheme no matter how many code points that requires, because we're dealing with user-visible characters now, not programmer-visible one. It is an unreasonable burden to make the programmer deal with the fine-grained details of low-level serialization schemes instead of at least(*) the code point level of operations, which is the minimum for getting real work done. (*Note that tr18 admits that accessing text at the code point level meets only programmer expectations, not those of the user, and therefore to meet user expectations much more elaborate patterns must necessarily be constructed than if logical groups of coarser granularity than code points alone are supported.) Python should not be subject to changing its behavior from one build to the next. This astonishing narrow-vs-wide build behavior makes it virtually impossible to write portable code to work on arbitrary Unicode text. You cannot even know whether you need to match one dot or two to get a single code point, and similarly for character indexing, etc. Even identifiers come into play. Surrogates should be utterly nonexistent/invisible at this, the normal level of operation. An API that minimally but uniformly deals with logical code points and nothing finer in granularity is the only way to go here. Please trust me on this one. Graphemes (tr18 Level 2) and collation elements (Level 3) will someday build on that, but one must first support code points properly. That's why it's a Level 1 requirement. --tom -- ___ Python tracker <http://bugs.python.org/issue12729> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12731] python lib re uses obsolete sense of \w in full violation of UTS#18 RL1.2a
Tom Christiansen added the comment: > Terry J. Reedy added the comment: > However desireable it would be, I do not believe there is any claim in the = > manual that the re module follows the evolving Unicode consortium r.e. stan= My from the hip thought is that if re cannot be fixed to follow the Unicode Standard, it should be deprecated in favor of code that can if such is available, because you cannot process Unicode text with regular expressions otherwise. > dard. If I understand, you are saying that this statement in the doc, "Matc= > hes Unicode word characters;" is not now correct and should be revised. Was= > it once correct? Could we add "by an older definition of 'word' character"= > ? Yes, your hunch is exactly correct. They once had a lesser definition that they have now. It is very very old. I had to track this down for Java once. There is some discussion of a "word_character class" at least as far back as tr18v3 from back in 1998. http://www.unicode.org/reports/tr18/tr18-3.html By the time tr18v5 rolled around just a year later in 1999, the overall document has changed substantially, and you can clearly see its current shape there. Word characters are supposed to include all code points with the Alphabetic property, for example. http://www.unicode.org/reports/tr18/tr18-5.html However, the word "alphabetic" has *never* been synonymous in Unicode with \p{gc=Lu} \p{gc=Ll} \p{gc=Lt} \p{gc=Lm} \p{gc=Lo} as many people incorrectly assume, nor certainly to \p{gc=Lu} \p{gc=Ll} \p{gc=Lt} let alone to \p{gc=Lu} \p{gc=Ll} Rather, it has since its creation included code points that are not letters, such as all GC=Nl and also certain GC=So code points. And, notoriously, U+0345. Indeed it is here I first noticed that that Python had already broken with the Standard, because U+0345 COMBINING GREEK YPOGEGRAMMENI is GC=Mn, but Alphabetic=True, yet I have shown that Python's title method is messing up there. I wouldn't spend too much in archaeological digs, though, because lots of stuff has changed since the less millennium. It was in tr18v7 from 2003-05 that we hit paydirt, because this is when the famous Annex C of RL1.2a fame first appeared: http://www.unicode.org/reports/tr18/tr18-7.html#Compatibility_Properties Notice how it defines \w to be nothing more than \p{alpha}, \p{digit}, and \p{gc=Pc}. It does not yet contain the requirement that all Marks be counted as part of the word, just the few that are alphas -- which the U+0345 counts for, since it has an uppercase map of a capital iota! That particular change did not occur until tr18v8 in 2003-08, barely a scant three months later. http://www.unicode.org/reports/tr18/tr18-8.html#Compatibility_Properties Now at last we see word characters defined in the modern way that we have become used to. They must match any of: \p{alpha} \p{gc=Mark} \p{digit} \p{gc=Connector_Punctuation} BTW, Python is matching all of \p{GC=N} meaning \p{GC=Nd} \p{GC=Nl} \p{GC=No} instead of the required \p{GC=Nd} which is a synonym for \p{digit}. I don't know had that happened, because \w has never included all number code points in Unicode, only the decimal number ones. That all goes to show why, when citing conformance to some aspect of The Unicode Standard, one must be exceedingly careful just how one does so! The Unicode Consortium recognizes this is an issue, and I am pretty sure I can hear it in your own subtext as well. Kindly bear with and forgive me for momentarily sounding like a standard lawyer. I do this because to show not just why it is important to get references to the Unicode Standard correct, but indeed, how to do so. After I have given the formal requirements, I will then produce illustrations of various purported claims, some of which meet the citation requirements, and others which do not. === To begin with, there is an entire technical report on conformance. It includes: http://unicode.org/reports/tr33/ The Unicode Standard [Unicode] is a very large and complex standard. Because of this complexity, and because of the nature and role of the standard, it is often rather difficult to determine, in any particular case, just exactly what conformance to the Unicode Standard means. ... Conformance claims must be specific to versions of the Unicode Standard, but the level of specificity needed for a claim may vary according to the nature of the particular conformance claim. Some standards developed by the Unicode Consortium require separate conformance to a specific version (or later), of the Unicode Standard. This version is sometimes called the base version. In such cases, the version of the standard and
[issue12732] Can't portably use Unicode in Python identifiers
Tom Christiansen added the comment: "Terry J. Reedy" wrote on Fri, 12 Aug 2011 23:05:27 -: > Ouch! > Do the rejected characters qualify as identifier characters as defined > in Reference 2.3 Identifiers and keywords? > http://docs.python.org/py3k/reference/lexical_analysis.html#identifiers Yes, that's right, they do. You're using the standard IDS and IDC, and XIDS and XIDC, definitions. Here were the three identifiers that were a problem: 𝔘𝔫𝔦𝔠𝔬𝔡𝔢 = "super" 𐐔𐐯𐑅𐐨𐑉𐐯𐐻 = "Deseret" 𐌰𐍄𐍄𐌰‿𐌿𐌽𐍃𐌰𐍂 = "Gothic our father" If you cannot read those, then when piped through `uniquote -v` they are: \N{MATHEMATICAL FRAKTUR CAPITAL U}\N{MATHEMATICAL FRAKTUR SMALL N}\N{MATHEMATICAL FRAKTUR SMALL I}\N{MATHEMATICAL FRAKTUR SMALL C}\N{MATHEMATICAL FRAKTUR SMALL O}\N{MATHEMATICAL FRAKTUR SMALL D}\N{MATHEMATICAL FRAKTUR SMALL E} = "super" \N{DESERET CAPITAL LETTER DEE}\N{DESERET SMALL LETTER SHORT E}\N{DESERET SMALL LETTER ES}\N{DESERET SMALL LETTER LONG I}\N{DESERET SMALL LETTER ER}\N{DESERET SMALL LETTER SHORT E}\N{DESERET SMALL LETTER TEE} = "Deseret" \N{GOTHIC LETTER AHSA}\N{GOTHIC LETTER TEIWS}\N{GOTHIC LETTER TEIWS}\N{GOTHIC LETTER AHSA}\N{UNDERTIE}\N{GOTHIC LETTER URUS}\N{GOTHIC LETTER NAUTHS}\N{GOTHIC LETTER SAUIL}\N{GOTHIC LETTER AHSA}\N{GOTHIC LETTER RAIDA} = "Gothic our father" I'm not sure whether you recognize the scripts they belong to, but they're all in the astral planes. Using `uniquote -x` on them shows: \x{1D518}\x{1D52B}\x{1D526}\x{1D520}\x{1D52C}\x{1D521}\x{1D522} = "super" \x{10414}\x{1042F}\x{10445}\x{10428}\x{10449}\x{1042F}\x{1043B} = "Deseret" \x{10330}\x{10344}\x{10344}\x{10330}\x{203F}\x{1033F}\x{1033D}\x{10343}\x{10330}\x{10342} = "Gothic our father" As to whether they're proper identifiers per your reference above, I will take the first letter from each of 𝔘𝔫𝔦𝔠𝔬𝔡𝔢, 𐐔𐐯𐑅𐐨𐑉𐐯𐐻, and 𐌰𐍄𐍄𐌰‿𐌿𐌽𐍃𐌰𐍂, which are repsectively 𝔘, 𐐔, and 𐌰, or MATHEMATICAL FRAKTUR CAPITAL U DESERET CAPITAL LETTER DEE GOTHIC LETTER AHSA or 1D518 10414 10330 and show you their full Unicode properties of these reject code points. This requires the uniprops command, given which, these three commands are then completely identical: % uniprops -ga "𝔘" "𐐔" "𐌰" % uniprops -ga 1D518 10414 10330 % uniprops -ga "MATHEMATICAL FRAKTUR CAPITAL U" "DESERET CAPITAL LETTER DEE" "GOTHIC LETTER AHSA" and produce this output: U+1D518 ‹𝔘› \N{MATHEMATICAL FRAKTUR CAPITAL U} \w \pL \p{LC} \p{L_} \p{L&} \p{Lu} All Any Alnum Alpha Alphabetic Assigned InMathematicalAlphanumericSymbols Cased Cased_Letter LC Changes_When_NFKC_Casefolded CWKCF Common Zyyy Lu L Gr_Base Grapheme_Base Graph GrBase ID_Continue IDC ID_Start IDS Letter L_ Uppercase_Letter Math Mathematical_Alphanumeric_Symbols Print Upper Uppercase Word XID_Continue XIDC XID_Start XIDS X_POSIX_Alnum X_POSIX_Alpha X_POSIX_Graph X_POSIX_Print X_POSIX_Upper X_POSIX_Word Age=3.1 Bidi_Class=L Bidi_Class=Left_To_Right BC=L Block=Mathematical_Alphanumeric_Symbols Canonical_Combining_Class=0 Canonical_Combining_Class=Not_Reordered CCC=NR Canonical_Combining_Class=NR General_Category=Cased_Letter Script=Common Decomposition_Type=Font DT=Font Decomposition_Type=Non_Canon Decomposition_Type=Non_Canonical DT=NonCanon East_Asian_Width=Neutral GC=LC General_Category=L General_Category=Letter General_Category=L_ General_Category=LC GC=L General_Category=Lu General_Category=Uppercase_Letter GC=Lu Grapheme_Cluster_Break=Other GCB=XX Grapheme_Cluster_Break=XX Hangul_Syllable_Type=NA Hangul_Syllable_Type=Not_Applicable HST=NA Joining_Group=No_Joining_Group JG=NoJoiningGroup Joining_Type=Non_Joining JT=U Joining_Type=U Line_Break=AL Line_Break=Alphabetic LB=AL Numeric_Type=None NT=None Numeric_Value=NaN NV=NaN Present_In=3.1 IN=3.1 Present_In=3.2 IN=3.2 Present_In=4.0 IN=4.0 Present_In=4.1 IN=4.1 Present_In=5.0 IN=5.0 Present_In=5.1 IN=5.1 Present_In=5.2 IN=5.2 Present_In=6.0 IN=6.0 SC=Zyyy Script=Zyyy Sentence_Break=UP Sentence_Break=Upper SB=UP Word_Break=ALetter WB=LE Word_Break=LE _X_Begin U+10414 ‹𐐔› \N{DESERET CAPITAL LETTER DEE} \w \pL \p{LC} \p{L_} \p{L&} \p{Lu} All Any Alnum Alpha Alphabetic Assigned InDeseret Cased Cased_Letter LC Changes_When_Casefolded CWCF Changes_When_Casemapped CWCM Changes_When_Lowercased CWL Changes_When_NFKC_Casefolded CWKCF Deseret Dsrt Lu L Gr_Base Grapheme_Base Graph GrBase ID_Continue IDC ID_Start IDS Letter L_ Uppercase_Letter Print Upper Uppercase Word XID_Continue XIDC XID_Start XIDS X_POSIX_Alnum X_POSIX_Alpha X_POSIX_Graph X_POSIX_Print X_PO
[issue12729] Python lib re cannot handle Unicode properly due to narrow/wide bug
Tom Christiansen added the comment: David Murray wrote: > Tom, note that nobody is arguing that what you are requesting is a bad > thing :) There looked to be minor some resistance, based on absolute backwards compatibility even if wrong, regarding changing anything *at all* in re, even things that to my jaded seem like actual bugs. There are bugs, and then there are bugs. In my survey of Unicode support across 7 programming languages for OSCON http://training.perl.com/OSCON/index.html I came across a lot of weirdnesses, especially as first when the learning curve was high. Sure, I found it odd that unlike Java, Perl, and Ruby, Python didn't offer regular casemapping on strings, only the simple character-based mapping. But that doesn't make it a bug, which is why I filed it as an feature/enhancement request/wish, not as a bug. I always count as bugs not handling Unicode text the way Unicode says it must be handled. Such things would be: Emitting CESU-8 when told to emit UTF-8. Violating the rule that UTF-8 must be in the shortest possible encoding. Not treating a code point as a letter when the supported version of the UCD says it is. (This can happen if internal rules get out of sync with the current UCD.) Claiming one does "the expected thing on Unicode" for case-insensitive matches when not doing what Unicode says you must minimally do: use at least the simple casefolds, if not in fact the full ones. Saying \w matches Unicode word characters when one's definition of word characters differs from that of the supported version of the UCD. Supporting Unicode vX.Y.Z is more than adding more characters. All the behaviors specified in the UCD have to be updated too, or else you are just ISO 10646. I believe some of Python's Unicode bugs happened because folks weren't aware which things in Python were defined by the UCD or by various UTS reports yet were not being directly tracked that way. That's why its important to always fully state which version of these things you follow. Other bugs, many actually, are a result of the narrow/wide-build untransparency. There is wiggle room in some of these. For example, which is the one that applies to re, in that you could -- in a sense -- remove the bug by no longer claiming to do case-insensitive matches on Unicode. I do not find that very useful. Javascript works this way: it doesn't do Unicode casefolding. Java you have to ask nicely with the extra UNICODE_CASE flag, aka "(?u)", used with the CASE_INSENSITIVE, aka "(?i)". Sometimes languages provide different but equivalent interfaces to the same functionality. For example, you may not support the Unicode property \p{NAME=foobar} in patterns but instead support \N{foobar} in patterns and hopefully also in strings. That's just fine. On slightly shakier ground but still I think defensible is how one approaches support for the standard UCD properties: Case_FoldingSimple_Case_Folding Titlecase_MappingSimple_Titlecase_Mapping Uppercase_MappingSimple_Uppercase_Mapping Lowercase_MappingSimple_Lowercase_Mapping One can support folding, for example, via (?i) and not have to directly supporting a Case_Folding property like \p{Case_Folding=s}, since "(?i)s" should be the same thing as "\p{Case_Folding=s}". > As far as I know, Matthew is the only one currently working on the > regex support in Python. (Other developers will commit small fixes if > someone proposes a patch, but no one that I've seen other than Matthew > is working on the deeper issues.) If you want to help out that would > be great. Yes, I actually would. At least as I find time for it. I'm a competent C programmer and Matthew's C code is very well documented, but that's very time consuming. For bang-for-buck, I do best on test and doc work, making sure things are actually working the way they say do. I was pretty surprised and disappointed by how much trouble I had with Unicode work in Python. A bit of that is learning curve, a bit of it is suboptimal defaults, but quite a bit of it is that things either don't work the way Unicode says, or because something is altogether missing. I'd like to help at least make the Python documentation clearer about what it is or is not doing in this regard. But be warned: one reason that Java 1.7 handles Unicode more according to the published Unicode Standard in its Character, String, and Pattern classes is because when they said they'd be supporting Unicode 6.0.0, I went through those classes and every time I found something in violation of that Standard, I filed a bug report that included a documentation patch explaining what they weren't doing right. Rather than apply my rather embarrassing doc patches, they instead fixed the code. :) > And
[issue12729] Python lib re cannot handle Unicode properly due to narrow/wide bug
Tom Christiansen added the comment: Matthew Barnett wrote on Sat, 13 Aug 2011 20:57:40 -: > There are occasions when you want to do string slicing, often of the form: > pos = my_str.index(x) > endpos = my_str.index(y) > substring = my_str[pos : endpos] Me, I would probably give the second call to index the first index position to guarantee the end comes after the start: str = "for finding the biggest of all the strings" x_at = str.index("big") y_at = str.index("the", x_at) some = str[x_at:y_at] print("GOT", some) But here's a serious question: is that *actually* a common usage pattern for accessing strings in Python? I ask because it wouldn't even *occur* to me to go at such a problem in that way. I would have always just written it this way instead: import re str = "for finding the biggest of all the strings" some = re.search("(big.*?)the", str).group(1) print("GOT", some) I know I would use the pattern approach, just because that's how I always do such things in Perl: $str = "for finding the biggest of all the strings"; ($some) = $str =~ /(big.*?)the/; print "GOT $some\n"; Which is obviously a *whole* lot simpler than the index approach: $str = "for finding the biggest of all the strings"; $x_at = index($str, "big"); $y_at = index($str, "the", $x_at); $len = $y_at - $x_at; $some = substr($str, $x_at, $len); print "GOT $some\n"; With no arithmetic and no need for temporary variables (you can't really escape needing x_at to pass to the second call to index), it's all a lot more WYSIWIG. See how much easier that is? Sure, it's a bit cleaner and less noisy in Perl than it is in Python by virtue of Perl's integrated pattern matching, but I would still use patterns in Python for this, not index. I honestly find the equivalent pattern operations a lot easier to read and write and maintain than I find the index/substring version. It's a visual thing. I find patterns a win in maintainability over all that busy index monkeywork. The index/rindex and substring approach is one I almost never ever turn to. I bet I use pattern matching 100 or 500 times for each time I use index, and maybe even more. I happen to think in patterns. I don't expect other people to do so. But because of this, I usually end up picking patterns even if they might be a little bit slower, because I think the gain in flexibility and especially maintability more than makes up for any minor performance concerns. This might also show you why patterns are so important to me: they're one of the most important tools we have for processing text. Index isn't, which is why I really don't care about whether it has O(1) access. > To me that suggests that if UTF-8 is used then it may be worth > profiling to see whether caching the last 2 positions would be > beneficial. Notice how with the pattern approach, which is inherently sequential, you don't have all that concern about running over the string more than once. Once you have the first piece (here, "big"), you proceed directly from there looking for the second piece in a straightforward, WYSIWIG way. There is no need to keep an extra index or even two around on the string structure itself, going at it this way. I would be pretty surprised if Perl could gain any speed by caching a pair of MRU index values against its UTF-8 [but see footnote], because again, I think the normal access pattern wouldn't make use of them. Maybe Python programmers don't think of strings the same way, though. That, I really couldn't tell you. But here's something to think about: If it *is* true that you guys do all this index stuff that Perl programmers just never see or do because of our differing comfort levels with regexes, and so you think Python that might still benefit from that sort of caching because its culture has promoted a different access pattern, then that caching benefit would still apply even if you were retain the current UTF-16 representation instead of going to UTF-8 (which might want it) or to UTF-32 (which wouldn't). After all, you have the same variable-width caching issue with UTF-16 as with UTF-8, so if it makes sense to have an MRU cache mapping character indices to byte indices, then it doesn't matter whether you use UTF-8 or UTF-16! However, I'd want some passive comparative benchmarks using real programs with real data, because I would be suspicious of incurring the memory cost of two more pointers in every string in the whole program. That's serious. --tom FOOTNOTE: The Perl 6 people are thinking about clever ways to set up byte offset indices. You have to do this if
[issue12729] Python lib re cannot handle Unicode properly due to narrow/wide bug
Tom Christiansen added the comment: Antoine Pitrou wrote on Sat, 13 Aug 2011 21:09:52 -: > And/or a lookup table giving the byte offset of, say, every 16th > character. It gives you a O(1) lookup with a relatively reasonable > constant cost (you have to scan for less than 16 characters after the > lookup). > On small strings (< 256 UTF-8 bytes) the space overhead for the lookup > table would be 1/16. It could also be constructed lazily whenever more > than 2 positions are cached. You really should talk to the Perl 6 people to see whether their current strategy for caching offset maps for grapheme positions might be of use to you. Larry explained it to me once but I no longer recall any details. I notice though that they don't seem to think it worth doing for UTF-8 or UTF-16, just for their synthetic "NFG" (Grapheme Normalization Form) strings, where it would be needed even if they used UTF-32 underneath. --tom -- ___ Python tracker <http://bugs.python.org/issue12729> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12729] Python lib re cannot handle Unicode properly due to narrow/wide bug
Tom Christiansen added the comment: >> Here's why I say that Python uses UTF-16 not UCS-2 on its narrow builds. >> Perhaps someone could tell me why the Python documentation says it uses >> UCS-2 on a narrow build. > There's a disagreement on that point between several developers. > See an example sub-thread at: > http://mail.python.org/pipermail/python-dev/2010-November/105751.html Some of those folks know what they're talking about, and some do not. Most of the postings miss the mark. Python uses UTF-16 for its narrow builds. It does not use UCS-2. The argument that it must be UCS-2 because it can store lone surrogates in memory is spurious. You have to read The Unicode Standard very *very* closely, but it is not necessary that all internal buffers always be in well-formed UTF-whatever. Otherwise it would be impossible to append a code unit at a time to buffer. I could pull out the reference if I worked at it, because I've had to find it before. It's in there. Trust me. I know. It is also spurious to pretend that because you can produce illegal output when telling it to generate something in UTF-16 that it is somehow not using UTF-16. You have simply made a mistake. You have generated something that you have promised you would not generate. I have more to say about this below. Finally, it is spurious to argue against UTF-16 because of the code unit interface. Java does exactly the same thing as Python does *in all regards* here, and no one pretends that Java is UCS-2. Both are UTF-16. It is simply a design error to pretend that the number of characters is the number of code units instead of code points. A terrible and ugly one, but it does not mean you are UCS-2. You are not. Python uses UTF-16 on narrow builds. The ugly terrible design error is digusting and wrong, just as much in Python as in Java, and perhaps moreso because of the idiocy of narrow builds even existing. But that doesn't make them UCS-2. If I could wave a magic wand, I would have Python undo its code unit blunder and go back to code points, no matter what. That means to stop talking about serialization schemes and start talking about logical code points. It means that slicing and index and length and everything only report true code points. This horrible code unit botch from narrow builds is most easily cured by moving to wide builds only. However, there is more. I haven't checked its UTF-16 codecs, but Python's UTF-8 codec is broken in a bunch of ways. You should be raising as exception in all kinds of places and you aren't. I can see I need to bug report this stuff to. I don't to be mean about this. HONEST! It's just the way it is. Unicode currently reserves 66 code points as noncharacters, which it guarantees will never be in a legal UTF-anything stream. I am not talking about surrogates, either. To start with, no code point which when bitwise added with 0xFFFE returns 0xFFFE can never appear in a valid UTF-* stream, but Python allow this without any error. That means that both 0xNN_FFFE and 0xNN_ are illegal in all planes, where NN is 00 through 10 in hex. So that's 2 noncharacters times 17 planes = 34 code points illegal for interchange that Python is passing through illegally. The remaining 32 nonsurrogate code points illegal for open interchange are 0xFDD0 through 0xFDEF. Those are not allowed either, but Python doesn't seem to care. You simply cannot say you are generating UTF-8 and then generate a byte sequence that UTF-8 guarantees can never occur. This is a violation. ***SIGH*** --tom -- ___ Python tracker <http://bugs.python.org/issue12729> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12729] Python lib re cannot handle Unicode properly due to narrow/wide bug
Tom Christiansen added the comment: Ezio Melotti added the comment: >> It is simply a design error to pretend that the number of characters >> is the number of code units instead of code points. A terrible and >> ugly one, but it does not mean you are UCS-2. > If you are referring to the value returned by len(unicode_string), it > is the number of code units. This is a matter of "practicality beats > purity". Returning the number of code units is O(1) (num_of_bytes/2). > To calculate the number of characters it's instead necessary to scan > all the string looking for surrogates and then count any surrogate > pair as 1 character. It was therefore decided that it was not worth > to slow down the common case just to be 100% accurate in the > "uncommon" case. If speed is more important than correctness, I can make any algorithm infinitely fast. Given the choice between correct and quick, I will take correct every single time. Plus your strings our immutable! You know how long they are and they never change. Correctness comes at a negligible cost. It was a bad choice to return the wrong answer. > That said it would be nice to have an API (maybe in unicodedata or as > new str methods?) able to return the number of code units, code > points, graphemes, etc, but I'm not sure that it should be the default > behavior of len(). Always code points, never code units. I even use a class whose length method returns the grapheme count, because even code points aren't good enough. Yes of course graphemes have to be counted. Big deal. How would you like it if you said to move three to the left in vim and it *didn't* count each graphemes as one position? Madness. >> The ugly terrible design error is digusting and wrong, just as much >> in Python as in Java, and perhaps moreso because of the idiocy of >> narrow builds even existing. > Again, wide builds use twice as much the space than narrow ones, but > one the other hand you can have fast and correct behavior with e.g. > len(). If people don't care about/don't need to use non-BMP chars and > would rather use less space, they can do so. Until we agree that the > difference in space used/speed is no longer relevant and/or that non- > BMP characters become common enough to prefer the "correct behavior" > over the "fast-but-inaccurate" one, we will probably keep both. Which is why I always put loud warnings in my Unicode-related Python programs that they do not work right on Unicode if running under a narrow build. I almost feel I should just exit. >> I haven't checked its UTF-16 codecs, but Python's UTF-8 codec is >> broken in a bunch of ways. You should be raising as exception in >> all kinds of places and you aren't. > I am aware of some problems of the UTF-8 codec on Python 2. It used > to follow RFC 2279 until last year and now it's been updated to follow > RFC 3629. Unicode says you can't put surrogates or noncharacters in a UTF-anything stream. It's a bug to do so and pretend it's a UTF-whatever. Perl has an encoding form, which it does not call "UTF-8", that you can use the UTF-8 algorithm on for any code point, include non-characters and surrogates and even non-Unicode code points far above 0x10_, up to in fact 0x___ on 64-bit machines. It's the internal format we use in memory. But we don't call it real UTF-8, either. It sounds like this is the kind of thing that would be useful to you. > However, for backward compatibility, it still encodes/decodes > surrogate pairs. This broken behavior has been kept because on Python > 2, you can encode every code point with UTF-8, and decode it back > without errors: No, that's not UTF-8 then. By definition. See the Unicode Standard. >>>> x = [unichr(c).encode('utf-8') for c in range(0x11)] >>>> > and breaking this invariant would probably make more harm than good. Why? Create something called utf8-extended or utf8-lax or utf8-nonstrict or something. But you really can't call it UTF-8 and do that. We actually equate "UTF-8" and "utf8-strict". Our internal extended UTF-8 is something else. It seems like you're still doing the old relaxed version we used to have until 2003 or so. It seems useful to be able to have both flavors, the strict and the relaxed one, and to call them different things. Perl defaults to the relaxed one, which gives warnings not exceptions, if you do things like setting PERLUNICODE to S or SD and such for the default I/I encoding. If you actually use "UTF-8" as the encoding on the stream, though, you get the version that gives exceptions instead. "UTF-8" = "utf8-s
[issue12749] lib re cannot match non-BMP ranges (all versions, all builds)
New submission from Tom Christiansen : On neither narrow nor wide builds does this UTF8-encoded bit run without raising an exception: if re.search("[𝒜-𝒵]", "𝒞", re.UNICODE): print("match 1 passed") else: print("match 2 failed") The best you can possibly do is to use both a wide build *and* symbolic literals, in which case it will pass. But remove either of both of those conditions and you fail. This is too restrictive for full Unicode use. There should never be any sitation where [a-z] fails to match c when a < c < z, and neither a nor z is something special in a character class. There is, or perhaps should be, no difference at all between "[a-z]" and "[𝒜-𝒵]", just as there is, or at least should b, no difference between "c" and "𝒞". You can’t have second-class citizens like this that can't be used. And no, this one is *not* fixed by Matthew Barnett's regex library. There is some dumb UCS-2 assumption lurking deep in Python somewhere that makes this break, even on wide builds, which is incomprehensible to me. -- components: Regular Expressions files: bigrange.py messages: 142058 nosy: Arfrever, ezio.melotti, jkloth, mrabarnett, pitrou, r.david.murray, tchrist, terry.reedy priority: normal severity: normal status: open title: lib re cannot match non-BMP ranges (all versions, all builds) type: behavior versions: Python 3.2 Added file: http://bugs.python.org/file22897/bigrange.py ___ Python tracker <http://bugs.python.org/issue12749> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12729] Python lib re cannot handle Unicode properly due to narrow/wide bug
Tom Christiansen added the comment: Ezio Melotti wrote on Sun, 14 Aug 2011 07:15:09 -: >> Unicode says you can't put surrogates or noncharacters in a >> UTF-anything stream. It's a bug to do so and pretend it's a >> UTF-whatever. > The UTF-8 codec described by RFC 2279 didn't say so, so, since our > codec was following RFC 2279, it was producing valid UTF-8. With RFC > 3629 a number of things changed in a non-backward compatible way. > Therefore we couldn't just change the behavior of the UTF-8 codec nor > rename it to something else in Python 2. We had to wait till Python 3 > in order to fix it. I'm a bit confused on this. You no longer fix bugs in Python 2? I've dug out the references that state that you are not allowed to do things the way you are doing them. This is from the published Unicode Standard version 6.0.0, chapter 3, Conformance. It is a very important chapter. http://www.unicode.org/versions/Unicode6.0.0/ch03.pdf Python is in violation of that published Standard by interpreting noncharacter code points as abstract characters and tolerating them in character encoding forms like UTF-8 or UTF-16. This explains that conformant processes are forbidden from doing this. Code Points Unassigned to Abstract Characters C1 A process shall not interpret a high-surrogate code point or a low-surrogate code point as an abstract character. · The high-surrogate and low-surrogate code points are designated for surrogate code units in the UTF-16 character encoding form. They are unassigned to any abstract character. ==> C2 A process shall not interpret a noncharacter code point as an abstract character. · The noncharacter code points may be used internally, such as for sentinel val- ues or delimiters, but should not be exchanged publicly. C3 A process shall not interpret an unassigned code point as an abstract character. · This clause does not preclude the assignment of certain generic semantics to unassigned code points (for example, rendering with a glyph to indicate the position within a character block) that allow for graceful behavior in the pres- ence of code points that are outside a supported subset. · Unassigned code points may have default property values. (See D26.) · Code points whose use has not yet been designated may be assigned to abstract characters in future versions of the standard. Because of this fact, due care in the handling of generic semantics for such code points is likely to provide better robustness for implementations that may encounter data based on future ver- sions of the standard. Next we have exactly how something you call UTF-{8,16-32} must be formed. *This* is the Standard against which these things are measured; it is not the RFC. You are of course perfectly free to say you conform to this and that RFC, but you must not say you conform to the Unicode Standard when you don't. These are different things. I feel it does users a grave disservice to ignore the Unicode Standard in this, and sheer casuistry to rely on an RFC definition while ignoring the Unicode Standard whence it originated, because this borders on being intentionally misleading. Character Encoding Forms C8 When a process interprets a code unit sequence which purports to be in a Unicode char- acter encoding form, it shall interpret that code unit sequence according to the corre- sponding code point sequence. ==>· The specification of the code unit sequences for UTF-8 is given in D92. · The specification of the code unit sequences for UTF-16 is given in D91. · The specification of the code unit sequences for UTF-32 is given in D90. C9 When a process generates a code unit sequence which purports to be in a Unicode char- acter encoding form, it shall not emit ill-formed code unit sequences. · The definition of each Unicode character encoding form specifies the ill- formed code unit sequences in the character encoding form. For example, the definition of UTF-8 (D92) specifies that code unit sequences such as are ill-formed. ==> C10 When a process interprets a code unit sequence which purports to be in a Unicode char- acter encoding form, it shall treat ill-formed code unit sequences as an error condition and shall not interpret such sequences as characters. · For example, in UTF-8 every code unit of the form 1102 must be followed by a code unit of the form 10xx2. A sequence such as 110x2 0xxx2 is ill-formed and must never be generated. When faced with this ill-formed code unit sequence while transforming or interpreting text, a conformant pro-
[issue12749] lib re cannot match non-BMP ranges (all versions, all builds)
Tom Christiansen added the comment: >Ezio Melotti added the comment: >On wide 3.2 it passes too, so the failure is limited to narrow builds (are = >you sure that it fails on wide builds for you?). You're right: my wide build is not Python3, just Python2. In fact, it's even worse, because it's the stock build on Linux, which seems on this machine to be 2.6 not 2.7. I have private builds that are 2.7 and 3.2, but those are both narrow. I do not have a 3.3 build. Should I? I'm remembering why I removed Python2 from my Unicode talk, because of how it made me pull my hair out. People at the talk wanted to know what I meant, but I didn't have time to go into it. I think this gets added to the hairpulling list. --tom -- ___ Python tracker <http://bugs.python.org/issue12749> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12729] Python lib re cannot handle Unicode properly due to narrow/wide bug
Tom Christiansen added the comment: Ezio Melotti wrote on Sun, 14 Aug 2011 07:15:09 -: > For example I don't think removing the 0x10 upper limit is going to > happen -- even if it might be useful for other things. I agree entirely. That's why I appended a triple exclamation point to where I said I certainly do not expect this. It can only work fully on UTF-8ish systems and up to 32 bits on UTF-32, and it is most emphatically *not* Unicode. Yes, there are things you can do with it, but it risks serious misunderstanding and even noncomformance if not done very carefully. The Standard does not forbid such things internally, but you are not allowed to pass them around in noninternal streams claiming they are real UTF streams. > Also regular expressions are not part of the core and are not used > that often, so I consider problems with narrow/wide builds, codecs and > the unicode type much more important than problems with the re/regex > module (they should be fixed too, but have lower priority IMHO). One advantage of having an external library is the ability to update it asynchronously. Another is the possibility to swap in out altogether. Perl only gained that ability, which Python has always had, some four years ago with its 5.10 release. To my knowledge, the only thing people tend to use this for is to get Russ Cox's re2 library, which has very different performance characteristics and guarantees that allow it to be used in potential starvation denial-of-service situations that the normal Perl, Python, Java, etc regex engine cannot be safely used for. -tom -- ___ Python tracker <http://bugs.python.org/issue12729> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12749] lib re cannot match non-BMP ranges (all versions, all builds)
Tom Christiansen added the comment: Ezio Melotti wrote on Sun, 14 Aug 2011 17:15:52 -: >> You're right: my wide build is not Python3, just Python2. > And is it failing? Here the tests pass on the wide builds, on both Python 2 > and 3. Perhaps I am doing something wrong? linux% python --version Python 2.6.2 linux% python -c 'import sys; print sys.maxunicode' 1114111 linux% cat -n bigrange.py 1 #!/usr/bin/env python 2 # -*- coding: UTF-8 -*- 3 4 from __future__ import print_function 5 from __future__ import unicode_literals 6 7 import re 8 9 flags = re.UNICODE 10 11 if re.search("[a-z]", "c", flags): 12 print("match 1 passed") 13 else: 14 print("match 1 failed") 15 16 if re.search("[𝒜-𝒵]", "𝒞", flags): 17 print("match 2 passed") 18 else: 19 print("match 2 failed") 20 21 if re.search("[\U0001D49C-\U0001D4B5]", "\U0001D49E", flags): 22 print("match 3 passed") 23 else: 24 print("match 3 failed") 25 26 if re.search("[\N{MATHEMATICAL SCRIPT CAPITAL A}-\N{MATHEMATICAL SCRIPT CAPITAL Z}]", 27 "\N{MATHEMATICAL SCRIPT CAPITAL C}", flags): 28 print("match 4 passed") 29 else: 30 print("match 4 failed") linux% python bigrange.py match 1 passed Traceback (most recent call last): File "bigrange.py", line 16, in if re.search("[𝒜-𝒵]", "𝒞", flags): File "/usr/lib64/python2.6/re.py", line 142, in search return _compile(pattern, flags).search(string) File "/usr/lib64/python2.6/re.py", line 245, in _compile raise error, v # invalid expression sre_constants.error: bad character range >> In fact, it's even worse, because it's the stock build on Linux, >> which seems on this machine to be 2.6 not 2.7. > What is worse? FWIW on my system the default `python` is a 2.7 wide. > `python3` is a 3.2 wide. I meant that it was running 2.6 not 2.7. >> I have private builds that are 2.7 and 3.2, but those are both narrow. >> I do not have a 3.3 build. Should I? > 3.3 is the version in development, not released yet. If you have an > HG clone of Python you can make a wide build of 3.x with ./configure > --with-wide-unicode andof 2.7 using ./configure --enable- > unicode=ucs4. And Antoine Pitrou wrote: >> I have private builds that are 2.7 and 3.2, but those are both narrow. >> I do not have a 3.3 build. Should I? > I don't know if you *should*. But you can make one easily by passing > "--with-wide-unicode" to ./configure. Oh good. I need to read configure --help more carefully next time. I have to some Lucene work this afternoon, so I can let several builds chug along. Is there a way to easily have these co-exist on the same system? I'm sure I have to rebuild all C extensions for the new builds, but I wonder what to about (for example) /usr/local/lib/python3.2 being able to be only one of narrow or wide. Probably I just to go reading the configure stuff better for alternate paths. Unsure. Variant Perl builds can coexist on the same system with some directories shared and others not, but I often find other systems aren't quite that flexible, usually requiring their own dedicated trees. Manpaths can get tricky, too. >> I'm remembering why I removed Python2 from my Unicode talk, because >> of how it made me pull my hair out. People at the talk wanted to know >> what I meant, but I didn't have time to go into it. I think this >> gets added to the hairpulling list. > I'm not sure what you are referring to here. There seem to many more things to get wrong with Unicode in v2 than in v3. I don't know how much of this just my slowness at ramping up the learning curve, how much is due to historical defaults that don't work well for Unicode, and how much is Python2: re.search(u"[\N{MATHEMATICAL SCRIPT CAPITAL A}-\N{MATHEMATICAL SCRIPT CAPITAL Z}]".encode('utf-8'), u"\N{MATHEMATICAL SCRIPT CAPITAL C}".encode('utf-8'), re.UNICODE) Python3: re.search("[\N{MATHEMATICAL SCRIPT CAPITAL A}-\N{MATHEMATICAL SCRIPT CAPITAL Z}]", "\N{MATHEMATICAL SCRIPT CAPITAL C}", re.UNICODE) The Python2 version is *much* noisier. (1) You have keep remembering to u"..." everything because neither # -*- coding: UTF-8 -*- nor even from __future__ import unicode_literals suffices. (2) You have to manually encode every s
[issue12729] Python lib re cannot handle Unicode properly due to narrow/wide bug
Tom Christiansen added the comment: Ezio Melotti wrote on Sun, 14 Aug 2011 17:46:55 -: >> I'm a bit confused on this. You no longer fix bugs in Python 2? > We do, but it's unlikely that we will introduce major changes in behavior. > Even if we had to get rid of narrow builds and/or fix len(), we would > probably only do it in the next 3.x version (i.e. 3.3), and not in the > next bug fix release of 3.2 (i.e. 3.2.2). Antoine Pitrou wrote on Sun, 14 Aug 2011 17:36:42 -: > This is even truer for stable branches, and Python 2 is very much a > stable branch now (no more feature releases after 2.7). Does that mean you now go to 2.7.1, 2.7.2, etc? I had thought that 2.6 was going to be the last, but then 2.7 ame out. I think I remember Guido said something about there never being a 2.10, so I wasn't too surprised to see 2.7. --tom -- ___ Python tracker <http://bugs.python.org/issue12729> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12729] Python lib re cannot handle Unicode properly due to narrow/wide bug
Tom Christiansen added the comment: "Terry J. Reedy" wrote on Mon, 15 Aug 2011 00:26:53 -: > PS: The OSCON link in msg142036 currently gives me 404 not found Sorry, I wrote http://training.perl.com/OSCON/index.html but meant http://training.perl.com/OSCON2011/index.html I'll fix it on the server in a short spell. I am trying to keep the document up to date as I learn more, so it isn't precisely the talk I gave in Portland. > Python's narrow builds are, in a sense, 'between' UCS-2 and UTF-16. So I'm finding. Perhaps that's why I keep getting confused. I do have a pretty firm notion of what UCS-2 and UTF-16 are, and so I get sometimes self-contradictory results. Can you think of anywhere that Python acts like UCS-2 and not UTF-16? I'm not sure I have found one, although the regex thing might count. Thank you guys for being so helpful and understanding. > They support non-BMP chars but only partially, because, BY DESIGN*, > indexing and len are by code units, not codepoints. That's what Java did, too, and for the same reason. Because they had a UCS-2 implementation for Unicode 1.1 so when Unicode 2.0 came out and they learned that they would need more than 16 bits, they piggybacked UTF-16 onto the top of it instead of going for UTF-8 or UTF-32, and they're still paying that price, and to my mind, heavily and continually. Do you use Java? It is very like Python in many of its 16-bit character issues. Most of the length and indexing type functions address things by code unit only, not copepoint. But they would never claim to be UCS-2. Oh, I realize why they did it. For one thing, they had bytecode out there that they had to support. For another, they had some pretty low-level APIs that didn't have enough flexibility of abstraction, so old source had to keep working as before, even though this penalized the future. Forever, kinda. While I wish they had done better, and kinda think they could have, it isn't my place to say. I wasn't there (well, not paying attention) when this was all happening, because I was so underwhelmed by the how annoyingly overhyped it was. A billion dollars of marketing can't be wrong, you know? I know that smart people looked at it, seriously. I just find the cure they devised to be more in the problem set than the solution set. I like how Python works on wide builds, especially with Python3. I was pretty surprised that the symbolic names weren't working right on the earlier version of the 2.6 wide build I tried them on. I know have both wide and narrow builds installed of both 2.7 and 3.2, so that shouldn't happen again. > They are documented as being UCS-2 because that is what M-A Lemburg, > the original designer and writer of Python's unicode type and the unicode- > capable re module, wants them to be called. The link to msg142037, > which is one of 50+ in the thread (and many or most other disagree), > pretty well explains his viewpoint. Count me as one of those many/most others who disagree. :) > The positive side is that we deliver more than we promise. The > negative side is that by not promising what perhaps we should allows > is not to deliver what perhaps we should. It is always better to deliver more than you say than to deliver less. > * While I think this design decision may have been OK a decade ago for > a first implementation of an *optional* text type, I do not think it > so for the future for revised implementations of what is now *the* > text type. I think narrow builds can and should be revised and > upgraded to index, slice, and measure by codepoints. Yes, I think so, too. If you look at the growth curve of UTF-8 alone, it has followed a mathematically exponential growth curve in the first decade of this century. I suspect that will turn into an S surve with with aymtoptotic shoulders any time now. I haven't looked at it lately, so maybe it already has. I know that huge corpora I work with at work are all absolutely 100% Unicode now. Thank XML for that. > Here is my current idea: > If the code unit stream contains any non-BMP characters (ie, surrogate > pair of 16-bit code units), construct a sequence of *indexes* of such > characters (pairs). The fixed length of the string in codepoints is > n-k, where n is the number of code units (the current length) and k is > the length of the auxiliary sequence and the number of pairs. For > indexing, look up the character index in the list of indexes by binary > search and increment the codepoint index by the index of the index > found to get the corresponding code unit index. (I have omitted the > details needed avoid off-by-1 errors.) > This would make indexing O(log(k)) when there are surrogates. If that > is really a problem because k is a substantial fraction
[issue12729] Python lib re cannot handle Unicode properly due to narrow/wide bug
Tom Christiansen added the comment: I wrote: >> Python's narrow builds are, in a sense, 'between' UCS-2 and UTF-16. > So I'm finding. Perhaps that's why I keep getting confused. I do have a > pretty firm > notion of what UCS-2 and UTF-16 are, and so I get sometimes > self-contradictory results. > Can you think of anywhere that Python acts like UCS-2 and not UTF-16? I'm > not sure I > have found one, although the regex thing might count. I just thought of one. The casemapping functions don't work right on Deseret, which is a non-BMP case-changing scripts. That's one I submitted as a bug, because I figure if the the UTF-8 decoder can decode the non-BMP code points into paired UTF-16 surrogates, then the casing functions had jolly well be able to deal with it. If the UTF-8 decoder knows it is only going to UCS-2, then it should have raised on exception on my non-BMP source. Since it went to UTF-16, the rest of the language should have behaved accordingly. Java does to this right, BTW, despite its UTF-16ness. --tom -- ___ Python tracker <http://bugs.python.org/issue12729> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12729] Python lib re cannot handle Unicode properly due to narrow/wide bug
Tom Christiansen added the comment: Ezio Melotti wrote on Mon, 15 Aug 2011 04:56:55 -: > Another thing I noticed is that (at least on wide builds) surrogate pairs are > not joined "on the fly": > >>> p > '\ud800\udc00' > >>> len(p) > 2 > >>> p.encode('utf-16').decode('utf-16') > '𐀀' > >>> len(_) > 1 (For those who may not immediately realize from reading the surrogates, '𐀀' is code point 0x1, the first non-BMP code point. I piped it through `uniquote -x` just to make sure.) Yes, that makes perfect sense. It's something of a buggy feature or featureful bug that UTF-16 does this. When you are thinking of arbitrary sequences of code points, which is something you have be able to do in memory but not in a UTF stream, then one can say that one has four code points of anything in the 0 .. 0x10 range. Those can be any arbitrary code points only (1) *while* in memory, *and* assuming a (2) non-UTF16, ie UTF-32 or UTF-8 representation. You cannot do that with UTF-16, which is why it works only on a Python wide build. Otherwise they join up. The reason they "join up" in UTF-16 is also the reason why unlike in regular memory where you might be able to use an alternate representation like UTF-8 or UTF-32, UTF streams cannot contain unpaired surrogates: because if that stream were in UTF-16, you would never be able to tell the difference between a sequence of a lead surrogate followed by a tail surrogate and the same thing meaning just one non-BMP code point. Since you would not be able to tell the difference, it always only means the latter, and the former sense is illegal. This is why lone surrogates are illegal in UTF streams. In case it isn't obvious, *this* is the source of the [𝒜--𝒵] bug in all the UTF-16 or UCS-2 regex languages. It is why Java 7 added \x{...}, so that they can rewrite that as [\x{1D49C}--\x{1D4B5}] to pass the regex compiler, so that it seems something indirect, not just surrogates. That's why I always check it in my cross-language regex tests. A 16-bit language has to have a workaround, somehow, or it will be in trouble. The Java regex compiler doesn't generate UTF-16 for itself, either. It generates UTF-32 for its pattern. You can see this right at the start of the source code. This is from the Java Pattern class: /** * Copies regular expression to an int array and invokes the parsing * of the expression which will create the object tree. */ private void compile() { // Handle canonical equivalences if (has(CANON_EQ) && !has(LITERAL)) { normalize(); } else { normalizedPattern = pattern; } patternLength = normalizedPattern.length(); // Copy pattern to int array for convenience // Use double zero to terminate pattern temp = new int[patternLength + 2]; hasSupplementary = false; int c, count = 0; // Convert all chars into code points for (int x = 0; x < patternLength; x += Character.charCount(c)) { c = normalizedPattern.codePointAt(x); if (isSupplementary(c)) { hasSupplementary = true; } temp[count++] = c; } patternLength = count; // patternLength now in code points See how that works? They use an int(-32) array, not a char(-16) array! It's reasonably clever, and necessary. Because it does that, it can now compile \x{1D49C} or erstwhile embedded UTF-8 non-BMP literals into UTF-32, and not get upset by the stormy sea of troubles that surrogates are. You can't have surrogates in ranges if you don't do something like this in a 16-bit language. Java couldn't fix the [𝒜--𝒵] bug except by doing the \x{...} indirection trick, because they are stuck with UTF-16. However, they actually can match the string "𝒜" against the pattern "^.$", and have it fail on "^..$". Yes, I know: the code-unit length of that string is 2, but its regex count is just one dot worth. I *believe* they did it that way because tr18 says it has to work that way, but they may also have done it just because it makes sense. My current contact at Oracle doing regex support is not the guy who originally wrote the class, so I am not sure. (He's very good, BTW. For Java 7, he also added named captures, script properties, *and* brought the class up to conformance with tr18's "level 1" requirements.) I'm thinking Python might be able to do in the regex engine on narrow builds the sort of thing that Java does. However, I am also thinking that that might be a lot of work for a situation more readily addressed by phasing out narrow builds or at least telling p
[issue12746] normalization is affected by unicode width
Changes by Tom Christiansen : -- nosy: +tchrist ___ Python tracker <http://bugs.python.org/issue12746> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12753] \N{...} neglects formal aliases and named sequences from Unicode charnames namespace
New submission from Tom Christiansen : Unicode character names share a common namespace with formal aliases and with named sequences, but Python recognizes only the original name. That means not everything in the namespace is accessible from Python. (If this is construed to be an extant bug from than an absent feature, you probably want to change this from a wish to a bug in the ticket.) This is a problem because aliases correct errors in the original names, and are the preferred versions. For example, ISO screwed up when they called U+01A2 LATIN CAPITAL LETTER OI. It is actually LATIN CAPITAL LETTER GHA according to the file NameAliases.txt in the Unicode Character Database. However, Python blows up when you try to use this: % env PYTHONIOENCODING=utf8 python3.2-narrow -c 'print("\N{LATIN CAPITAL LETTER OI}")' Ƣ % env PYTHONIOENCODING=utf8 python3.2-narrow -c 'print("\N{LATIN CAPITAL LETTER GHA}")' File "", line 1 SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 0-27: unknown Unicode character name Exit 1 This unfortunate, because the formal aliases correct egregious blunders, such as the Standard reading "BRAKCET" instead of "BRACKET": $ uninames '^\s+%' Ƣ 01A2LATIN CAPITAL LETTER OI % LATIN CAPITAL LETTER GHA ƣ 01A3LATIN SMALL LETTER OI % LATIN SMALL LETTER GHA * Pan-Turkic Latin alphabets ೞ 0CDEKANNADA LETTER FA % KANNADA LETTER LLLA * obsolete historic letter * name is a mistake for LLLA ຝ 0E9DLAO LETTER FO TAM % LAO LETTER FO FON = fo fa * name is a mistake for fo sung ຟ 0E9FLAO LETTER FO SUNG % LAO LETTER FO FAY * name is a mistake for fo tam ຣ 0EA3LAO LETTER LO LING % LAO LETTER RO = ro rot * name is a mistake, lo ling is the mnemonic for 0EA5 ລ 0EA5LAO LETTER LO LOOT % LAO LETTER LO = lo ling * name is a mistake, lo loot is the mnemonic for 0EA3 ࿐ 0FD0TIBETAN MARK BSKA- SHOG GI MGO RGYAN % TIBETAN MARK BKA- SHOG GI MGO RGYAN * used in Bhutan ꀕ A015YI SYLLABLE WU % YI SYLLABLE ITERATION MARK * name is a misnomer ︘ FE18PRESENTATION FORM FOR VERTICAL RIGHT WHITE LENTICULAR BRAKCET % PRESENTATION FORM FOR VERTICAL RIGHT WHITE LENTICULAR BRACKET * misspelling of "BRACKET" in character name is a known defect # 3017 𝃅 1D0C5 BYZANTINE MUSICAL SYMBOL FHTORA SKLIRON CHROMA VASIS % BYZANTINE MUSICAL SYMBOL FTHORA SKLIRON CHROMA VASIS * misspelling of "FTHORA" in character name is a known defect There are only In Perl, \N{...} grants access to the single, shared, common namespace of Unicode character names, formal aliases, and named sequences without distinction: % env PERL_UNICODE=S perl -Mcharnames=:full -le 'print("\N{LATIN CAPITAL LETTER OI}")' Ƣ % env PERL_UNICODE=S perl -Mcharnames=:full -le 'print("\N{LATIN CAPITAL LETTER GHA}")' Ƣ % env PERL_UNICODE=S perl -Mcharnames=:full -le 'print("\N{LATIN CAPITAL LETTER OI}")' | uniquote -x \x{1A2} % env PERL_UNICODE=S perl -Mcharnames=:full -le 'print("\N{LATIN CAPITAL LETTER GHA}")' | uniquote -x \x{1A2} It is my suggestion that Python do the same thing. There are currently only 11 of these. The third element in this shared namespace of name, named sequences, are multiple code points masquerading under one name. They come from the NamedSequences.txt file in the Unicode Character Database. An example entry is: LATIN CAPITAL LETTER A WITH MACRON AND GRAVE;0100 0300 There are 418 of these named sequences as of Unicode 6.0.0. This shows that Perl can also access named sequences: $ env PERL_UNICODE=S perl -Mcharnames=:full -le 'print("\N{LATIN CAPITAL LETTER A WITH MACRON AND GRAVE}")' Ā̀ $ env PERL_UNICODE=S perl -Mcharnames=:full -le 'print("\N{LATIN CAPITAL LETTER A WITH MACRON AND GRAVE}")' | uniquote -x \x{100}\x{300} $ env PERL_UNICODE=S perl -Mcharnames=:full -le 'print("\N{KATAKANA LETTER AINU P}")' ㇷ゚ $ env PERL_UNICODE=S perl -Mcharnames=:full -le 'print("\N{KATAKANA LETTER AINU P}")' | uniquote -x \x{31F7}\x{309A} Since it is a single namespace, it makes sense that all members of that namespace should be accessible using \N{...} as a sort of equal-opportunity accessor mechanism, and it does not make sense that they not be. Just makes sure you take only the approved named sequences from the NamedSequences.txt file. It would be unwise to give users access to the provisiona
[issue12730] Python's casemapping functions are untrustworthy due to narrow/wide build issues
Tom Christiansen added the comment: >Terry J. Reedy added the comment: > My Firefox is already set at utf-8. More likely a font limitation. I > will look again after installing one of the fonts Tom suggested. Symbola is best for exotic glyphs, especially astral ones. Alfios just looks nice as a normal default roman. --tom -- ___ Python tracker <http://bugs.python.org/issue12730> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12730] Python's casemapping functions are untrustworthy due to narrow/wide build issues
Tom Christiansen added the comment: >Terry J. Reedy added the comment: > You are right, FF switched on me without notice. Bad FF. Thank you! What > I now see makes much more sense. >[ "𐐼𐐯𐑅𐐨𐑉𐐯𐐻", "𐐼𐐯𐑅𐐨𐑉𐐯𐐻", "𐐔𐐯𐑅𐐨𐑉𐐯𐐻", "𐐔𐐇𐐝𐐀𐐡𐐇𐐓" ], > and I now know to check on other pages (although Tom's Unicode talk > slides still have boxes even in utf-8, so that must be a font lack). Do you have Symbola installed? Here's Appendix I on Fonts for things that should look right for the presentation to look right. * I recommend two free fonts from George Douros at users.teilar.gr/~g1951d/ known to work with this presentation: his Alfios font for regular text, and his Symbola font for fancy emoji. If any of these don’t look right to you, you probably need to supplement your system fonts: Ligatures: fi ffi ff ffl fl β ẞ ſt st Math letters: 𝒜 𝒟 𝔅 𝔎 𝔼 𝔽 Gothic & Deseret: 𐌸𐌼𐌽𐍂, 𐐔𐐯𐑅𐐨𐑉𐐯𐐻 Symbols: ✔ ✅ 🐪 📖 🛂 🐍 Emoticons: 😇 😈 😉 😨 😭 😱 Upside‐down: ¡pɐəɥ ɹnoʎ uo ƃuᴉpuɐʇs ʎq sᴉɥʇ pɐəᴚ Combining characters: ◌̂,◌̃,◌⃞,◌̲,◌︀,◌̵,◌̷ * The last line with combining characters is especially hard to get to look right. You may find that the shareware font Everson Mono works when all else fails. You do need Unicode 5.1 support for the LATIN CAPITAL LETTER SHARP S, and you need Unicode 6.0 support for most of the emoji (I think Snow Leopard has colorized versions of these. The Ligature line above looks good in Alfios. It turns out it may not always the font used with combining chars as it is whether and well your browser supports true combining characters dynamically generated, or whether it runs stuff through NFC and looks for substitution glyphs. I am not a GUI person, so am mostly just guessing. But this I find interesting: If you look at slide 33 of my first talk or slide 5 of my second talk, which are duplicates entitled Canonical Conundra, the second column which is labelled Glyphs explicitly uses Time New Roman because of this issue. Even so you can tell it is doing the NFC trick, because lines 1+2 have the same NFC of \x{F5} or õ, as do 3+4+5 with \x{22D} with ȭ, and and 6+7 with ō̃. The glyphs from the first group are both identical, and so are all three those of the second group, as both the first two groups have a single precomposed character available for their NFC. In contrast, there is no single precomposed glyph available for 6+7, and you can tell that it's stacking it on the fly using slightly less tight grouping rules than the font has in the precomposed versions above it. I use Safari, but I am told Firefox looks ok, too. Opera is my normal browser but it does the copout I just described on combining chars without ever being able to dynamically stack them if the copout fail, so I can't use it for this presentation. --tom $ uniprops -a 'LATIN CAPITAL LETTER SHARP S' 'DESERET CAPITAL LETTER DEE' 'GOTHIC LETTER MANNA' 'SNAKE' 'FACE SCREAMING IN FEAR' U+1E9E <ẞ> \N{LATIN CAPITAL LETTER SHARP S} \w \pL \p{LC} \p{L_} \p{L&} \p{Lu} All Any Alnum Alpha Alphabetic Assigned InLatinExtendedAdditional Cased Cased_Letter LC Changes_When_Casefolded CWCF Changes_When_Casemapped CWCM Changes_When_Lowercased CWL Changes_When_NFKC_Casefolded CWKCF Lu L Gr_Base Grapheme_Base Graph GrBase ID_Continue IDC ID_Start IDS Letter L_ Latin Latn Latin_Extended_Additional Uppercase_Letter Print Upper Uppercase Word XID_Continue XIDC XID_Start XIDS X_POSIX_Alnum X_POSIX_Alpha X_POSIX_Graph X_POSIX_Print X_POSIX_Upper X_POSIX_Word Age=5.1 Bidi_Class=L Bidi_Class=Left_To_Right BC=L Block=Latin_Extended_Additional Canonical_Combining_Class=0 Canonical_Combining_Class=Not_Reordered CCC=NR Canonical_Combining_Class=NR Decomposition_Type=None DT=None East_Asian_Width=Neutral Grapheme_Cluster_Break=Other GCB=XX Grapheme_Cluster_Break=XX Hangul_Syllable_Type=NA Hangul_Syllable_Type=Not_Applicable HST=NA Joining_Group=No_Joining_Group JG=NoJoiningGroup Joining_Type=Non_Joining JT=U Joining_Type=U Script=Latin Line_Break=AL Line_Break=Alphabetic LB=AL Numeric_Type=None NT=None Numeric_Value=NaN NV=NaN Present_In=5.1 IN=5.1 Present_In=5.2 IN=5.2 Present_In=6.0 IN=6.0 SC=Latn Script=Latn Sentence_Break=UP Sentence_Break=Upper SB=UP Word_Break=ALetter WB=LE Word_Break=LE _X_Begin U+10414 <𐐔> \N{DESERET CAPITAL LETTER DEE} \w \pL \p{LC} \p{L_} \p{L&} \p{Lu} All Any Alnum Alpha Alphabetic Assigned InDeseret Cased Cased_Letter LC Changes_When_Casefolded CWCF Changes_When_Casemapped CWCM Changes_When_Lowercased CWL Changes_When_NF
[issue12734] Request for property support in Python re lib
Tom Christiansen added the comment: Sorry I didn't include a test case. Hope this makes up for it. If not, please tell me how to write better test cases. :( Yeah ok, so I'm a bit persnickety or even unorthodox about my vertical alignment, but it really helps to make what is different from one to line to the next stand out if the parts that are the same from line to line are at the same column every time. -- Added file: http://bugs.python.org/file22902/nametests.py ___ Python tracker <http://bugs.python.org/issue12734> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12734] Request for property support in Python re lib
Tom Christiansen added the comment: Oh whoops, that was the long ticket. Shall I reupload to the right number? -- ___ Python tracker <http://bugs.python.org/issue12734> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12730] Python's casemapping functions are untrustworthy due to narrow/wide build issues
Tom Christiansen added the comment: >Terry J. Reedy added the comment: >Adding Symbola filled in the symbols and emoticons lines. >The gothic chars are still missing even with Alfios. That's too bad, as the Gothic paternoster is kinda cute. :) Hm, I wonder where I got them from. I think there must be a way to figure that out using the Mac FontBook program, but I don't know what it is other than pasting them in the sample screen and scrolling through the fonts to see how those get rendered. --tom -- ___ Python tracker <http://bugs.python.org/issue12730> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12753] \N{...} neglects formal aliases and named sequences from Unicode charnames namespace
Tom Christiansen added the comment: Here’s the right test file for the right ticket. -- Added file: http://bugs.python.org/file22903/nametests.py ___ Python tracker <http://bugs.python.org/issue12753> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12734] Request for property support in Python re lib
Changes by Tom Christiansen : Removed file: http://bugs.python.org/file22902/nametests.py ___ Python tracker <http://bugs.python.org/issue12734> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10542] Py_UNICODE_NEXT and other macros for surrogates
Tom Christiansen added the comment: >Ezio Melotti added the comment: >I think the 4 macros: > #define _Py_UNICODE_ISSURROGATE > #define _Py_UNICODE_ISHIGHSURROGATE > #define _Py_UNICODE_ISLOWSURROGATE > #define _Py_UNICODE_JOIN_SURROGATES >are quite straightforward and can avoid using the trailing _. For what it's worth, I've seen Unicode documentation that talks about that prefers the terms "lead surrogate" and "trail surrogate" as being clearer than the terms "high surrgoate" and "low surrogate". For example, from the Unicode BOM FAQ at http://unicode.org/faq/utf_bom.html Q: What are surrogates? A: Surrogates are code points from two special ranges of Unicode values, reserved for use as the leading, and trailing values of paired code units in UTF-16. Leading, also called high, surrogates are from D800₁₆ to DBFF₁₆, and trailing, or low, surrogates are from DC00₁₆ to DFFF₁₆. They are called surrogates, since they do not represent characters directly, but only as a pair. BTW, considering recent discussions, you might want to read: Q: Are there any 16-bit values that are invalid? A: The two values FFFE₁₆ and ₁₆ as well as the 32 values from FDD0₁₆ to FDEF₁₆ represent noncharacters. They are invalid in interchange, but may be freely used internal to an implementation. Unpaired surrogates are invalid as well, i.e. any value in the range D800₁₆ to DBFF₁₆ not followed by a value in the range DC00₁₆ to DFFF₁₆, or any value in the range DC00₁₆ to DFFF₁₆ not preceded by a value in the range D800₁₆ to DBFF₁₆. [AF] and also the answer to: Q: Are there any paired surrogates that are invalid? whose answer I here omit for brevity, as it is a table. I suspect that you guys are now increasingly sold on the answer to the next FAQ right after that one, now. :) Q: Because supplementary characters are uncommon, does that mean I can ignore them? A: Just because supplementary characters (expressed with surrogate pairs in UTF-16) are uncommon does not mean that they should be neglected. They include: * emoji symbols and emoticons, for interoperating with Japanese mobile phones * uncommon (but not unused) CJK characters, important for personal and place names * variation selectors for ideographic variation sequences * important symbols for mathematics * numerous minority scripts and historic scripts, important for some user communities Another example of using "lead" and "trail" surrogates is in the first sentence from http://icu-project.org/apiref/icu4j/com/ibm/icu/text/UTF16.html * Naming: For clarity, High and Low surrogates are called Lead and Trail in the API, which gives a better sense of their ordering in a string. offset16 and offset32 are used to distinguish offsets to UTF-16 boundaries vs offsets to UTF-32 boundaries. int char32 is used to contain UTF-32 characters, as opposed to char16, which is a UTF-16 code unit. * Roundtripping Offsets: You can always roundtrip from a UTF-32 offset to a UTF-16 offset and back. Because of the difference in structure, you can roundtrip from a UTF-16 offset to a UTF-32 offset and back if and only if bounds(string, offset16) != TRAIL. * Exceptions: The error checking will throw an exception if indices are out of bounds. Other than than that, all methods will behave reasonably, even if unmatched surrogates or out-of-bounds UTF-32 values are present. UCharacter.isLegal() can be used to check for validity if desired. * Unmatched Surrogates: If the string contains unmatched surrogates, then these are counted as one UTF-32 value. This matches their iteration behavior, which is vital. It also matches common display practice as missing glyphs (see the Unicode Standard Section 5.4, 5.5). * Optimization: The method implementations may need optimization if the compiler doesn't fold static final methods. Since surrogate pairs will form an exceeding small percentage of all the text in the world, the singleton case should always be optimized for. You can also see this reflected in the utf.h file from the ICU project as part of their C API in ICU4C: #define U_SENTINEL (-1) This value is intended for sentinel values for APIs that (take or) return single code points (UChar32). #define U_IS_UNICODE_NONCHAR(c) Is this code point a Unicode noncharacter? #define U_IS_UNICODE_CHAR(c) Is c a Unicode code point value (0..U+10) that can be assigned a character? #define U_IS_BMP(c) ((uint32_t)(c)<=0x) Is this code point a BMP code point (U+..U+)? #define U_IS_SUPPLEMENTARY(c) ((uint32_t)((c)-0x1)<=0xf) Is this code
[issue10542] Py_UNICODE_NEXT and other macros for surrogates
Tom Christiansen added the comment: I now see there are lots of good things in the BOM FAQ that have come up lately regarding surrogates and other illegal characters, and about what can go in data streams. I quote a few of these from http://unicode.org/faq/utf_bom.html below: Q: How do I convert an unpaired UTF-16 surrogate to UTF-8? A: A different issue arises if an unpaired surrogate is encountered when converting ill-formed UTF-16 data. By represented such an *unpaired* surrogate on its own as a 3-byte sequence, the resulting UTF-8 data stream would become ill-formed. While it faithfully reflects the nature of the input, Unicode conformance requires that encoding form conversion always results in valid data stream. Therefore a converter *must* treat this as an error. Q: How do I convert an unpaired UTF-16 surrogate to UTF-32? A: If an unpaired surrogate is encountered when converting ill-formed UTF-16 data, any conformant converter must treat this as an error. By representing such an unpaired surrogate on its own, the resulting UTF-32 data stream would become ill-formed. While it faithfully reflects the nature of the input, Unicode conformance requires that encoding form conversion always results in valid data stream. Q: Can a UTF-8 data stream contain the BOM character (in UTF-8 form)? If yes, then can I still assume the remaining UTF-8 bytes are in big-endian order? A: Yes, UTF-8 can contain a BOM. However, it makes no difference as to the endianness of the byte stream. UTF-8 always has the same byte order. An initial BOM is only used as a signature — an indication that an otherwise unmarked text file is in UTF-8. Note that some recipients of UTF-8 encoded data do not expect a BOM. Where UTF-8 is used transparently in 8-bit environments, the use of a BOM will interfere with any protocol or file format that expects specific ASCII characters at the beginning, such as the use of "#!" of at the beginning of Unix shell scripts. Q: What should I do with U+FEFF in the middle of a file? A: In the absence of a protocol supporting its use as a BOM and when not at the beginning of a text stream, U+FEFF should normally not occur. For backwards compatibility it should be treated as ZERO WIDTH NON-BREAKING SPACE (ZWNBSP), and is then part of the content of the file or string. The use of U+2060 WORD JOINER is strongly preferred over ZWNBSP for expressing word joining semantics since it cannot be confused with a BOM. When designing a markup language or data protocol, the use of U+FEFF can be restricted to that of Byte Order Mark. In that case, any U+FEFF occurring in the middle of a file can be treated as an unsupported character. Q: How do I tag data that does not interpret U+FEFF as a BOM? A: Use the tag UTF-16BE to indicate big-endian UTF-16 text, and UTF-16LE to indicate little-endian UTF-16 text. If you do use a BOM, tag the text as simply UTF-16. Q: Why wouldn’t I always use a protocol that requires a BOM? A: Where the data has an associated type, such as a field in a database, a BOM is unnecessary. In particular, if a text data stream is marked as UTF-16BE, UTF-16LE, UTF-32BE or UTF-32LE, a BOM is neither necessary *nor permitted*. Any U+FEFF would be interpreted as a ZWNBSP. Do not tag every string in a database or set of fields with a BOM, since it wastes space and complicates string concatenation. Moreover, it also means two data fields may have precisely the same content, but not be binary-equal (where one is prefaced by a BOM). Somewhat frustratingly, I am now almost more confused than ever by the last two sentences here: Q: What is a UTF? A: A Unicode transformation format (UTF) is an algorithmic mapping from every Unicode code point (except surrogate code points) to a unique byte sequence. The ISO/IEC 10646 standard uses the term “UCS transformation format” for UTF; the two terms are merely synonyms for the same concept. Each UTF is reversible, thus every UTF supports *lossless round tripping*: mapping from any Unicode coded character sequence S to a sequence of bytes and back will produce S again. To ensure round tripping, a UTF mapping *must also* map all code points that are not valid Unicode characters to unique byte sequences. These invalid code points are the 66 *noncharacters* (including FFFE and ), as well as unpaired surrogates. My confusion is about the invalid code points. The first two FAQs I cite at the top are quite clear that it is illegal to have unpaired surrogates in a UTF stream. I don’t understand therefore what it saying about “must also” mapping all code points that aren’t valid Unicode characters to “unique byte sequences” to ensure roundtripping.
[issue10542] Py_UNICODE_NEXT and other macros for surrogates
Tom Christiansen added the comment: Antoine Pitrou wrote on Tue, 16 Aug 2011 09:18:46 -: >> I think the 4 macros: >> #define _Py_UNICODE_ISSURROGATE >> #define _Py_UNICODE_ISHIGHSURROGATE >> #define _Py_UNICODE_ISLOWSURROGATE >> #define _Py_UNICODE_JOIN_SURROGATES >> are quite straightforward and can avoid using the trailing _. > I don't want to bikeshed, but can we have proper consistent word separation? > _Py_UNICODE_IS_HIGH_SURROGATE, not _Py_UNICODE_ISHIGHSURROGATE > (etc.) Oh good, I thought it was only me whohadtroublereadingthose. :) --tom -- ___ Python tracker <http://bugs.python.org/issue10542> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10542] Py_UNICODE_NEXT and other macros for surrogates
Tom Christiansen added the comment: Ezio Melotti wrote on Tue, 16 Aug 2011 09:23:50 -: > All the other macros[0] follow the same convention, e.g. Py_UNICODE_ISLOWER > and Py_UNICODE_TOLOWER. I agree that keeping the words separate makes them > more readable though. > [0]: Include/unicodeobject.h:328 I am guessing that that is not quite why those don't have underscores in them. I bet it is actually something else. Watch: % unigrep '^\s*#\s*define\s+Py_[\p{Lu}_]+\b' unicodeobject.h #define Py_UNICODEOBJECT_H #define Py_USING_UNICODE #define Py_UNICODE_WIDE #define Py_UNICODE_ISSPACE(ch) \ #define Py_UNICODE_ISLOWER(ch) _PyUnicode_IsLowercase(ch) #define Py_UNICODE_ISUPPER(ch) _PyUnicode_IsUppercase(ch) #define Py_UNICODE_ISTITLE(ch) _PyUnicode_IsTitlecase(ch) #define Py_UNICODE_ISLINEBREAK(ch) _PyUnicode_IsLinebreak(ch) #define Py_UNICODE_TOLOWER(ch) _PyUnicode_ToLowercase(ch) #define Py_UNICODE_TOUPPER(ch) _PyUnicode_ToUppercase(ch) #define Py_UNICODE_TOTITLE(ch) _PyUnicode_ToTitlecase(ch) #define Py_UNICODE_ISDECIMAL(ch) _PyUnicode_IsDecimalDigit(ch) #define Py_UNICODE_ISDIGIT(ch) _PyUnicode_IsDigit(ch) #define Py_UNICODE_ISNUMERIC(ch) _PyUnicode_IsNumeric(ch) #define Py_UNICODE_ISPRINTABLE(ch) _PyUnicode_IsPrintable(ch) #define Py_UNICODE_TODECIMAL(ch) _PyUnicode_ToDecimalDigit(ch) #define Py_UNICODE_TODIGIT(ch) _PyUnicode_ToDigit(ch) #define Py_UNICODE_TONUMERIC(ch) _PyUnicode_ToNumeric(ch) #define Py_UNICODE_ISALPHA(ch) _PyUnicode_IsAlpha(ch) #define Py_UNICODE_ISALNUM(ch) \ #define Py_UNICODE_COPY(target, source, length) \ #define Py_UNICODE_FILL(target, value, length) \ #define Py_UNICODE_MATCH(string, offset, substring) \ #define Py_UNICODE_REPLACEMENT_CHARACTER ((Py_UNICODE) 0xFFFD) It looks like what is actually happening there is that you started out with names of the normal ctype(3) macroish thingies: isalpha isupper islower isdigit isxdigit isalnum isspace ispunct isprint isgraph iscntrl isblank isascii toupper isblank isascii toupper tolower toascii and wanted to preserve those, which would lead to Py_UNICODE_TOLOWER and Py_UNICODE_TOUPPER, since there are no functions in the original C versions those seem to mirror. Then when you wanted more of that ilk, you sensibly kept to the same naming convention. I eyeball few exceptions to that style here: % perl -nle '/^\s*#\s*define\s+(Py_[\p{Lu}_]+)\b/ and print $1' Include/*.h | sort -dfu | fmt -150 Py_ABSTRACTOBJECT_H Py_ALIGNED Py_ALLOW_RECURSION Py_ARITHMETIC_RIGHT_SHIFT Py_ASDL_H Py_AST_H Py_ATOMIC_H Py_BEGIN_ALLOW_THREADS Py_BITSET_H Py_BLOCK_THREADS Py_BLTINMODULE_H Py_BOOLOBJECT_H Py_BYTEARRAYOBJECT_H Py_BYTES_CTYPE_H Py_BYTESOBJECT_H Py_CAPSULE_H Py_CELLOBJECT_H Py_CEVAL_H Py_CHARMASK Py_CLASSOBJECT_H Py_CLEANUP_SUPPORTED Py_CLEAR Py_CODECREGISTRY_H Py_CODE_H Py_COMPILE_H Py_COMPLEXOBJECT_H Py_CURSES_H Py_DECREF Py_DEPRECATED Py_DESCROBJECT_H Py_DICTOBJECT_H Py_DTSF_ALT Py_DTSF_SIGN Py_DTST_FINITE Py_DTST_INFINITE Py_DTST_NAN Py_END_ALLOW_RECURSION Py_END_ALLOW_THREADS Py_ENUMOBJECT_H Py_EQ Py_ERRCODE_H Py_ERRORS_H Py_EVAL_H Py_FILEOBJECT_H Py_FILEUTILS_H Py_FLOATOBJECT_H Py_FORCE_DOUBLE Py_FORCE_EXPANSION Py_FORMAT_PARSETUPLE Py_FRAMEOBJECT_H Py_FUNCOBJECT_H Py_GCC_ATTRIBUTE Py_GE Py_GENOBJECT_H Py_GETENV Py_GRAMMAR_H Py_GT Py_HUGE_VAL Py_IMPORT_H Py_INCREF Py_INTRCHECK_H Py_INVALID_SIZE Py_ISALNUM Py_ISALPHA Py_ISDIGIT Py_IS_FINITE Py_IS_INFINITY Py_ISLOWER Py_IS_NAN Py_ISSPACE Py_ISUPPER Py_ISXDIGIT Py_ITEROBJECT_H Py_LE Py_LISTOBJECT_H Py_LL Py_LOCAL Py_LOCAL_INLINE Py_LONGINTREPR_H Py_LONGOBJECT_H Py_LT Py_MARSHAL_H Py_MARSHAL_VERSION Py_MATH_E Py_MATH_PI Py_MEMCPY Py_MEMORYOBJECT_H Py_METAGRAMMAR_H Py_METHODOBJECT_H Py_MODSUPPORT_H Py_MODULEOBJECT_H Py_NAN Py_NE Py_NODE_H Py_OBJECT_H Py_OBJIMPL_H Py_OPCODE_H Py_OSDEFS_H Py_OVERFLOWED Py_PARSETOK_H Py_PGEN_H Py_PGENHEADERS_H Py_PRINT_RAW Py_PYARENA_H Py_PYDEBUG_H Py_PYFPE_H Py_PYGETOPT_H Py_PYMATH_H Py_PYMEM_H Py_PYPORT_H Py_PYSTATE_H Py_PYTHON_H Py_PYTHONRUN_H Py_PYTHREAD_H Py_PYTIME_H Py_RANGEOBJECT_H Py_REFCNT Py_REF_DEBUG Py_RETURN_FALSE Py_RETURN_INF Py_RETURN_NAN Py_RETURN_NONE Py_RETURN_TRUE Py_SAFE_DOWNCAST Py_SET_ERANGE_IF_OVERFLOW Py_SET_ERRNO_ON_MATH_ERROR Py_SETOBJECT_H Py_SIZE Py_SLICEOBJECT_H Py_STRCMP_H Py_STRTOD_H Py_STRUCTMEMBER_H Py_STRUCTSEQ_H Py_SYMTABLE_H Py_SYSMODULE_H Py_TOKEN_H Py_TOLOWER Py_TOUPPER Py_TPFLAGS_BASE_EXC_SUBCLASS Py_TPFLAGS_BASETYPE Py_TPFLAGS_BYTES_SUBCLASS Py_TPFLAGS_DEFAULT Py_TPFLAGS_DICT_SUBCLASS Py_TPFLAGS_HAVE_GC Py_TPFLAGS_HAVE_STACKLESS_EXTENSION Py_TPFLAGS_HAVE_VERSION_TAG Py_TPFLAGS_HEAPTYPE Py_TPFLAGS_INT_SUBCLASS Py_TPFLAGS_IS_ABSTRACT Py_TPFLAGS_LIST_SUBCLASS Py_TPFLAGS_LONG_SUBCLASS Py_TPFLAGS_READY Py_TPFLAGS_READYING Py_TPFLAGS_TU
[issue10542] Py_UNICODE_NEXT and other macros for surrogates
Tom Christiansen added the comment: Marc-Andre Lemburg wrote on Tue, 16 Aug 2011 12:11:22 -: > The reasoning behind e.g. "ISSURROGATE" is that those names originate > from and are consistent with the already existing ISLOWER/ISUPPER/ISTITLE > macros which in return stem from the C APIs of the same names > (see unicodeobject.h for reference). I eventually figured that part out in the larger context. Makes sense looked at that way. > Regarding low/high vs. lead/trail: The Unicode database uses the terms > low/high and we do in Python as well, so let's stick with those. Yes, those are their block assignments, Block=High_Surrogates and Block=Low_Surrogates. I just thought I should mention that in the time since those were invented (which cannot be changed), after using them in real code for some years, their lingo seems to have evolved away from those initial names and toward lead/trail as less confusing. > What I don't understand is why those macros should be declared > private to Python (with the leading underscore). They are quite > useful for extensions implementing codecs or other transformations > as well. I was wondering about that myself. Beyond there being a lot fewer of those private macros in the Python *.h files, they also seem to be of rather different character than the iswhatever() macros: $ perl -nle '/^\s*#\s*define\s+(_Py_[\p{Lu}_]+)\b/ and print $1' *.h | sort -dfu | fmt -160 _Py_ANNOTATE_BARRIER_DESTROY _Py_ANNOTATE_BARRIER_INIT _Py_ANNOTATE_BARRIER_WAIT_AFTER _Py_ANNOTATE_BARRIER_WAIT_BEFORE _Py_ANNOTATE_BENIGN_RACE _Py_ANNOTATE_BENIGN_RACE_SIZED _Py_ANNOTATE_BENIGN_RACE_STATIC _Py_ANNOTATE_CONDVAR_LOCK_WAIT _Py_ANNOTATE_CONDVAR_SIGNAL _Py_ANNOTATE_CONDVAR_SIGNAL_ALL _Py_ANNOTATE_CONDVAR_WAIT _Py_ANNOTATE_ENABLE_RACE_DETECTION _Py_ANNOTATE_EXPECT_RACE _Py_ANNOTATE_FLUSH_STATE _Py_ANNOTATE_HAPPENS_AFTER _Py_ANNOTATE_HAPPENS_BEFORE _Py_ANNOTATE_IGNORE_READS_AND_WRITES_BEGIN _Py_ANNOTATE_IGNORE_READS_AND_WRITES_END _Py_ANNOTATE_IGNORE_READS_BEGIN _Py_ANNOTATE_IGNORE_READS_END _Py_ANNOTATE_IGNORE_SYNC_BEGIN _Py_ANNOTATE_IGNORE_SYNC_END _Py_ANNOTATE_IGNORE_WRITES_BEGIN _Py_ANNOTATE_IGNORE_WRITES_END _Py_ANNOTATE_MUTEX_IS_USED_AS_CONDVAR _Py_ANNOTATE_NEW_MEMORY _Py_ANNOTATE_NO_OP _Py_ANNOTATE_PCQ_CREATE _Py_ANNOTATE_PCQ_DESTROY _Py_ANNOTATE_PCQ_GET _Py_ANNOTATE_PCQ_PUT _Py_ANNOTATE_PUBLISH_MEMORY_RANGE _Py_ANNOTATE_PURE_HAPPENS_BEFORE_MUTEX _Py_ANNOTATE_RWLOCK_ACQUIRED _Py_ANNOTATE_RWLOCK_CREATE _Py_ANNOTATE_RWLOCK_DESTROY _Py_ANNOTATE_RWLOCK_RELEASED _Py_ANNOTATE_SWAP_MEMORY_RANGE _Py_ANNOTATE_THREAD_NAME _Py_ANNOTATE_TRACE_MEMORY _Py_ANNOTATE_UNPROTECTED_READ _Py_ANNOTATE_UNPUBLISH_MEMORY_RANGE _Py_AS_GC _Py_CHECK_REFCNT _Py_COUNT_ALLOCS_COMMA _Py_DEC_REFTOTAL _Py_DEC_TPFREES _Py_INC_REFTOTAL _Py_INC_TPALLOCS _Py_INC_TPFREES _Py_PARSE_PID _Py_REF_DEBUG_COMMA _Py_SET_EDOM_FOR_NAN > BTW: I think the other issues mentioned in the discussion are more > important to get right, than the names of those macros. Yup. Just paint it red. :) --tom -- ___ Python tracker <http://bugs.python.org/issue10542> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12753] \N{...} neglects formal aliases and named sequences from Unicode charnames namespace
Tom Christiansen added the comment: "Terry J. Reedy" wrote on Fri, 19 Aug 2011 22:50:58 -: > My current opinion is that adding the aliases might be done in current > releases. It certainly would serve the any user who does not know to > misspell 'FTHORA' as 'FHTORA' for just one of the 17 'FTHORA' chars. Yes, I think the 11 aliases pose no problem. It's amazing the trouble you get into from having a fat-fingered amanuesis typing your laws into indelible stone tablets. > Adding named sequences is definitely a feature request. The definition > of .lookup(name) would be enlarged to "Look up character by name, > alias, or named sequence" with reference to the specific files. The > meaning of \N{} would also have to be enlarged. But these do. The problem is bracketed character classes. Yes, if you got named reference into the regex compiler as a raw string, it could in theory rewrite [abc\N{seq}] as (?:[abc]|\N{seq}) but that doesn't help if the sequence got replaced as a string escape. At which point you have different behavior in the two lookalike cases. If you ask how we do this in Perl, the answer is "poorly". It really only works well in strings, not charclasses, although there is a proposal to do a rewrite during compilation like I've spelled out above. Seems messy for something that might(?) not get much use. But it would be nice for \N{} to work to access the whole namespace without prejudice. I have a feeling this may be a case of trying to keep one's cake and eating it too, as the two goals seem to rule each other out. >> "If you look at the ICU UCharacter class, you can see that they provide a >> more" > More what ;-) More expressive set of lookup functions where it is clear which thing you are getting. I believe the ICU regexes only support one-char returns for \N{...}, not multis per the sequences. But I may not be looking at the right docs for ICU; not sure. > I presume ICU =International Components for Unicode, icu-project.org/ > "Offers a portable set of C/C++ and Java libraries for Unicode support, > software internationalization (I18N) and globalization (G11N)." [appears > to be free, open source, and possibly usable within Python] Well, there are some Python bindings for ICU that I was eager to try out, because I wanted to see whether I couild get at full/real Unicode collation that way, but I had trouble getting the Python bindings to compile. Not sure why. The documentation for the Python bindings isn't very um wordy, and it isn't clear how tightly integrated it all is: there's talk about C++ strings that kind of scares me. :) Hm, and maybe they are only for Python 2 not Python 3, which I try to do all my Python stuff in because it seems like it has a better Unicode model. --tom -- ___ Python tracker <http://bugs.python.org/issue12753> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12753] \N{...} neglects formal aliases and named sequences from Unicode charnames namespace
Tom Christiansen added the comment: Matthew Barnett wrote on Fri, 19 Aug 2011 23:36:45 -: > For the "Line_Break" property, one of the possible values is > "Inseparable", with 2 permitted aliases, the shorter "IN" (which > is reasonable) and "Inseperable" (ouch!). Yeahy, I've shaken my head at that one, too. It's one thing to make an alias for something you typo'd in the first place, but to have something that's correct which you then make a typo alias for is just encouraging bad/sloppy/wrong behavior. Bidi_Class=Paragraph_Separator Bidi_Class=Common_Separator Bidi_Class=European_Separator Bidi_Class=Segment_Separator General_Category=Line_Separator General_Category=Paragraph_Separator General_Category=Separator General_Category=Space_Separator Line_Break=Inseparable Line_Break=Inseperable And there's still set, which makes you wonder why they couldn't spell at least *one* of them out: Sentence_Break=Sep SB=SE Sentence_Break=Sp SB=Sp You really have to look those up to realize they're two different things: SB ; SE; Sep SB ; SP; Sp And that none of them have something like SB=Space or SB=Separator so you know what you're talking about. Grrr. --tom -- ___ Python tracker <http://bugs.python.org/issue12753> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12735] request full Unicode collation support in std python library
Tom Christiansen added the comment: > Sounds like a fair feature request for Python 3.3, as long as the > intention is that users must import some module from the standard > library and use functions defined in that module. The operations and > methods defined for str instances (e.g. ==, <, etc.) should not change > their behavior. > Is there an existing 3rd party library that we could adopt (even if it isn't > perfect yet)? I *think* you could use ICU's. I'm pretty sure the Parrot people use ICU libraries. --tom -- ___ Python tracker <http://bugs.python.org/issue12735> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12737] str.title() is overzealous by upcasing combining marks inappropriately
Tom Christiansen added the comment: Guido van Rossum wrote on Fri, 26 Aug 2011 21:16:57 -: > Yeah, this should be fixed in 3.3 and probably backported to 3.2 > and 2.7. (There is already no guarantee that len(s) == > len(s.title()), right?) Well, *I* don't know of any such guarantee, but I don't know Python very well. In general, Unicode makes very few guarantees about casing. Under full casemapping, which is the only way to do the silly Turkish stuff amongst quite a bit else, any of the three casemappings can change the length of the string. Other things you can't rely on are round tripping and "single paths". By roundtripping, just look at the two lowercase sigmas and think about how you can't get back to one of them if you uppercase them both. By single paths, I mean that code that does some sort of conversion where it first lowercases everything and then titlecases the first letter can produce something different from titlecasing just the original first letter and then lowercasing the rest of them. That's because tc(x) and tc(lc(x)) can be different. --tom -- title: str.title() is overzealous by upcasing combining marks inappropriately -> str.title() is overzealous by upcasing combining marks inappropriately ___ Python tracker <http://bugs.python.org/issue12737> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12735] request full Unicode collation support in std python library
Tom Christiansen added the comment: Raymond Hettinger added the comment: > I would like to be involved in the design of the API for a UCA module > and its routines for loading Unicode Collation Element Tables (not > making the mistake of using global state like the locale module does). Is this the problem where a locale is global to a process (or thread)? The way I'm used to using the UCA module in Perl, that's never a problem, because it's completely object-oriented. There's no global state. You instantiate a collator object with all the state it needs, like collation_level upper_before_lower backwards_levels normalization override_CJK override_Hangul katakana_before_hiragana variable locale preprocess And then you use that object for all your collation needs, including not just sorting but also string comparison and even searches. For example, you could instantiate a first collator object with its level set to one, meaning just compare base alphanumerics not diacritics or case or nonletters, and a second with the defaults so that it uses all four levels or a different normalization. I have on occasion had more than one collator object around at once each with its own locale, like if I want to compare different locales' comparisons. --tom -- ___ Python tracker <http://bugs.python.org/issue12735> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12735] request full Unicode collation support in std python library
Tom Christiansen added the comment: I should probably mention the importance in the design of a UCA module of being able to specify which UCA version number you want it to behave like in case you plan to override some of the DUCET entries. That way if you run under a later UCA with different DUCET weights, your own tailorings will still make sense. If you don't do this, your collation tailorings can break in a new release of the UCA. --tom -- ___ Python tracker <http://bugs.python.org/issue12735> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12735] request full Unicode collation support in std python library
Tom Christiansen added the comment: Guido van Rossum wrote on Fri, 26 Aug 2011 21:55:03 -: > I know I sound like NIH, but I'm always reluctant to add a big 3rd > party lib like ICU to the permanent dependencies of all future Python > distros. If people want to use ICU they already can. OTOH I don't > have a better idea. :-( I know exactly what you mean. I would not want to push that on anyone, being dependent on a gigantic 3rd-party module. I just tried to answer the question. The only two full UCA implementations I know of are ICU's and Perl's, which does not use ICU (since we're UTF-8, etc). I just wish Python had Unicode collation, is all. --tom PS: (I haven't had good luck the ICU bindings in 3.2.) -- ___ Python tracker <http://bugs.python.org/issue12735> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12736] Request for python casemapping functions to use full not simple casemaps per Unicode's recommendation
Tom Christiansen added the comment: Guido van Rossum wrote on Fri, 26 Aug 2011 21:11:24 -: > Guido van Rossum added the comment: > I presume this applies to builtin str methods like .lower(), right? I > think it is a good thing to do for Python 3.3. Yes, the full casemaps are for upper, title, and lowercase. There is also a full casefold and turkic case fold (which is full), but you don't have a casefold function so I guess that doesn't matter. > We'd need to define what should happen in edge cases, e.g. when > (against all odds) a string happens to contain a lone surrogate or > some other code point or sequence of code points that the Unicode > standard considers illegal. I think it should not fail but just leave > those code points alone. Well, it's a funny thing. There are properties given for all Unicode code points, even noncharacter code points. This includes the casing properties, oddly enough. >From UnicodeData.txt, which has a few surrogate entries; notice no casing is given: D800;;Cs;0;L;N; DB7F;;Cs;0;L;N; DB80;;Cs;0;L;N; DBFF;;Cs;0;L;N; DC00;;Cs;0;L;N; DFFF;;Cs;0;L;N; And in SpecialCasing.txt, which does not have surrogates but does have a default clause: # This file is a supplement to the UnicodeData file. # It contains additional information about the casing of Unicode characters. # (For compatibility, the UnicodeData.txt file only contains case mappings for # characters where they are 1-1, and independent of context and language. # For more information, see the discussion of Case Mappings in the Unicode Standard. # # All code points not listed in this file that do not have a simple case mappings # in UnicodeData.txt map to themselves. And in CaseFolding.txt, which also does not have surrogates but again does have a default clause: # The data supports both implementations that require simple case foldings # (where string lengths don't change), and implementations that allow full case folding # (where string lengths may grow). Note that where they can be supported, the # full case foldings are superior: for example, they allow "MASSE" and "Maße" to match. # # All code points not listed in this file map to themselves. Taken all together, it follows that the surrogates have case{map,fold}s back to themselves, since they have no case{map,fold}s listed. It's ok to have arbitrary code points in memory, including surrogates and the 66 noncharacters. It just isn't legal to have them in a UTF stream for "open interchange", whatever that means. > Does this require us to import more data files from the Unicode > standard? By itself that doesn't scare me. One way or the other, yes, notably the SpecialCasing file for casemapping and the CaseFolding file for casefolding (which you should do anyway to fix re.I). But you can and should process the new files into some tighter format optimized for your own lookups. Oddly, Java doesn't provide for String methods that do full casing on titlecase, even those they do do so on lowercase and uppercase. On titlecase they only expose the simple casemaps via the Character class, which are the ones from UnicodeData. They recognize that this is flaw, but it was too late to fix it for JAva 7. > Would this also affect .islower() and friends? Well, it shouldn't, but .islower() and friends are already mistaken. They seem to be checking for GC=Ll and such, but they need to be checking the Unicode binary property Lowercase and such. Watch: test 37 for string Ⅷ wanted <ⅷ> to be lowercase of <Ⅷ> but python disagrees wanted <Ⅷ> to be titlecase of <Ⅷ> but python disagrees wanted <Ⅷ> to be uppercase of <Ⅷ> but python disagrees test 37 failed 3 subtests test 39 for string Ⓚ wanted <ⓚ> to be lowercase of <Ⓚ> but python disagrees wanted <Ⓚ> to be titlecase of <Ⓚ> but python disagrees wanted <Ⓚ> to be uppercase of <Ⓚ> but python disagrees test 39 failed 3 subtests That's because the Roman numerals are GC=Nl but still have case and change case. Similarly for the circled letters which are GC=So but have case and change case. Plus there's U+0345, the iota subscript, which is GC=Mn but has case and changes case. I don't remember whether I've sent in my full test suite or not. If I haven't yet, I should attach it to the bug report. --tom -- ___ Python tracker <http://bugs.python.org/issue12736> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12736] Request for python casemapping functions to use full not simple casemaps per Unicode's recommendation
Tom Christiansen added the comment: Here’s my casing test suite; I thought I sent it in but the mux file here isn’t the full thing. It does several things, including letting you run it with regex vs re. It also checks for the islower, etc functions. It has both simple and full (and turkic) maps and folds in it, but is configured to only check the simple versions for now. The islower and isupper etc functions seem to be checking the wrong Unicode property. Yes, it has my quaint Unixisms in it, because it needs to run with UTF-8 output, or you can't read what's going on. -- Added file: http://bugs.python.org/file23051/casing-tests.py ___ Python tracker <http://bugs.python.org/issue12736> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12729] Python lib re cannot handle Unicode properly due to narrow/wide bug
Tom Christiansen added the comment: Guido van Rossum wrote on Sat, 27 Aug 2011 03:26:21 -: > To me, making (default) iteration deviate from indexing is anathema. So long is there's a way to interate through a string some other way that by code unit, that's fine. However, the Java way of 16-bit code units is so annoying because there often aren't code point APIs, and so you get a lot of niggling errors creeping in. This is part of why I strongly prefer wide builds, so that code point and code unit are the same thing again. > However, there is nothing wrong with providing a library function that > takes a string and returns an iterator that iterates over code points, > joining surrogate pairs as needed. You could even have one that > iterates over characters (I think Tom calls them graphemes), if that > is well-defined and useful. "Character" can sometimes be a confusing term when it means something different to us programmers as it does to users. Code point to mean the integer is a lot clearer to us but to no one else. At work I often just give in and go along with the crowd and say character for the number that sits in a char or wchar_t or Character variable, even though of course that's a code point. I only rebel when they start calling code units characters, which (inexperienced) Java people tend to do, because that leads to surrogate splitting and related errors. By grapheme I mean something the user perceives as a single character. In full Unicodese, this is an extended grapheme cluster. These are code point sequences that start with a grapheme base and have zero or more grapheme extenders following it. For our purposes, that's *mostly* like saying you have a non-Mark followed by any number of Mark code points, the main excepting being that a CR followed by a LF also counts as a single grapheme in Unicode. If you are in an editor and wanted to swap two "characters", the one under the user's cursor and the one next to it, you have to deal with graphemes not individual code points, or else you'd get the wrong answer. Imagine swapping the last two characters of the first string below, or the first two characters of second one: contrôléecontro\x{302}le\x{301}e élèvee\x{301}le\x{300}ve While you can sometimes fake a correct answer by considering things in NFC not NFD, that's doesn't work in the general case, as there are only a few compatibility glyphs for round-tripping for legacy encodings (like ISO 8859-1) compared with infinitely many combinations of combining marks. Particularly in mathematics and in phonetics, you often end up using marks on characters for which no pre-combined variant glyph exists. Here's the IPA for a couple of Spanish words with their tight (phonetic, not phonemic) transcriptions: anécdota[a̠ˈne̞ɣ̞ð̞o̞t̪a̠] rincón [rĩŋˈkõ̞n] NFD: ane\x{301}cdota [a\x{320}\x{2C8}ne\x{31E}\x{263}\x{31E}\x{F0}\x{31E}o\x{31E}t\x{32A}a\x{320}] rinco\x{301}n [ri\x{303}\x{14B}\x{2C8}ko\x{31E}\x{303}n] NFD: an\x{E9}cdota [a\x{320}\x{2C8}ne\x{31E}\x{263}\x{31E}\x{F0}\x{31E}o\x{31E}t\x{32A}a\x{320}] rinc\x{F3}n [r\x{129}\x{14B}\x{2C8}k\x{F5}\x{31E}n] So combining marks don't "just go away" in NFC, and you really do have to deal with them. Notice that to get the tabs right (your favorite subject :), you have to deal with print widths, which is another place that you get into trouble if you only count code points. BTW, did you know that the stress mark used in the phonetics above is actually a (modifier) letter in Unicode, not punctuation? # uniprops -a 2c8 U+02C8 ‹ˈ› \N{MODIFIER LETTER VERTICAL LINE} \w \pL \p{L_} \p{Lm} All Any Alnum Alpha Alphabetic Assigned InSpacingModifierLetters Case_Ignorable CI Common Zyyy Dia Diacritic L Lm Gr_Base Grapheme_Base Graph GrBase ID_Continue IDC ID_Start IDS Letter L_ Modifier_Letter Print Spacing_Modifier_Letters Word XID_Continue XIDC XID_Start XIDS X_POSIX_Alnum X_POSIX_Alpha X_POSIX_Graph X_POSIX_Print X_POSIX_Word Age=1.1 Bidi_Class=ON Bidi_Class=Other_Neutral BC=ON Block=Spacing_Modifier_Letters Canonical_Combining_Class=0 Canonical_Combining_Class=Not_Reordered CCC=NR Canonical_Combining_Class=NR Script=Common Decomposition_Type=None DT=None East_Asian_Width=Neutral Grapheme_Cluster_Break=Other GCB=XX Grapheme_Cluster_Break=XX Hangul_Syllable_Type=NA Hangul_Syllable_Type=Not_Applicable HST=NA Joining_Group=No_Joining_Group JG=NoJoiningGroup Joining_Type=Non_Joining JT=U Joining_Type=U Line_Break=BB Line_Break=Break_Before LB=BB Numeric_Type=None NT=None Numeric_Value=NaN NV=NaN Present_In=1.1 IN=1.1 Present_In=2.0 IN=2.0 Present_In=2.1 IN=2.1 Present_In=3.0 IN=3.0 Present_In=3.1 IN=3.1 Present_In=3.2 IN=3.2 Present_In=4.0 IN=4.0 Present_In=4.1 IN=4.1 Present_In=5.0 IN=5.0 P
[issue12736] Request for python casemapping functions to use full not simple casemaps per Unicode's recommendation
Tom Christiansen added the comment: Guido van Rossum wrote on Fri, 26 Aug 2011 21:11:24 -: > Would this also affect .islower() and friends? SHORT VERSION: (7 lines) I don't believe so, but the relationship between lower() and islower() is not as clear to me as I would have thought, and more importantly, the code and the documentation for Python's islower() etc currently seem to disagree. For future releases, I recommend fixing the code, but if compatibility is an issue, then perhaps for previous releases still in maintenance mode fixing only the documentation would possibly be good enough--your call. === MEDIUM VERSION: (87 lines) I was initially confused with Python's islower() family because of the way they are defined to operate on full strings. They don't check that everything is lowercase even though they say they do. < http://docs.python.org/py3k/library/stdtypes.html#sequence-types-str-bytes-bytearray-list-tuple-range str.lower() Return a copy of the string with all the cased characters [4] converted to lowercase. str.islower() Return true if all cased characters [4] in the string are lowercase and there is at least one cased character, false otherwise. [4] (1, 2, 3, 4) Cased characters are those with general category property being one of “Lu” (Letter, uppercase), “Ll” (Letter, lowercase), or “Lt” (Letter, titlecase). This is strange in several ways. Of lesser importance is that strings can be considered lowercase even if they don't match ^\p{lowercase}+$ Another is that the result of calling str.lower() may not be .islower(). I'm not sure what these are particularly for, since I myself would just use a regex to get finer-grained control. (I suppose that's because re doesn't give access to the Unicode properties needed that this approach never gained any traction in the Python community.) However, the worst of this is that the documentation defines both cased characters and lowercase characters *differently* from how Unicode does defines those very same terms. This was quite confusing. Unicode distinguishes Cased code points from Cased_*Letter* code points. Python is using the Cased_Letter property but calling it Cased. Cased in a proper superset of Cased_Letter. From the DerivedCoreProperties file in the Unicode Character Database: # Derived Property: Cased (Cased) # As defined by Unicode Standard Definition D120 # C has the Lowercase or Uppercase property or has a General_Category value of Titlecase_Letter. In the same way, the Lowercase and Uppercase properties are not the same as the Lowercase_*Letter* and Uppercase_*Letter* properties. Rather, the former are respectively proper supersets of the latter. # Derived Property: Lowercase # Generated from: Ll + Other_Lowercase [...] # Derived Property: Uppercase # Generated from: Lu + Other_Uppercase In all these, you almost always want the superset versions not the restricted subset versions you are using. If it were in the regex engine, the user could select either. Java used to miss all these, too. But in 1.7, they updated their character methods to use the properties that they'd all along said they were using: < http://download.oracle.com/javase/7/docs/api/java/lang/Character.html#isLowerCase(char) public static boolean isLowerCase(char ch) Determines if the specified character is a lowercase character. A character is lowercase if its general category type, provided by Character.getType(ch), is LOWERCASE_LETTER, or it has contributory -> property Other_Lowercase as defined by the Unicode Standard. Note: This method cannot handle supplementary characters. To support all Unicode characters, including supplementary characters, use the isLowerCase(int) method. (And yes, that's where Java uses "character" to mean "code unit" not "code point", alas. No wonder people get confused) I'm pretty sure that Python needs to either update its documentation to match its code, update its code to match its documentation, or both. Java chose to update the code to match the documentation, and this is the course I would recommend if at all possible. If you say you are checking for cased code points, then you should use the Unicode definition of cased code points not your own, and if you say you are checking for lowercase code points, then you should use the Unicode definition not your own. Both of these require access to contributory properties from the UCD and not just general categories alone. --tom === LONG VERSION: (222 lines) Essential tools I use for inspecting Unicode
[issue12736] Request for python casemapping functions to use full not simple casemaps per Unicode's recommendation
Tom Christiansen added the comment: Guido van Rossum wrote on Sat, 27 Aug 2011 16:15:33 -: > Although personally I don't have much of an intuition for what > titlecase means (and why it's important), perhaps because I'm not > familiar with any language where there is a third case for some > letters. Neither am I. Even in "old-style" English with ae and oe, one wrote ÆGYPT and ÆSIR all caps but Ægypt and Æsir in titlecase, not *Aegypt or *Aesir. Similarly with ŒNOLOGY / Œnology / œnology, never *Oenology. (BTW, in French you really shouldn't split up the œ into oe, nor in Old English, Old Norse, or Icelandic the æ in ae; although in contemporary English, it's usually ok to do so.) I believe that almost but not quite all the sticky situations with Unicode casing involve compatibility characters for clean round-trips with legacy encodings. Exceptions include the German sharp s (both of them now) and the two Greek lowercase sigmas. Thank goodness we don't use the long s in English anymore. What is it with s's, anyway? :) Most of the titlecase letters are in Greek, with a few in Armenian. I know no Armenian (their letters all look the same to me :), and the folks I talked to about the Greek are skeptical. The German sharp s is a red herring, because you can never have it as the first letter (although it needn't be the last, as in Rußland). That's no more possible than having the old legacy ff ligature appear at the beginning of an English world. In any event, there are only 129 total code points that are "problematic" in terms of their case, where by problematic I mean one or more of: --- titlecase differs from uppercase --- foldcase differs from lowercase --- any of fold/lower/title/uppercase yields more than one code point Of all these, it's the (now two!) sharp s's and the Turkic i that are the most annoying. It's really quite a lot of trouble to go through for so few code points of so little (perceived) use. But I suppose you never know what new ones they'll uncover, either. Here are those 129 case-problematicals arranged in UCA order. Some of these normilizations forms that decompose into graphemes with four code points (not shown). There are a few other oddities, like the Kelvin sign and other "singletons", but these are most of the trouble. They're all in the BMP; I guess we learned our lesson. :) --tom 1: U+0345 ○ͅ COMBINING GREEK YPOGEGRAMMENI fc=ι U+3B9 lc=○ͅ U+345 tc=Ι U+399 uc=Ι U+399 2: U+1E9A ẚ LATIN SMALL LETTER A WITH RIGHT HALF RING fc=aʾ U+61.2BE lc=ẚ U+1E9A tc=Aʾ U+41.2BE uc=Aʾ U+41.2BE 3: U+01F3 dz LATIN SMALL LETTER DZ fc=dz U+1F3 lc=dz U+1F3 tc=Dz U+1F2 uc=DZ U+1F1 4: U+01F2 Dz LATIN CAPITAL LETTER D WITH SMALL LETTER Z fc=dz U+1F3 lc=dz U+1F3 tc=Dz U+1F2 uc=DZ U+1F1 5: U+01F1 DZ LATIN CAPITAL LETTER DZ fc=dz U+1F3 lc=dz U+1F3 tc=Dz U+1F2 uc=DZ U+1F1 6: U+01C6 dž LATIN SMALL LETTER DZ WITH CARON fc=dž U+1C6 lc=dž U+1C6 tc=Dž U+1C5 uc=DŽ U+1C4 7: U+01C5 Dž LATIN CAPITAL LETTER D WITH SMALL LETTER Z WITH CARON fc=dž U+1C6 lc=dž U+1C6 tc=Dž U+1C5 uc=DŽ U+1C4 8: U+01C4 DŽ LATIN CAPITAL LETTER DZ WITH CARON fc=dž U+1C6 lc=dž U+1C6 tc=Dž U+1C5 uc=DŽ U+1C4 9: U+FB00 ff LATIN SMALL LIGATURE FF fc=ff U+66.66 lc=ff U+FB00 tc=Ff U+46.66 uc=FF U+46.46 10: U+FB03 ffi LATIN SMALL LIGATURE FFI fc=ffi U+66.66.69 lc=ffi U+FB03 tc=Ffi U+46.66.69 uc=FFI U+46.46.49 11: U+FB04 ffl LATIN SMALL LIGATURE FFL fc=ffl U+66.66.6C lc=ffl U+FB04 tc=Ffl U+46.66.6C uc=FFL U+46.46.4C 12: U+FB01 fi LATIN SMALL LIGATURE FI fc=fi U+66.69 lc=fi U+FB01 tc=Fi U+46.69 uc=FI U+46.49 13: U+FB02 fl LATIN SMALL LIGATURE FL fc=fl U+66.6C lc=fl U+FB02 tc=Fl U+46.6C uc=FL U+46.4C 14: U+1E96 ẖ LATIN SMALL LETTER H WITH LINE BELOW fc=ẖ U+68.331 lc=ẖ U+1E96 tc=H̱ U+48.331 uc=H̱ U+48.331 15: U+0130 İ LATIN CAPITAL LETTER I WITH DOT ABOVE fc=i̇ U+69.307 lc=i̇ U+69.307 tc=İ U+130 uc=İ U+130 16: U+01F0 ǰ LATIN SMALL LETTER J WITH CARON fc=ǰ U+6A.30C lc=ǰ U+1F0 tc=J̌ U+4A.30C uc=J̌ U+4A.30C 17: U+01C9 lj LATIN SMALL LETTER LJ fc=lj U+1C9 lc=lj U+1C9 tc=Lj U+1C8 uc=LJ U+1C7 18: U+01C8 Lj LATIN CAPITAL LETTER L WITH SMALL LETTER J fc=lj U+1C9 lc=lj U+1C9 tc=Lj U+1C8 uc=LJ U+1C7 19: U+01C7 LJ LATIN CAPITAL LETTER LJ fc=lj U+1C9 lc=lj U+1C9 tc=Lj U+1C8 uc=LJ U+1C7 20: U+01CC nj LATIN SMALL LETTER NJ fc=nj U+1CC lc=nj U+1CC tc=Nj U+1CB uc=NJ U+1CA 21: U+01CB Nj LATIN CAPITAL LETTER N WITH SMALL LETTER J fc=nj U+1CC lc=nj U+1CC tc=Nj U+1CB uc=NJ U+1CA 22: U+01CA
[issue12736] Request for python casemapping functions to use full not simple casemaps per Unicode's recommendation
Tom Christiansen added the comment: Antoine Pitrou wrote on Sat, 27 Aug 2011 20:04:56 -: >> Neither am I. Even in "old-style" English with ae and oe, one wrote >> ÆGYPT and ÆSIR all caps but Ægypt and Æsir in titlecase, not *Aegypt or >> *Aesir. Similarly with ŒNOLOGY / Œnology / œnology, never *Oenology. > Trying to disprove you a bit: > http://ecx.images-amazon.com/images/I/51G6CH9XFFL._SL500_AA300_.jpg > http://ecx.images-amazon.com/images/I/51k7TmosPdL._SL500_AA300_.jpg > http://ecx.images-amazon.com/images/I/518UzMeLFCL._SL500_AA300_.jpg > but classical typographies seem to write either the uppercase Œ or the > lowercase œ. That's what I meant: one only ever sees œufs or ŒUFS, never OEUFS. French doesn't fit into ISO 8859-1. That's one of the changes to ISO-8859-15 compared with ISO-8859-1 (and Unicode): iso-8859-1 A4 ⇔ U+00A4 < ¤ > \N{CURRENCY SIGN} iso-8859-15 A4 ⇒ U+20AC < € > \N{EURO SIGN} iso-8859-1 A6 ⇔ U+00A6 < ¦ > \N{BROKEN BAR} iso-8859-15 A6 ⇒ U+0160 < Š > \N{LATIN CAPITAL LETTER S WITH CARON} iso-8859-1 A8 ⇔ U+00A8 < ¨ > \N{DIAERESIS} iso-8859-15 A8 ⇒ U+0161 < š > \N{LATIN SMALL LETTER S WITH CARON} iso-8859-1 B4 ⇔ U+00B4 < ´ > \N{ACUTE ACCENT} iso-8859-15 B4 ⇒ U+017D < Ž > \N{LATIN CAPITAL LETTER Z WITH CARON} iso-8859-1 B8 ⇔ U+00B8 < ¸ > \N{CEDILLA} iso-8859-15 B8 ⇒ U+017E < ž > \N{LATIN SMALL LETTER Z WITH CARON} iso-8859-1 BC ⇔ U+00BC < ¼ > \N{VULGAR FRACTION ONE QUARTER} iso-8859-15 BC ⇒ U+0152 < Œ > \N{LATIN CAPITAL LIGATURE OE} iso-8859-1 BD ⇔ U+00BD < ½ > \N{VULGAR FRACTION ONE HALF} iso-8859-15 BD ⇒ U+0153 < œ > \N{LATIN SMALL LIGATURE OE} iso-8859-1 BE ⇔ U+00BE < ¾ > \N{VULGAR FRACTION THREE QUARTERS} iso-8859-15 BE ⇒ U+0178 < Ÿ > \N{LATIN CAPITAL LETTER Y WITH DIAERESIS} > That said, I wonder why Unicode even includes ligatures like ff. Sounds > like mission creep to me (and horrible annoyances for people like us). I'm pretty sure that typographic ligatures are there for roundtripping with legacy encodings. I believe that œ/Œ is the only code point with ligature in its name that you're "supposed" to still use, and that all others should be figured out by modern fonting software. --tom -- ___ Python tracker <http://bugs.python.org/issue12736> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12736] Request for python casemapping functions to use full not simple casemaps per Unicode's recommendation
Tom Christiansen added the comment: Antoine Pitrou wrote on Mon, 29 Aug 2011 13:21:06 -: > It's not only "typographically speaking", it's really a spelling error, > even in hand-written text :-) Sure, and so too is omitting an accent mark or diaeresis. But—alas!—you’ll never convince most monoglot anglophones of that, the ones who keep wanting to strip them from résumé, façade, châteaux, crème brûlée, fête, tête-à-tête, à la française, or naïveté, not to mention José, jalapeño, the erstwhile American Secretary of State Federico Peña, or nearby Cañon City, Colorado, where I have family. I think œnonlogy has survived solely on its rarity, and the Encyclopædia Britannica is that way because the ligat(ur)ed letter is in their actual trademark. Cell phone users sending text messages have long suffered the grievous injuries to their language(s) that naked ASCII imparts, but this is nothing like the crossdressing nightmare called Greeklish, also variously known as Grenglish, Latinoellinika/Λατινοελληνικά, or ASCII Greek. http://en.wikipedia.org/wiki/Greeklish [...] The reason for this is the fact that text written in Greeklish is considerably less aesthetically pleasing, and also much harder to read, compared to text written in the Greek alphabet. A non-Greek speaker/reader can guess this by this example: "δις ιζ χαρντ του ριντ" would be the way to write "this is hard to read" in English but utilizing the Greek alphabet. I especially enjoy George Baloglou’s "Byzantine" Grenglish, wherein: Ὀδυσσεύς=> Oducceusinstead of Odysseus Ἀχιλλεύς=> Axilleusinstead of Achilleus Σίσυφος => Sicuphosinstead of Sisyphus Περικλῆς=> 5epiklhsinstead of Pericles Χθονός => X8onos instead of Chthonos Οι Ατρείδες=> Oi Atpeides instead of the Atreïdes Terrible though the depredations upon the French language that may have been committed by ASCII, surely these go even further. :) --tom Η ΙλιάδαH Iliada Μῆνιν ἄειδε, θεὰ, Πηληϊάδεω Ἀχιλῆος Mhnin aeide, 8ea, 5hlhiadeo Axilhos οὐλομένην, ἣ μυρί’ Ἀχαιοῖς ἄλγε’ ἔθηκε, oulomenhn, 'h mupi’ Axaiois alge’ e8hke, πολλὰς δ’ ἰφθίμους ψυχὰς Ἄϊδι προῒαψενnollas d’ iph8imous yuxas Aidi npoiayen ἡρώων, αὐτοὺς δὲ ἑλώρια τεῦχε κύνεσσιν'hpoon, autous de elopia teuxe kuneccin οἰωνοῖσί τε πᾶσι· Διὸς δ’ ἐτελείετο βουλή·oionoici te naci· Dios d’ eteleieto boulh· ἐξ οὗ δὴ τὰ πρῶτα διαστήτην ἐρίσαντε eks o'u dh ta npota diacththn epicante Ἀτρεΐδης τε ἄναξ ἀνδρῶν καὶ δῖος Ἀχιλλεύς.Atpeidhs te anaks andpon kai dios Axilleus. -- ___ Python tracker <http://bugs.python.org/issue12736> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12729] Python lib re cannot handle Unicode properly due to narrow/wide bug
Tom Christiansen added the comment: Ezio Melotti wrote on Sat, 03 Sep 2011 00:28:03 -: > Ezio Melotti added the comment: > Or they are still called UTF-8 but used in combination with different error > handlers, like surrogateescape and surrogatepass. The "plain" UTF-* codecs > should produce data that can be used for "open interchange", rejecting all the > invalid data, both during encoding and decoding. > Chapter 03, D79 also says: >To ensure that the mapping for a Unicode encoding form is one-to-one, >all Unicode scalar values, including those corresponding to >noncharacter code points and unassigned code points, must be mapped to >unique code unit sequences. Note that this requirement does not extend >to high-surrogate and low-surrogate code points, which are excluded by >definition from the set of Unicode scalar values. > and this seems to imply that the only unencodable codepoint are the non-scalar > values, i.e. surrogates and codepoints >U+10. Noncharacters shouldn't > thus receive any special treatment (at least during encoding). > Tom, do you agree with this? What does Perl do with them? I agree that one needs to be able to encode any scalar value and store it in memory in a designated character encoding form. This is different from streams, though. The 3 different Unicode "character encoding *forms*" -- UTF-8, UTF-16, and UTF-32 -- certainly need to support all possible scalar values. These are the forms used to store code points in memory. They do not have BOMs, because one knows one's memory layout. These are specifically allowed to contain the noncharacters: http://www.unicode.org/reports/tr17/#CharacterEncodingForm The third type is peculiar to the Unicode Standard: the noncharacter. This is a kind of internal-use user-defined character, not intended for public interchange. The problem is that one must make a clean distinction between character encoding *forms* and character encoding *schemes*. http://www.unicode.org/reports/tr17/#CharacterEncodingScheme It is important not to confuse a Character Encoding Form (CEF) and a CES. 1. The CEF maps code points to code units, while the CES transforms sequences of code units to byte sequences. 2. The CES must take into account the byte-order serialization of all code units wider than a byte that are used in the CEF. 3. Otherwise identical CESs may differ in other aspects, such as the number of user-defined characters allowed. Some of the Unicode encoding schemes have the same labels as the three Unicode encoding forms. [...] As encoding schemes, UTF-16 and UTF-32 refer to serialized bytes, for example the serialized bytes for streaming data or in files; they may have either byte orientation, and a single BOM may be present at the start of the data. When the usage of the abbreviated designators UTF-16 or UTF-32 might be misinterpreted, and where a distinction between their use as referring to Unicode encoding forms or to Unicode encoding schemes is important, the full terms should be used. For example, use UTF-16 encoding form or UTF-16 encoding scheme. They may also be abbreviated to UTF-16 CEF or UTF-16 CES, respectively. The Unicode Standard has seven character encoding schemes: UTF-8, UTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32BE, and UTF-32LE. * UTF-8, UTF-16BE, UTF-16LE, UTF-32BE and UTF32-LE are simple CESs. * UTF-16 and UTF-32 are compound CESs, consisting of an single, optional byte order mark at the start of the data followed by a simple CES. I believe that what this comes down to is that you can have noncharacters in memory as a CEF, but that you cannot have them in a CES meant for open interchange. And what you do privately is a different, third matter. What Perl does differs somewhat depending on whether you are just playing around with encodings in memory verus using streams that have particular encodings associated with them. I belive that you can think of this as the first being for CEF stuff and the second is for CES stuff. Streams are strict. Memory isn't. Perl will never ever produce nor accept one of the 66 noncharacers on any stream marked as one of the 7 character encoding schemes. However, we aren't always good about whether we generate an exception or whether we return replacement characters. Here the first process created a (for the nonce, nonfatal) warning, whereas the second process raised an exception: % perl -wle 'binmode(STDOUT, "encoding(UTF-16)")|| die; print chr(0xFDD0)' | perl -wle 'binmode(STDIN, "encoding(UTF-16)")||die; print ord ' Unicode non-character U+FDD0 is illegal for open interchange at -e line 1. UTF-16:Un
[issue12729] Python lib re cannot handle Unicode properly due to narrow/wide bug
Tom Christiansen added the comment: "Terry J. Reedy" wrote on Thu, 08 Sep 2011 18:56:11 -: >On 9/8/2011 4:32 AM, Ezio Melotti wrote: >> So to summarize a bit, there are different possible level of strictness: >>1) all the possible encodable values, including the ones>10; >>2) values in range 0..10; >>3) values in range 0..10 except surrogates (aka scalar values); >>4) values in range 0..10 except surrogates and noncharacters; >> and this is what is currently available in Python: >>1) not available, probably it will never be; >>2) available through the 'surrogatepass' error handler; >>3) default behavior (i.e. with the 'strict' error handler); >>4) currently not available. >> Now, assume that we don't care about option 1 and want to implement the >> missing option 4 (which I'm still not 100% sure about). The possible >> options are: >>* add a new codec (actually one for each UTF encoding); >>* add a new error handler that explicitly disallows noncharacters; >>* change the meaning of 'strict' to match option 4; > If 'strict' meant option 4, then 'scalarpass' could mean option 3. > 'surrogatepass' would then mean 'pass surragates also, in addition to > non-char scalers'. I'm pretty sure that anything that claims to be UTF-{8,16,32} needs to reject both surrogates *and* noncharacters. Here's something from the published Unicode Standard's p.24 about noncharacter code points: • Noncharacter code points are reserved for internal use, such as for sentinel values. They should never be interchanged. They do, however, have well-formed representations in Unicode encoding forms and survive conversions between encoding forms. This allows sentinel values to be preserved internally across Unicode encoding forms, even though they are not designed to be used in open interchange. And here from the Unicode Standard's chapter on Conformance, section 3.2, p. 59: C2 A process shall not interpret a noncharacter code point as an abstract character. • The noncharacter code points may be used internally, such as for sentinel values or delimiters, but should not be exchanged publicly. I'd have to check the fine print, but I am pretty sure that "shall not" is an imperative form. We have understand that to read that a comforming process *must*not* do that. It's because of that wording that in Perl, using either of {en,de}code() with any of the "UTF-{8,16,32}" encodings, including the LE/BE versions as appropriate, it will not produce nor accept a noncharacter code point like FDD0 or FFFE. Do you think we may perhaps have misread that conformance clause? Using Perl's special, loose-fitting "utf8" encoding, you can get it do noncharacter code points and even surrogates, but you have to suppress certain things to make that happen quietly. You can only do this with "utf8", not any of the UTF-16 or UTF-32 flavors. There we give them no choice, so you must be strict. I agree this is not fully orthogonal. Note that this is the normal thing that people do: binmode(STDOUT, ":utf8"); which is the *loose* version. The strict one is "utf8-strict" or "UTF-8": open(my $fh, "< :encoding(UTF-8)", $pathname) So it is a bit too easy to get the loose one. We felt we had to do this because we were already using the loose definition (and allowing up to chr(2**32) etc) when the Unicode Consortium made clear what sorts of things must not be accepted, or perhaps, before we made ourselves clear on this. This will have been back in 2003, when I wasn't paying very close attention. I think that just like Perl, Python has a legacy of the original loose definition. So some way to accommodate that legacy while still allowing for a comformant application should be devised. My concern with Python is that people tend to make they own manual calls to encode/decode a lot more often than they do in Perl. That people that if you only catch it on a stream encoding, you'll miss it, because they will use binary I/O and miss the check. --tom Below I show a bit of how this works in Perl. Currently the builtin utf8 encoding is controlled somewhat differently from how the Encode module's encode/decode functions are. Yes, this is not my idea of good. This shows that noncharacters and surrogates do not survive the encoding/decoding process for UTF-16: % perl -CS -MEncode -wle 'print decode("UTF-16", encode("UTF-16", chr(0xFDD0)))' | uniquote -v \N{REPLACEMENT CHARACTER} % perl -CS -MEncode -
[issue12729] Python lib re cannot handle Unicode properly due to narrow/wide bug
Tom Christiansen added the comment: Ezio Melotti wrote on Mon, 19 Sep 2011 11:11:48 -: > We could also look at what other languages do and/or ask to the > Unicode consortium. I will look at what Java does a bit later on this morning, which is the only other commonly used language besides C that I feel even reasonably competent at. I seem to recall that Java changed its default behavior on certain Unicode decoding issues from warnings to exceptions between one release and the next, but can't remember any details. As the Perl Foundation is a member of the Unicode Consortium and I am on the mailing list, I suppose I could just ask them. I feel a bit timid though because the last thing I brought up there was based on a subtle misunderstanding of mine regarding the IDC and Pattern_Syntax properties. I hate looking dumb twice in row. :) --tom -- ___ Python tracker <http://bugs.python.org/issue12729> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12729] Python lib re cannot handle Unicode properly due to narrow/wide bug
Tom Christiansen added the comment: No good news on the Java front. They do all kinds of things wrong. For example, they allow intermixed CESU-8 and UTF-8 in a real UTF-8 input stream, which is illegal. There's more they do wrong, including in their documentation, but I won't bore you with their errors. I'm going to seek clarification on some matters here. --tom -- title: Python lib re cannot handle Unicode properly due to narrow/wide bug -> Python lib re cannot handle Unicode properly due to narrow/wide bug ___ Python tracker <http://bugs.python.org/issue12729> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12729] Python lib re cannot handle Unicode properly due to narrow/wide bug
Tom Christiansen added the comment: It appears that I'm right about surrogates, but wrong about noncharacters. I'm seeking a clarification there. --tom -- ___ Python tracker <http://bugs.python.org/issue12729> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12737] str.title() is overzealous by upcasing combining marks inappropriately
Tom Christiansen added the comment: > Martin v. Löwis added the comment: > "Split S into words. Change the first letter in a word to upper-case, Except that I think you actually mean that the first "letter" is changed into titlecase not uppercase. One might also say *try* to change for all these, in that not all cased code points in Unicode have casemaps that are different from themselves. For example, a superscript lowercase a or b has no distinct uppercase mapping, the way the non-superscript versions do: % (echo xyz; echo ab AB | unisupers) | uc XYZ ᵃᵇ ᴬᴮ > and all subsequent letters to lower case. A word is a sequence that > starts with a letter, followed by letter-related characters." I don't like the way you have defined letters and letter-related characters. The first already has a definition, which is not the one you are using. Word characters also has a definition in Unicode, and it is not the one you are using. I strongly advise against redefining standard Unicode properties. Choose other, unused terms if you must. It is very confusing otherwise. > Letters are all characters from the "Alphabetic" category, i.e. > Lu+Ll+Lt+Lm+Lo+Nl + Other_Alphabetic. Except that is exactly the definition of the Unicode Alphabetic property, not the Unicode Letter property. It is a mistake to equate Letter=Alphabetic, and very confusing too. I agree that this probably what you want, though. I just don't think you should use "letter-related characters" when there is an existing formal definition that works, or that you should redefine Letter. > "letter-related" characters are letters + marks (Mn, Mc, Me). That isn't quite right. * Letters are Lu+Ll+Lt+Lm+Lo. * Alphabetic is Letters + Other_Alphabetic. * Other_Alphabetic is certain marks (like the iota subscript) and the letter numbers (Nl), as well as a few symbols. * Word characters are Alphabetic + Mn+Mc+Me + Nd + Pc. I think you are looking for here are Word characters without Nd + Pc, so just Alphabetic + Mn+Mc+Me. Is that right? --tom PS: You can do union/intersection stuff with properties to see what the resulting sets look like using the unichars command-line tool. This is everything that is both alphabetic and also a mark: % unichars -gs '\p{Alphabetic}' '\pM' ○ͅ U+0345 GC=Mn SC=InheritedCOMBINING GREEK YPOGEGRAMMENI ○ְ U+05B0 GC=Mn SC=Hebrew HEBREW POINT SHEVA ○ֱ U+05B1 GC=Mn SC=Hebrew HEBREW POINT HATAF SEGOL ○ֲ U+05B2 GC=Mn SC=Hebrew HEBREW POINT HATAF PATAH ○ֳ U+05B3 GC=Mn SC=Hebrew HEBREW POINT HATAF QAMATS ... ○ं U+0902 GC=Mn SC=Devanagari DEVANAGARI SIGN ANUSVARA ः U+0903 GC=Mc SC=Devanagari DEVANAGARI SIGN VISARGA ा U+093E GC=Mc SC=Devanagari DEVANAGARI VOWEL SIGN AA ि U+093F GC=Mc SC=Devanagari DEVANAGARI VOWEL SIGN I ी U+0940 GC=Mc SC=Devanagari DEVANAGARI VOWEL SIGN II ○ु U+0941 GC=Mn SC=Devanagari DEVANAGARI VOWEL SIGN U ○ू U+0942 GC=Mn SC=Devanagari DEVANAGARI VOWEL SIGN UU ○ृ U+0943 GC=Mn SC=Devanagari DEVANAGARI VOWEL SIGN VOCALIC R ○ॄ U+0944 GC=Mn SC=Devanagari DEVANAGARI VOWEL SIGN VOCALIC RR ... While these are the NON-alphabetic marks, which are still Word characters though of course: % unichars -gs '\P{Alphabetic}' '\pM' ○̀ U+0300 GC=Mn SC=InheritedCOMBINING GRAVE ACCENT ○́ U+0301 GC=Mn SC=InheritedCOMBINING ACUTE ACCENT ○̂ U+0302 GC=Mn SC=InheritedCOMBINING CIRCUMFLEX ACCENT ○̃ U+0303 GC=Mn SC=InheritedCOMBINING TILDE ○̄ U+0304 GC=Mn SC=InheritedCOMBINING MACRON ○̅ U+0305 GC=Mn SC=InheritedCOMBINING OVERLINE ○̆ U+0306 GC=Mn SC=InheritedCOMBINING BREVE ○̇ U+0307 GC=Mn SC=InheritedCOMBINING DOT ABOVE ○̈ U+0308 GC=Mn SC=InheritedCOMBINING DIAERESIS ○̉ U+0309 GC=Mn SC=InheritedCOMBINING HOOK ABOVE ○̊ U+030A GC=Mn SC=InheritedCOMBINING RING ABOVE ○̋ U+030B GC=Mn SC=InheritedCOMBINING DOUBLE ACUTE ACCENT ○̌ U+030C GC=Mn SC=InheritedCOMBINING CARON ... And here are the Cased code points that are do not change when upper-, title-, or lowercased: % unichars -gs '\p{Cased}' '[^\p{CWU}\p{CWT}\p{CWL}]' ª U+00AA GC=Ll SC=LatinFEMININE ORDINAL INDICATOR º U+00BA GC=Ll SC=LatinMASCULINE ORDINAL INDICATOR ĸ U+0138 GC=Ll SC=LatinLATIN SMALL LETTER KRA ƍ U+018D GC=Ll SC=LatinLATIN SMALL LETTER TURNED DELTA ƛ U+019B GC=Ll SC=LatinLATIN SMALL LETTER LAMBDA WITH STROKE ƪ U+01AA GC=Ll SC=LatinLATIN LETTER REVERSED ESH LOOP ƫ U+01AB GC=Ll SC=LatinLATIN SMALL LETTER T WITH PALATAL HOOK ƺ U+01BA GC=Ll SC=Latin
[issue12753] \N{...} neglects formal aliases and named sequences from Unicode charnames namespace
Tom Christiansen added the comment: >Ezio Melotti added the comment: > Leaving named sequences for unicodedata.lookup() only (and not for > \N{}) makes sense. There are certainly advantages to that strategy: you don't have to deal with [\N{sequence}] issues. If the argument to unicode.lookup() and be any of name, alias, or sequence, that seems ok. \N{} should still do aliases, though, since those don't have the complication that sequences have. You may wish unicode.name() to return the alias in preference, however. That's what we do. And of course, there is no issue of sequences there. The rest of this perhaps painfully long message is just elaboration and icing on what I've said above. --tom > The list of aliases is so small (11 entries) that I'm not sure using a > binary search for it would bring any advantage. Having a single > lookup algorithm that looks in both tables doesn't work because the > aliases lookup must be in _getcode for \N{...} to work, whereas the > lookup of named sequences will happen in unicodedata_lookup > (Modules/unicodedata.c:1187). I think we can leave the for loop over > aliases in _getcode and implement a separate (and binary) search in > unicodedata_lookup for the named sequences. Does that sound fine? If you mean, is it ok to add just the aliases and not the named sequences to \N{}, it is certainly better than not doing so at all. Plus that way you do *not* have to figure out what in the world to to do with [^a-c\N{sequence}], since that would have be something like (?!\N{sequence})[^a-c]), which is hardly obvious, especially if \N{sequence} actually starts with [a-c]. However, because the one namespace comprises all three of names, aliases, and named sequences, it might be best to have a functional (meaning, non-regex) API that allows one to do a fetch on the whole namespace, or on each individual component. The ICU library supports this sort of thing. In ICU4J's Java bindings, we find this: static int getCharFromExtendedName(String name) [icu] Find a Unicode character by either its name and return its code point value. static int getCharFromName(String name) [icu] Finds a Unicode code point by its most current Unicode name and return its code point value. static int getCharFromName1_0(String name) [icu] Find a Unicode character by its version 1.0 Unicode name and return its code point value. static int getCharFromNameAlias(String name) [icu] Find a Unicode character by its corrected name alias and return its code point value. The first one obviously has a bug in its definition, as the English doesn't scan. Looking at the full definition is even worse. Rather than dig out the src jar, I looked at ICU4C, but its own bindings are completely different. There you have only one function, with an enum to say what namespace to access: UChar32 u_charFromName ( UCharNameChoice nameChoice, const char *name, UErrorCode *pErrorCode ) The UCharNameChoice enum tells what sort of thing you want: U_UNICODE_CHAR_NAME, U_UNICODE_10_CHAR_NAME, U_EXTENDED_CHAR_NAME, U_CHAR_NAME_ALIAS, U_CHAR_NAME_CHOICE_COUNT Looking at the src for the Java is no more immediately illuminating, but I think that "extended" may refer to a union of the old 1.0 names with the current names. Now I'll tell you what Perl does. I do this not to say it is "right", but just to show you one possible strategy. I also am in the middle of writing about this for the Camel, so it is in my head. Perl does not provide the old 1.0 names at all. We don't have a Unicode 1.0 legacy to support, which makes this cleaner. However, we do provide for the names of the C0 and C1 Control Codes, because apart from Unicode 1.0, they don't condescend to name the ASCII or Latin1 control codes. We also provide for certain well known aliases from the Names file: anything that says "* commonly abbreviated as ...", so things like LRO and ZWJ and such. Perl makes no distinction between anything in the namespace when using the \N{} form for string and regex escapes. That means when you use "\N{...}" or /\N{...}/, you don't know which it is, nor can you. (And yes, the bracketed character class issue is annoying and unsolved.) However, the "functional" API does make a slight distinction. -- charnames::vianame() takes a name or alias (as a string) and returns a single integer code point. eg: This therefore converts "LATIN SMALL LETTER A" into 0x61. It also converts both BYZANTINE MUSICAL SYMBOL FHTORA SKLIRON CHROMA VASIS and BYZANTINE MUSICAL SYMBOL FTHORA SKLIRON CHROMA VASIS into 0x1D0C5. See bel
[issue12737] str.title() is overzealous by upcasing combining marks inappropriately
Tom Christiansen added the comment: Martin v. Löwis wrote on Sat, 01 Oct 2011 10:59:48 -: >> * Word characters are Alphabetic + Mn+Mc+Me + Nd + Pc. > Where did you get that definition from? UTS#18 defines > "", which is Alphabetic + U+200C + U+200D > (i.e. not including marks, but including those >From UTS#18 RL1.2A in Annex C, where a \p{word} or \w character is defined to be \p{alpha} \p{gc=Mark} \p{digit} \p{gc=Connector_Punctuation} >> I think you are looking for here are Word characters without >> Nd + Pc, so just Alphabetic + Mn+Mc+Me. >> >> Is that right? > > With your definition of "Word character" above, yes, that's right. It's not mine. It's tr18's. > Marks won't start a word, though. That's the smarter boundary thing they talk about. I'm not myself familiar with \pM > As for terminology: I think the documentation should continue to > speak about "words" and "letters", and then define what is meant > in this context. It's not that the Unicode consortium invented > the term "letter", so we should use it more liberally than just > referring to the L* categories. I really don't think it wise to have private definitions of these. If Letter doesn't mean L?, things get too weird. That's why there are separate definitions of alphabetic, word, etc. --tom -- title: str.title() is overzealous by upcasing combining marks inappropriately -> str.title() is overzealous by upcasing combining marks inappropriately ___ Python tracker <http://bugs.python.org/issue12737> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12753] \N{...} neglects formal aliases and named sequences from Unicode charnames namespace
Tom Christiansen added the comment: >> Perl does not provide the old 1.0 names at all. We don't have a Unicode >> 1.0 legacy to support, which makes this cleaner. However, we do provide >> for the names of the C0 and C1 Control Codes, because apart from Unicode >> 1.0, they don't condescend to name the ASCII or Latin1 control codes. =20 > If there would be a reasonably official source for these names, and one > that guarantees that there is no collision with UCD names, I could > accept doing so for Python as well. The C0 and C1 control code names don't change. There is/was one stability issue where they screwed up, because they ended up having a UAX (required) and a UTS (not required) fighting because of the dumb stuff they did with the Emoji names. They neglected to prefix them with "Emoji ..." or some such, the way things like "GREEK ... LETTER ..." or "MATHEMATICAL ..." or "MUSICAL ..." did. The problem is they stole BELL without calling it EMOJI BELL. This is C0 name for Control-G. Dimwits. The problem with official names is that they have things in them that you are not expected in names. Do you really and truly mean to tell me you think it is somehow **good** that people are forced to write \N{LINE FEED (LF)} Rather than the more obvious pair of \N{LINE FEED} \N{LF} ?? If so, then I don't understand that. Nobody in their right mind prefers "\N{LINE FEED (LF)}" over "\N{LINE FEED}" -- do they? % perl -Mcharnames=:full -le 'printf "U+%04X\n", ord "\N{LINE FEED}"' U+000A % perl -Mcharnames=:full -le 'printf "U+%04X\n", ord "\N{LF}"' U+000A % perl -Mcharnames=:full -le 'printf "U+%04X\n", ord "\N{LINE FEED (LF)}"' U+000A % perl -Mcharnames=:full -le 'printf "U+%04X\n", ord "\N{NEXT LINE}"' U+0085 % perl -Mcharnames=:full -le 'printf "U+%04X\n", ord "\N{NEL}"' U+0085 % perl -Mcharnames=:full -le 'printf "U+%04X\n", ord "\N{NEXT LINE (NEL)}"' U+0085 >> We also provide for certain well known aliases from the Names file: >> anything that says "* commonly abbreviated as ...", so things like LRO >> and ZWJ and such. > -1. Readability counts, writability not so much (I know this is > different for Perl :-). I actually very strongly resent and rebuff that entire mindset in the most extreme way possible. Well-written Perl code is perfectly readable by people who speak that langauge. If you find Perl code that isn't readable, it is by definition not well-written. *PLEASE* don't start. Yes, I just got done driving 16 hours and am overtired, but it's something I've been fighting against all of professional career. It's a "leyenda negra". > If there is too much aliasing, people will > wonder what these codes actually mean. There are 15 "commonly abbreviated as" aliases in the Names.txt file. * commonly abbreviated as NBSP * commonly abbreviated as SHY * commonly abbreviated as CGJ * commonly abbreviated ZWSP * commonly abbreviated ZWNJ * commonly abbreviated ZWJ * commonly abbreviated LRM * commonly abbreviated RLM * commonly abbreviated LRE * commonly abbreviated RLE * commonly abbreviated PDF * commonly abbreviated LRO * commonly abbreviated RLO * commonly abbreviated NNBSP * commonly abbreviated WJ All of the standards documents *talk* about things like LRO and ZWNJ. I guess the standards aren't "readable" then, right? :) >From the charnames manpage, which shows that we really don't just make these up as we feel like (although we could; see below). They're all from this or that standard: ALIASES A few aliases have been defined for convenience: instead of having to use the official names LINE FEED (LF) FORM FEED (FF) CARRIAGE RETURN (CR) NEXT LINE (NEL) (yes, with parentheses), one can use LINE FEED FORM FEED CARRIAGE RETURN NEXT LINE LF FF CR NEL All the other standard abbreviations for the controls, such as "ACK" for "ACKNOWLEDGE" also can be used. One can also use BYTE ORDER MARK BOM and these abbreviations AbbreviationFull Name CGJ COMBINING GRAPHEME JOINER FVS1MONGOLIAN FREE VARIATION SELECTOR ONE FVS2MONGOLIAN FREE VARIATION SELECTOR TWO FVS3MONGOLIAN FREE VARIATION SELECTOR THREE
[issue12753] \N{...} neglects formal aliases and named sequences from Unicode charnames namespace
Tom Christiansen added the comment: Ezio Melotti wrote on Sun, 02 Oct 2011 06:46:26 -: > Actually Python doesn't seem to support \N{LINE FEED (LF)}, most likely bec= > ause that's a Unicode 1 name, and nowadays these codepoints are simply mark= > ed as ''. Yes, but there are a lot of them, 65 of them in fact. I do not care to see people being forced to use literal control characters or inscrutable magic numbers. It really bothers me that you have all these defined code points with properties and all that have no name. People do use these. Some of them a lot. I don't mind \n and such -- and in fact, prefer them even -- but I feel I should not have scratch my head over character \033, \0177, and brethren. The C0 and C1 standards are not just inventions, so we use them. Far better than one should write \N{ESCAPE} for \033 or \N{DELETE} for \0177, don't you think? >> If so, then I don't understand that. Nobody in their right=20 >> mind prefers "\N{LINE FEED (LF)}" over "\N{LINE FEED}" -- do they? > They probably don't, but they just write \n anyway. I don't think we need = > to support any of these aliases, especially if they are not defined in the = > Unicode standard. If you look at Names.txt, there are significant "aliases" there for the C0/C1 stuff. My bottom line is that I don't like to be forced to use magic numbers. I prefer to name my abstactions. It is more readable and more maintainble that way. There are still "holes" of course. Code point 128 has no name even in C1. But something is better than nothing. Plus at least in Perl we *can* give things names if we want, per the APPLE LOGO example for U+F8FF. So nothing needs to remain nameless. Why, you can even name your Kanji if you want, using whatever Romanization you prefer. I think the private-use case example is really motivating, but I have no idea how to do this for Python because there is no lexical scope. I suppose you could attach it to the module, but that still doesn't really work because of how things get evaluated. With a Perl compile-time use, we can change the compiler's ideas about things, like adding function prototypes and even extending the base types: % perl -Mbigrat -le 'print 1/2 + 2/3 * 4/5' 31/30 % perl -Mbignum -le 'print 21->is_odd' 1 % perl -Mbignum -le 'print 18->is_odd' 0 % perl -Mbignum -le 'print substr(2**5000, -3)' 376 % perl -Mbignum -le 'print substr(2**5000-1, -3)' 375 % perl -Mbignum -le 'print length(2**5000)' 1506 % perl -Mbignum -le 'print length(10**5000)' 5001 % perl -Mbignum -le 'print ref 10**5000' Math::BigInt % perl -Mbigrat -le 'print ref 1/3' Math::BigRat I recognize that redefining what sort of object the compiler treats some of its constants as is never going to happen in Python, but we actually did manage that with charnames without having to subclass our strings: the hook for \N{...} doesn't require object games like the ones above. But it still has to happen at compile time, of course, so I don't know what you could do in Python. Is there any way to change how the compiler behaves even vaguely along these lines? The run-time looks of Python's unicodedata.lookup (like Perl's charnames::viacode) and unicodedata.name (like Perl's charnames::viacode on the ord) could be managed with a hook, but the compile-time lookups of \N{...} I don't see any way around. But I don't know anything about Python's internals, so don't even know what is or is not possible. I do note that if you could extend \N{...} the way we do with charname aliases for private-use characters, the user could load something that did the C0 and C1 control if they wanted to. I just don't know how to do that early enough that the Python compiler would see it. Your import happens at run-time or at compile-time? This would be some sort of compile-time binding of constants. d=20 >> Python doesn't require it. :)/2 > I actually find those *less* readable. If there's something fancy in the r= > egex, a comment *before* it is welcomed, but having to read a regex divided= > on several lines and remove meaningless whitespace and redundant comments = > just makes the parsing more difficult for me. Really? White space makes things harder to read? I thought Pythonistas believed the opposite of that. Whitespace is very useful for cognitive chunking: you see how things logically group together. Inomorewantaregexwithoutwhitespacethananyothercodeortext. :) I do grant you that chatty comments may be a separate matter. White space in patterns is also good when you have successive patterns across multiple lines that
[issue12753] \N{...} neglects formal aliases and named sequences from Unicode charnames namespace
Tom Christiansen added the comment: >> Really? White space makes things harder to read? I thought Pythonistas >> believed the opposite of that. > I was surprised at that too ;-). One person's opinion in a specific > context. Don't generalize. The example I initially showed probably wasn't the best for that. Mostly I was trying to demonstrate how useful it is to have user-defined properties is all. But I have no asked for that (I have asked for properties, though). >> English titling rules >> only capitalize the first word in hyphenated words, which is why it's >> Anti‐intellectual not Anti-Intellectual. > Except that I can imagine someone using the latter as a noun to make the > work more officious or something. If Good-Looking looks more officous than Good-looking, I bet GOOD-LOOKING is better still. :) > There are no official English titling rules and as you noted, > publishers vary. If there aren't any rules, then how come all book and movie titles always look the same? :) I don't think anyone would argue with these two: 1. Capitalize the first word, the last word, and the word right after a colon (or semicolon). 2. Capitalize all intervening words except for articles (a, an, the) and short prepositions. Those are the basic rules. The main problem is that "short" isn't well defined--and indeed, there are even places where "preposition" isn't well defined either. English has sentence casing (only the first word) and headline casing (most of them). It's problematic that computer people call capitalizing each word titlecasing, since in English, this is never correct. http://www.chicagomanualofstyle.org/CMS_FAQ/CapitalizationTitles/CapitalizationTitles23.html Although Chicago style lowercases prepositions (but see CMOS 8.157 for exceptions), some style guides uppercase them. Ask your editor for a style guide. I myself usually fall back to the Chicago Manual of Style or the Oxford Guide to Style. I don't think I do anything neither of them says to do. But I completely agree that this should *not* be in the titlecase() function. I think the docs for the function might perhaps say something about how it does not mean correct English headline case when it says titlecase, but that's largely just nitpicking. > I agree that str.title should do something sensible > based on Unicode, with the improvements you mentioned. One of the goals of Unicode is that casing not be language dependent. And they almost got there, too. The Turkic I is the most notable exception. Did you know there is a problem with all the case stuff in Python? It was clearly put in before they had realized that they needed to have things other the Lu/Lt/Ll have casing properties. That's why there is a difference betwen GC=Ll and the Lowercase property. str.islower() Return true if all cased characters in the string are lowercase and there is at least one cased character, false otherwise. Cased characters are those with general category property being one of “Lu”, “Ll”, or “Lt” and lowercase characters are those with general category property “Ll”. http://docs.python.org/release/3.2/library/stdtypes.html That really isn't right. A cased character is one with the Unicode "Cased" property, and a lowercase character is one wiht the Unicode "Lowercase" property. The General Category is actually immaterial here. I've spent all bloody day trying to model Python's islower, isupper, and istitle functions, but I get all kinds of errors, both in the definitions and in the models of the definitions.Under both 2.7 and 3.2, I get all these bugs: ᶜ not islower() but has at least one cased character with all cased characters lowercase! ᴰ not islower() but has at least one cased character with all cased characters lowercase! ⓚ not islower() but has at least one cased character with all cased characters lowercase! ͅ not islower() but has at least one cased character with all cased characters lowercase! Ⅷ not isupper() but has at least one cased character with all cased characters uppercase! Ⅷ not istitle() but should be ⅷ not islower() but has at least one cased character with all cased characters lowercase! 2ⁿᵈ not islower() but has at least one cased character with all cased characters lowercase! 2ᴺᴰ not islower() but has at least one cased character with all cased characters lowercase! Ὰͅ isupper() but fails to have at least one cased character with all cased characters uppercase! ThisIsInTitleCaseYouKnow not istitle() but should be Mᶜ isupper() but fails to have at least one cased character with all cased characters uppercase! ᶜM isupper() but fails to have at least one cased character with all cased characters uppercase!
[issue12753] \N{...} neglects formal aliases and named sequences from Unicode charnames namespace
Tom Christiansen added the comment: Ezio Melotti wrote on Mon, 03 Oct 2011 04:15:51 -: >> But it still has to happen at compile time, of course, so I don't know >> what you could do in Python. Is there any way to change how the compiler >> behaves even vaguely along these lines? > I think things like "from __future__ import ..." do something similar, > but I'm not sure it will work in this case (also because you will have > to provide the list of aliases somehow). Ah yes, that's right. Hm. I bet then it *would* be possible, just perhaps a bit of a run-around to get there. Not a high priority, but interesting. > less readable than: > > def my_capitalize(s): >return s[0].upper() + s[1:].lower() > You could argue that the first is much more explicit and in a way > clearer, but overall I think you agree with me that is less readable. Certainly. It's a bit like the way bug rate per lines of code is invariant across programming languages. When you have more opcodes, it gets harder to understand because there are more interactions and things to remember. >> That really isn't right. A cased character is one with the Unicode "Cased" >> property, and a lowercase character is one wiht the Unicode "Lowercase" >> property. The General Category is actually immaterial here. > You might want to take a look and possibly add a comment on #12204 about this. >> I've spent all bloody day trying to model Python's islower, isupper, and >> istitle >> functions, but I get all kinds of errors, both in the definitions and in the >> models of the definitions. > If by "model" you mean "trying to figure out how they work", it's > probably easier to look at the implementation (I assume you know > enough C to understand what they do). You can find the code for > str.istitle() at http://hg.python.org/cpython/file/default/Objects/un- > icodeobject.c#l10358 and the actual implementation of some macros like > Py_UNICODE_ISTITLE at > http://hg.python.org/cpython/file/default/Objects/unicodectype.c. Thanks, that helps immensely. I'm completely fluent in C. I've gone and built a tags file of your whole v3.2 source tree to help me navigate. The main underlying problem is that the internal macros are defined in a way that made sense a long time ago, but no longer do ever since (for example) the Unicode lowercase property stopped being synonymous with GC=Ll and started also including all code points with the Other_Lowercase property as well. The originating culprit is Tools/unicode/makeunicodedata.py. It builds your tables only using UnicodeData.txt, which is not enough. For example: if category in ["Lm", "Lt", "Lu", "Ll", "Lo"]: flags |= ALPHA_MASK if category == "Ll": flags |= LOWER_MASK if 'Line_Break' in properties or bidirectional == "B": flags |= LINEBREAK_MASK linebreaks.append(char) if category == "Zs" or bidirectional in ("WS", "B", "S"): flags |= SPACE_MASK spaces.append(char) if category == "Lt": flags |= TITLE_MASK if category == "Lu": flags |= UPPER_MASK It needs to use DerivedCoreProperties.txt to figure out whether something is Other_Uppercase, Other_Lowercase, etc. In particular: Alphabetic := Lu+Ll+Lt+Lm+Lo + Nl + Other_Alphabetic Lowercase := Ll + Other_Lowercase Uppercase := Ll + Other_Uppercase This affects a lot of things, but you should be able to just fix it in Tools/unicode/makeunicodedata.py and have all of them start working correctly. You will probably also want to add Py_UCS4 _PyUnicode_IsWord(Py_UCS4 ch) that uses the UTS#18 Annex C definition, so that you catch marks, too. That definition is: Word := Alphabetic + Mc+Me+Mn + Nd + Pc where Alphabetic is defined above to include Nl and Other_Alphabetic. Soemwhat related is stuff like this: typedef struct { const Py_UCS4 upper; const Py_UCS4 lower; const Py_UCS4 title; const unsigned char decimal; const unsigned char digit; const unsigned short flags; } _PyUnicode_TypeRecord; There are two different bugs here. First, you are missing const Py_UCS4 fold; which is another field from UnicodeData.txt, one that is critical for doing case-insensitive matches correctly. Second, there's also the problem that Py_UCS4 is an int. That means you are stuck with just the character-based simple versions of upper-, title-, lower-, and foldcase. You need to have fields for the full mappings, which are now strings (well, int arrays) not single ints. I'll use ??? for the int-array type that I don
[issue12753] \N{...} neglects formal aliases and named sequences from Unicode charnames namespace
Tom Christiansen added the comment: Ezio Melotti wrote on Sun, 09 Oct 2011 13:21:00 -: > Here is a new patch that stores the names of aliases and named > sequences in the Private Use Area. Looks good! Thanks! --tom -- title: \N{...} neglects formal aliases and named sequences from Unicode charnames namespace -> \N{...} neglects formal aliases and named sequences from Unicode charnames namespace ___ Python tracker <http://bugs.python.org/issue12753> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12568] Add functions to get the width in columns of a character
Tom Christiansen added the comment: > Martin v. Löwis added the comment: > I think the WideCharToMultibyte approach is just incorrect. > I'm -1 on using wcswidth, though. Like you, I too seriously question using wcswidth() for this at all: The wcswidth() function either shall return 0 (if pwcs points to a null wide-character code), or return the number of column positions to be occupied by the wide-character string pointed to by pwcs, or return -1 (if any of the first n wide-character codes in the wide- character string pointed to by pwcs is not a printable wide- character code). I would be willing to bet (a small amount of) money it does not correctly inplmented Unicode print widths, even though one would certainly *think* it does according to this: The wcswidth() function determines the number of column positions required for the first n characters of pwcs, or until a null wide character (L'\0') is encountered. There are a bunch of "interesting" cases I would want it tested against. > We already have unicodedata.east_asian_width, which implements > http://unicode.org/reports/tr11/ > The outcomes of this function are these: > - F: full-width, width 2, compatibility character for a narrow char > - H: half-width, width 1, compatibility character for a narrow char > - W: wide, width 2 > - Na: narrow, width 1 > - A: ambiguous; width 2 in Asian context, width 1 in non-Asian context > - N: neutral; not used in Asian text, so has no width. Practically, width can > be considered as 1 Um, East_Asian_Width=Ambiguous (EA=A) isn't actually good enough for this. And EA=N cannot be consider 1, either. For example, some of the Marks are EA=A and some are EA=N, yet how may print columns they take varies. It is usually 0, but can be 1 at the start of the file/string or immediately after a linebreak sequence. Then there are things like the variation selectors which are never anything. Now consider the many \pC code points, like U+0009 CHARACTER TABULATION U+00AD SOFT HYPHEN U+200C ZERO WIDTH NON-JOINER U+FEFF ZERO WIDTH NO-BREAK SPACE U+2062 INVISIBLE TIMES A TAB is its own problem but SHY we know is only width=1 immediately before a linebreak or EOF, and ZWNJ and ZWNBSP are both certainly width=0. So are the INVISIBLE * code points. Context: Imagine you're trying to format a string so that it takes up exactly 20 columns: you need to know how many spaces to pad it with based on the print width. That is what the #12568 is needing to do, and you have to do much more than East Asian Width properties. I really do think that what #12568 is asking for is to have the equivalent of the Perl Unicode::GCString's columns() method, and that you aren't going to be able to handle text alignment of Unicode with anything that is much less of that. After all, #12568's title is "Add functions to get the width in columns of a character". I would very much like to compare what columns() thinks compared with what wcswidth() thinks. I bet wcswidth() is very simple-minded at best. I may of course be wrong. --tom -- ___ Python tracker <http://bugs.python.org/issue12568> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12753] \N{...} neglects formal aliases and named sequences from Unicode charnames namespace
Tom Christiansen added the comment: Yes, it looks good. Thank you very much. -tom -- ___ Python tracker <http://bugs.python.org/issue12753> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com