New submission from Greg Price <gnpr...@gmail.com>: Splitting this out from #32771 for more specific discussion. Benjamin writes there that it would be good to:
> implement the locale-specific case mappings of > https://www.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt and §3.13 of > the Unicode 12 standard in str.lower/upper/casefold. and adds that an implementation would require having available in the core the data on canonical combining classes, which is currently only in the unicodedata module. --- First, I'd like to better understand what functionality we have now and what else the standard describes. Reading https://www.unicode.org/Public/12.0.0/ucd/SpecialCasing.txt , I see * a bunch of rules that aren't language-specific * some other rules that are. I also see in makeunicodedata.py that we don't even parse the language-specific rules. Here's, IIUC, a demo of us correctly implementing the language-independent rules. One line in the data file reads: FB00; FB00; 0046 0066; 0046 0046; # LATIN SMALL LIGATURE FF And in fact the `lower`, `title`, and `upper` of `\uFB00` are those strings respectively: $ unicode --brief "$(./python -c \ 's="\ufb00"; print(" ".join((s.lower(), s.title(), s.upper())))')" ff U+FB00 LATIN SMALL LIGATURE FF U+0020 SPACE F U+0046 LATIN CAPITAL LETTER F f U+0066 LATIN SMALL LETTER F U+0020 SPACE F U+0046 LATIN CAPITAL LETTER F F U+0046 LATIN CAPITAL LETTER F OK, great. --- Then here's something we don't implement. Another line in the file reads: 00CD; 0069 0307 0301; 00CD; 00CD; lt; # LATIN CAPITAL LETTER I WITH ACUTE IOW `'\u00CD'` should lowercase to `'\u0069\u0307\u0301'`, i.e.: i U+0069 LATIN SMALL LETTER I ̇ U+0307 COMBINING DOT ABOVE ́ U+0301 COMBINING ACUTE ACCENT ... but only in a Lithuanian (`lt`) locale. One question is: what would the right API for this be? I'm not sure I'd want `str.lower`'s results to depend on the process's current Unix locale... and I definitely wouldn't want to get that without some way of instead telling it what locale to use. (Either to use a single constant locale, or to use a per-user locale in e.g. a web application.) Perhaps `str.lower` and friends would take a keyword argument `locale`? Oh, one more link for reference: the said section of the standard is in this PDF: https://www.unicode.org/versions/Unicode12.0.0/ch03.pdf , near the end. And a related previous issue: #12736. ---------- components: Unicode messages: 349646 nosy: Greg Price, benjamin.peterson, ezio.melotti, lemburg, vstinner priority: normal severity: normal status: open title: More fully implement Unicode's case mappings versions: Python 3.9 _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue37848> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com