Eryk Sun <eryk...@gmail.com> added the comment:

> From Eryk's description it sounds like we should always add 
> WC_NO_BEST_FIT_CHARS as an option to MultiByteToWideChar() 
> in order to make sure it doesn't use best fit variants 
> unless explicitly requested.

The concept of a "best fit" encoding is unrelated to decoding with 
MultiByteToWideChar(). By default WideCharToMultiByte() best-fit encodes some 
otherwise unmapped ordinals to characters in the code page that have similar 
glyphs. This doesn't round trip (e.g. "α" -> b"a" -> "a"). The 
WC_NO_BEST_FIT_CHARS flag prevents this behavior. code_page_encode() uses 
WC_NO_BEST_FIT_CHARS for legacy encodings, unless the "replace" error handler 
is used.

Windows maps every value in single-byte ANSI code pages to a Unicode ordinal, 
which round trips between MultiByteToWideChar() and WideCharToMultiByte(). 
Unless otherwise defined, a value in the range 0x80-0x9F is mapped to the 
corresponding ordinal in the C1 controls block. Otherwise values that have no 
legacy definition are mapped to a private use area (e.g. U+E000 - U+F8FF). 

There is no option to make MultiByteToWideChar() fail for byte values that map 
to a C1 control code. But mappings to the private use area are strictly 
invalid, and MultiByteToWideChar() will fail in these cases if the flag 
MB_ERR_INVALID_CHARS is used. code_page_decode() always uses this flag, but to 
reliably fail one needs to pass final=True, since the codec doesn't know it's a 
single-byte encoding. For example:

    >>> codecs.code_page_decode(1253, b'\xaa', 'strict')
    ('', 0)

    >>> codecs.code_page_decode(1253, b'\xaa', 'strict', True)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    UnicodeDecodeError: 'cp1253' codec can't decode bytes in position 0--1: 
    No mapping for the Unicode character exists in the target code page.

Here are the mappings to the private use area in the single-byte "ANSI" code 
pages:

    1255 Hebrew
    0xD9    U+F88D
    0xDA    U+F88E
    0xDB    U+F88F
    0xDC    U+F890
    0xDD    U+F891
    0xDE    U+F892
    0xDF    U+F893
    0xFB    U+F894
    0xFC    U+F895
    0xFF    U+F896

Note that 0xCA is defined as the Hebrew character U+05BA [1]. The definition is 
missing in the unicode.org data and Python's "cp1255" encoding.

    874 Thai
    0xDB    U+F8C1
    0xDC    U+F8C2
    0xDD    U+F8C3
    0xDE    U+F8C4
    0xFC    U+F8C5
    0xFD    U+F8C6
    0xFE    U+F8C7
    0xFF    U+F8C8

    1253 Greek
    0xAA    U+F8F9
    0xD2    U+F8FA
    0xFF    U+F8FB

    1257 Baltic
    0xA1    U+F8FC
    0xA5    U+F8FD

There's no way to get these private use area results from code_page_decode(), 
but code_page_encode() allows them. For example:

    >>> codecs.code_page_encode(1253, '\uf8f9')[0]
    b'\xaa'

---

[1] https://en.wikipedia.org/wiki/Windows-1255

----------

_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue45120>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Reply via email to