Eryk Sun <eryk...@gmail.com> added the comment: > From Eryk's description it sounds like we should always add > WC_NO_BEST_FIT_CHARS as an option to MultiByteToWideChar() > in order to make sure it doesn't use best fit variants > unless explicitly requested.
The concept of a "best fit" encoding is unrelated to decoding with MultiByteToWideChar(). By default WideCharToMultiByte() best-fit encodes some otherwise unmapped ordinals to characters in the code page that have similar glyphs. This doesn't round trip (e.g. "α" -> b"a" -> "a"). The WC_NO_BEST_FIT_CHARS flag prevents this behavior. code_page_encode() uses WC_NO_BEST_FIT_CHARS for legacy encodings, unless the "replace" error handler is used. Windows maps every value in single-byte ANSI code pages to a Unicode ordinal, which round trips between MultiByteToWideChar() and WideCharToMultiByte(). Unless otherwise defined, a value in the range 0x80-0x9F is mapped to the corresponding ordinal in the C1 controls block. Otherwise values that have no legacy definition are mapped to a private use area (e.g. U+E000 - U+F8FF). There is no option to make MultiByteToWideChar() fail for byte values that map to a C1 control code. But mappings to the private use area are strictly invalid, and MultiByteToWideChar() will fail in these cases if the flag MB_ERR_INVALID_CHARS is used. code_page_decode() always uses this flag, but to reliably fail one needs to pass final=True, since the codec doesn't know it's a single-byte encoding. For example: >>> codecs.code_page_decode(1253, b'\xaa', 'strict') ('', 0) >>> codecs.code_page_decode(1253, b'\xaa', 'strict', True) Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeDecodeError: 'cp1253' codec can't decode bytes in position 0--1: No mapping for the Unicode character exists in the target code page. Here are the mappings to the private use area in the single-byte "ANSI" code pages: 1255 Hebrew 0xD9 U+F88D 0xDA U+F88E 0xDB U+F88F 0xDC U+F890 0xDD U+F891 0xDE U+F892 0xDF U+F893 0xFB U+F894 0xFC U+F895 0xFF U+F896 Note that 0xCA is defined as the Hebrew character U+05BA [1]. The definition is missing in the unicode.org data and Python's "cp1255" encoding. 874 Thai 0xDB U+F8C1 0xDC U+F8C2 0xDD U+F8C3 0xDE U+F8C4 0xFC U+F8C5 0xFD U+F8C6 0xFE U+F8C7 0xFF U+F8C8 1253 Greek 0xAA U+F8F9 0xD2 U+F8FA 0xFF U+F8FB 1257 Baltic 0xA1 U+F8FC 0xA5 U+F8FD There's no way to get these private use area results from code_page_decode(), but code_page_encode() allows them. For example: >>> codecs.code_page_encode(1253, '\uf8f9')[0] b'\xaa' --- [1] https://en.wikipedia.org/wiki/Windows-1255 ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue45120> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com