On 2013-10-08, Steven D'Aprano <steve+comp.lang.pyt...@pearwood.info> wrote:
> py> c = '\N{LINEAR B SYLLABLE B038 E}'
> py> surr_pair = c.encode('utf-16be')
> py> print(surr_pair)
> b'\xd8\x00\xdc\x01'
>
> and then use those same values as the code points, I ought to be able to
> encode to UTF-8, as if it were the same \N{LINEAR B SYLLABLE B038 E} code
> point. But I can't:
>
> py> s = '\ud800\udc01'
> py> s.encode('utf-8')
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in
> position 0: surrogates not allowed
>
> Have I misunderstood? I think that Python is being too strict
> about rejecting surrogate code points. It should only reject
> lone surrogates, or invalid pairs, not valid pairs. Have I
> misunderstood the Unicode FAQs, or is this a bug in Python's
> handling of UTF-8?
>From RFC 3629:
The definition of UTF-8 prohibits encoding character numbers
between U+D800 and U+DFFF, which are reserved for use with the
UTF-16 encoding form (as surrogate pairs) and do not directly
represent characters. When encoding in UTF-8 from UTF-16 data,
it is necessary to first decode the UTF-16 data to obtain
character numbers, which are then encoded in UTF-8 as described
above. This contrasts with CESU-8 [CESU-8], which is a
UTF-8-like encoding that is not meant for use on the Internet.
CESU-8 operates similarly to UTF-8 but encodes the UTF-16 code
values (16-bit quantities) instead of the character number
(code point). This leads to different results for character
numbers above 0xFFFF; the CESU-8 encoding of those characters
is NOT valid UTF-8.
The Wikipedia article points out:
Whether an actual application should [refuse to encode these
character numbers] is debatable, as it makes it impossible to
store invalid UTF-16 (that is, UTF-16 with unpaired surrogate
halves) in a UTF-8 string. This is necessary to store unchecked
UTF-16 such as Windows filenames as UTF-8. It is also
incompatible with CESU encoding (described below).
So Python's interpretation is conformant, though not without some
disadvantages.
In any case, "\ud800\udc01" isn't a valid unicode string. In a
perfect world it would automatically get converted to
'\u00010001' without intervention.
--
Neil Cerutti
--
https://mail.python.org/mailman/listinfo/python-list
