On Sat, Apr 06, 2002 at 01:08:11AM +0900, Dan Kogai wrote:
> On Saturday, April 6, 2002, at 12:18 , Jarkko Hietaniemi wrote:
> >> P.S. Does utf8 support surrogates? Surrogate pair is definitely the
> >
> > No. Surrogates are solely for UTF-16. There's no need for surrogates
> > in UTF-8 -- if we wanted to encode U+D800 using UTF-8, we *could* --
> > BUT we should not. Encoding U+D800 as UTF-8 should not be attempted,
> > the whole surrogate space is a discontinuity in the Unicode code point
> > space reserved for the evils of UTF-16.
>
> Yes. I know that. My question is whether we support CONVERSION.
> Internals have nothing to do with that. When we say UCS-2,
> \x{10000}-\x{10ffff} must be discarded or croak for error. When we say
I suggest croak.
> UTF-16, however, We have to convert them into surrogate pairs when we
> convert and decode back to \x{10000}-\x{10ffff} when we decode.
Well, there seems to be
Perl_utf16_to_utf8(pTHX_ U8* p, U8* d, I32 bytelen, I32 *newlen)
in utf8.c that seems to be doing surrogate arithmetics, but I think
that's not much used (if at all), and I cannot see utf8_to_utf16.
(There's also
Perl_utf16_to_utf8_reversed(pTHX_ U8* p, U8* d, I32 bytelen, I32 *newlen)
which does first a byteswap and then calls the non-reversed version).
I also can see that the Perl_utf16_to_utf8 is non-EBCDIC aware...
> FYI I have already cleaned up UCS-2 part. Now their canonical names are
> UCS-2BE and UCS-2LE (modules are renamed as well to be more cannonical,
> ucs_2(be|le).pm. Yes, underscore first). UTF-32 is trivial because we
> only have to pack the ord value to 32-bit. It's UTF-16 in question.
>
> If we want perl to be surrogates-free, then ironically we have to
> support UTF-16 because ucs_2*.pm simply let \x{D800}-\x{DFFF} in so far.
>
> Dan the Man with Too Many UnicodeS to tackle
--
$jhi++; # http://www.iki.fi/jhi/
# There is this special biologist word we use for 'stable'.
# It is 'dead'. -- Jack Cohen
