please use the "Reply" Button
On 08.06.2010 08:50, Ruslan Nikolaev wrote:
Maybe "lousy" is too strong a word, but aside from
compatibility with other
libs/software that use it (which I'll address separately),
UTF-16 is not
particularly useful compared to UTF-8 and UTF-32:
...
I tried to avoid commenting this because I am afraid we'll stray away from the main point
(which is not discussion about which Unicode is better). But in short I would say:
"Not quite right". UTF-16 as already mentioned is generally faster for
non-Latin letters (as reading 2 bytes of aligned data takes the same time as reading 1
byte). Although, I am not familiar with Asian languages, I believe that UTF-16 requires
just 2 bytes instead of 3 for most of symbols. That is one of the reason they don't like
UTF-8. UTF-32 doesn't have any advantage except for being fixed length. It has a lot of
unnecessary memory, cache, etc. overhead (the worst case scenario for both UTF8/16) which
is not justified for any language.
First of all, it's not exactly unheard of for big projects
to make a
sub-optimal decision.
I would say, the decision was quite optimal for many reasons, including that "lousy
programming" will not cause too many problems as in case of UTF-8.
Secondly, Java and Windows adapted 16-bit encodings back
when many people
were still under the mistaken impression that would allow
them to hold any
character in one code-unit. If that had been true, then it
I doubt that it was the only reason. UTF-8 was already available before Windows
NT was released. It would be much easier to use UTF-8 instead of ANSI as
opposed to creating parallel API. Nonetheless, UTF-16 has been chosen. In
addition, C# has been released already when UTF-16 became variable length. I
doubt that conversion overhead (which is small compared to VM) was the main
reason to preserve UTF-16.
Concerning why I say that it's good to have conversion to UTF-32 (you asked
somewhere):
I think you did not understand correctly what I meant. This a very common
practice, and in fact - required, to convert from both UTF-8 and UTF-16 to
UTF-32 when you need to do character analysis (e.g. mbtowc() in C). In fact, it
is the only place where UTF-32 is commonly used and useful.
