On Friday, 7 March 2014 at 02:37:11 UTC, Walter Bright wrote:
In "Lots of low hanging fruit in Phobos" the issue came up
about the automatic encoding and decoding of char ranges.
Throughout D's history, there are regular and repeated
proposals to redesign D's view of char[] to pretend it is not
UTF-8, but UTF-32. I.e. so D will automatically generate code
to decode and encode on every attempt to index char[].
I'm glad I'm not the only one who feels this way. Implicit
decoding must die.
I strongly believe that implicit decoding of character points in
std.range has been a mistake.
- Algorithms such as "countUntil" will count code points. These
numbers are useless for slicing, and can introduce hard-to-find
bugs.
- In lots of places, I've discovered that Phobos did UTF decoding
(thus murdering performance) when it didn't need to. Such cases
included format (now fixed), appender (now fixed), startsWith
(now fixed - recently), skipOver (still unfixed). These have
caused latent bugs in my programs that happened to be fed non-UTF
data. There's no reason for why D should fail on non-UTF data if
it has no reason to decode it in the first place! These failures
have only served to identify places in Phobos where redundant
decoding was occurring.
Furthermore, it doesn't actually solve anything completely! The
only thing it solves is a subset of cases for a subset of
languages!
People want to look at a string "character by character". If a
Unicode code point is a character in your language and alphabet,
I'm really happy for you, but that's not how it is for everyone.
Combining marks, complex scripts etc. make this point just a
fallacy that in the end will cause programmers to make mistakes
that will affect certain users somewhere.
Why do people want to look at individual characters? There are a
lot of misconceptions about Unicode, and I think some of that
applies here.
- Do you want to split a string by whitespace? Some languages
have no notion of whitespace. What do you need it for? Line
wrapping? Employ the Unicode line-breaking algorithm instead.
- Do you want to uppercase the first letter of a string? Some
language have no notion of letter case, and some use it for
different reasons. Furthermore, even languages with a Latin-based
alphabet may not have 1:1 mapping for case, e.g. the German ß
letter.
- Do you want to count how wide a string will be in a fixed-point
font? Wrong... Combining and control characters, zero-width
whitespace, etc. will render this approach futile.
- Do you want to split or flush a stream to a character device at
a point so that there's no garbage? I believe, this is the case
in TDPL's mention of the subject. Again, combining characters or
complex scripts will still be broken by this approach.
You need to either go all-out and provide complete
implementations of the relevant Unicode algorithms to perform
tasks such as the above that will work in all locales, or you
need to draw a line somewhere for which languages, alphabets,
locales do you want to support in your program. D's line is drawn
at the point where it considers that code points == characters,
however the outcome of this is clear nowhere in its documentation
and for such an arbitrary decision (from a cultural point of
view), it is embedded too deep into the language itself. With
std.ascii, at least, it's clear to the user that the functions
there will only work with English or languages using the same
alphabet.
This doesn't apply universally. There are still cases like, e.g.,
regular expression ranges. [a-z] makes sense in English, and
[а-я] makes sense in Russian, but I don't think that makes sense
for all languages. However, for the most part, I think implicit
decoding must be axed, and instead we need implementations of
Unicode algorithms and the documentation to instruct users why
and how to use them.
