It's no secret that string et al. are not a magic recipe for writing
correct Unicode code. However, things are pretty good and could be
further improved by operating the following changes in std.array and
std.range:
- make front() and back() for UTF-8 and UTF-16 automatically decode the
first and last Unicode character
- make popFront() and popBack() skip one entire Unicode character
(instead of just one code unit)
- alter isRandomAccessRange to return false for UTF-8 and UTF-16 strings
- change hasLength to return false for UTF-8 and UTF-16 strings
These changes effectively make UTF-8 and UTF-16 bidirectional ranges,
with the quirk that you still have a sort of a random-access operator.
I'm very strongly in favor of this change. Bidirectional strings allow
beautiful correct algorithms to be written that handle encoded strings
without any additional effort; with these changes, everything applicable
of std.algorithm works out of the box (with the appropriate fixes here
and there), which is really remarkable.
The remaining WTF is the length property. Traditionally, a range
offering length also implies the expectation that a range of length n
allows you to call popFront n times and then assert that the range is
empty. However, if you check e.g. hasLength!string it will yield false,
although the string does have an accessible member by that name and of
the appropriate type.
Although Phobos always checks its assumptions, people might occasionally
write code that just uses .length without checking hasLength. Then,
they'll be annoyed when the code fails with UTF-8 and UTF-16 strings.
(The "real" length of the range is not stored, but can be computed by
using str.walkLength() in std.range.)
What can be done about that? I see a number of solutions:
(a) Do not operate the change at all.
(b) Operate the change and mention that in range algorithms you should
check hasLength and only then use "length" under the assumption that it
really means "elements count".
(c) Deprecate the name .length for UTF-8 and UTF-16 strings, and define
a different name for that. Any other name (codeUnits, codes etc.) would
do. The entire point is to not make algorithms believe strings have a
.length property.
(d) Have std.range define a distinct property called e.g. "count" and
then specialize it appropriately. Then change all references to .length
in std.algorithm and elsewhere to .count.
What would you do? Any ideas are welcome.
Andrei
