On Tuesday, 28 April 2015 at 09:11:10 UTC, Chris wrote:
Would it be much work to show have example code or even an
experimental module that gets rid of auto-decoding, so we could
see what would be affected in general and how actual code we
have would be affected by it?
The topic keeps coming up again and again, and while I'm in
favor of anything that enhances performance, I'm afraid of
having to refactor large chunks of my code. However, this fear
may be unfounded, but I would need some examples to visualize
the problem.
Honestly, most code won't care. If we just switched out all of
the auto-decoding right now, pretty much anything using only
ASCII would just work, and most anything that's trying to
manipulate ASCII characters in a Unicode string will just work,
whereas code that's specifically manipulating Unicode characters
might have problems (e.g. comparing front with a dchar will no
longer have the same result, since front would just be the first
code unit rather than necessarily the first code point). Since
most Phobos range-based functions which operate on strings are
special-cased on strings already, many of them would continue to
just work (e.g. find returns the same range type as what's passed
to it even if it's given a string, so it might just work with the
change, or it might need to be tweaked slightly), and those that
would then generally either need to call encode on an argument to
make it match the string type in the cases string types mix (e.g.
"foo".find("fo"d) would need to call encode on "fo"d to make it a
string for comparison), or the caller would need to use
std.utf.byDchar or std.uni.byGrapheme to operate on code points
or graphemes rather than code units.
The two biggest places in Phobos that would potentially have
problems are functions that special-cased strings but still used
front and those which have to return a new range type. e.g.
filter would be a good example, because it's forced to return a
new range type. Right now, it would filter on dchars, but with
the change, it would filter on the code unit type (most typically
char). If you're filtering on ASCII characters, it wouldn't
matter aside from the fact that the resulting range would have an
element type of char rather than dchar, but if you're filtering
on Unicode characters, it wouldn't work anymore. For situations
like that, you'd be forced do use std.utf.byDchar or
std.uni.byGrapheme. However, since most string code tends to
operate on substrings rather than characters, I don't know how
common it even is to use a function like filter on a string (as
opposed to a range of strings). Such code might actually be
fairly rare.
So, there _are_ a few functions which stop working the same way
in a potentially silent manner if we just made it so that front
didn't autodecode anymore. However, in general, because Phobos
almost always special-cases strings, calls to Phobos functions
probably wouldn't need to change in most cases, and when they do,
a call to byDchar would restore the old behavior. But of course,
we'd want to do the transition in a way that didn't result in
silent behavioral changes that would break code, even though in
most cases, it wouldn't matter, because most code will be
operating on ASCII strings even if the strings themselves contain
Unicode - e.g. unicodeString.find(asciiString) is far more common
than unicodeString.find(otherUnicodeString).
I suspect that the code that's at the greatest risk is code that
checks for is(Unqual!(ElementType!Range) == dchar) to operate on
strings and wrapper ranges around strings, since it would then
only match the cases where byDchar had been used. In general
though, the code that's going to run into the most trouble is
user code that contains range-based functions similar to what you
might find in Phobos rather than code that's simply using the
Phobos functions like startsWith and find - i.e. if you're
writing range-base code that worries about doing stuff like
special-casing strings or which specifically needs to operate on
code points, then you're going to have to make changes, whereas
to a great extent, if all you're doing is passing strings to
Phobos functions, your code will tend to just work.
To actually see what the impact would be, we'd have to just
change Phobos, I think, and then see what the impact was on user
code. It could be surprising how much or how little it affects
things, though in most cases, I expect that it'll mean that code
will just work. And if we really wanted to do that, we could
create a version flag that turned of autodecoding and version the
changes in Phobos appropriately to see what we got. In many
cases, if we simply made sure that Phobos functions which
special-cased strings didn't use front directly but instead
didn't care whether they were operating on ranges of char, wchar,
or dchar, then we wouldn't even need to version anything (e.g.
find could easily be made to work that way if it doesn't
already), but some functions (like filter) would need to be
versioned differently.
So, maybe what we need to do to start is to just go through
Phobos and make as many functions as possible not care about
whether they're dealing with strings as ranges of char, wchar, or
dchar. And at least then, we'd minimize how much code would have
to be versioned differently if we were to test out getting rid of
autodecoding with versioning.
- Jonathan M Davis
