Re: Challenge: write a really really small front() for UTF8

[dennis luehring]

Am 24.03.2014 17:44, schrieb Andrei Alexandrescu:
On 3/24/14, 5:51 AM, w0rp wrote:
On Monday, 24 March 2014 at 09:02:19 UTC, monarch_dodra wrote:
On Sunday, 23 March 2014 at 21:23:18 UTC, Andrei Alexandrescu wrote:
Here's a baseline: http://goo.gl/91vIGc. Destroy!
Andrei

Before we roll this out, could we discuss a strategy/guideline in
regards to detecting and handling invalid UTF sequences?

Having a fast "front" is fine and all, but if it means your program
asserting in release (or worst, silently corrupting memory) just
because the client was trying to read a bad text file, I'm unsure this
is acceptable.

I would strongly advise to at least offer an option

Options are fine for functions etc. But front would need to find an
all-around good compromise between speed and correctness.

Andrei


b"\255".decode("utf-8", errors="strict") # UnicodeDecodeError
b"\255".decode("utf-8", errors="replace") # replacement character used
b"\255".decode("utf-8", errors="ignore") # Empty string, invalid
sequence removed.

i think there should be a base range for UTF8 iteration - with policy 
based error extension (like in python) and some variants that defer this 
base UTF8 range with different error behavior - and one of these become 
the phobos standard = default parameter so its still switchable