25-May-2013 10:44, Joakim пишет:
On Friday, 24 May 2013 at 21:21:27 UTC, Dmitry Olshansky wrote:
You seem to think that not only UTF-8 is bad encoding but also one
unified encoding (code-space) is bad(?).
Yes, on the encoding, if it's a variable-length encoding like UTF-8, no,
on the code space. I was originally going to title my post, "Why
Unicode?" but I have no real problem with UCS, which merely standardized
a bunch of pre-existing code pages. Perhaps there are a lot of problems
with UCS also, I just haven't delved into it enough to know. My problem
is with these dumb variable-length encodings, so I was precise in the
title.
UCS is dead and gone. Next in line to "640K is enough for everyone".
Simply put Unicode decided to take into account all diversity of
luggages instead of ~80% of these. Hard to add anything else. No offense
meant but it feels like you actually live in universe that is 5-7 years
behind current state. UTF-16 (a successor to UCS) is no random-access
either. And it's shitty beyond measure, UTF-8 is a shining gem in
comparison.
Separate code spaces were the case before Unicode (and utf-8). The
problem is not only that without header text is meaningless (no easy
slicing) but the fact that encoding of data after header strongly
depends a variety of factors - a list of encodings actually. Now
everybody has to keep a (code) page per language to at least know if
it's 2 bytes per char or 1 byte per char or whatever. And you still
work on a basis that there is no combining marks and regional specific
stuff :)
Everybody is still keeping code pages, UTF-8 hasn't changed that.
Legacy. Hard to switch overnight. There are graphs that indicate that
few years from now you might never encounter a legacy encoding anymore,
only UTF-8/UTF-16.
Does
UTF-8 not need "to at least know if it's 2 bytes per char or 1 byte per
char or whatever?"
It's coherent in its scheme to determine that. You don't need extra
information synced to text unlike header stuff.
It has to do that also. Everyone keeps talking about
"easy slicing" as though UTF-8 provides it, but it doesn't. Phobos
turns UTF-8 into UTF-32 internally for all that ease of use, at least
doubling your string size in the process. Correct me if I'm wrong, that
was what I read on the newsgroup sometime back.
Indeed you are - searching for UTF-8 substring in UTF-8 string doesn't
do any decoding and it does return you a slice of a balance of original.
In fact it was even "better" nobody ever talked about header they just
assumed a codepage with some global setting. Imagine yourself creating
a font rendering system these days - a hell of an exercise in
frustration (okay how do I render 0x88 ? mm if that is in codepage XYZ
then ...).
I understand that people were frustrated with all the code pages out
there before UCS standardized them, but that is a completely different
argument than my problem with UTF-8 and variable-length encodings. My
proposed simple, header-based, constant-width encoding could be
implemented with UCS and there go all your arguments about random code
pages.
No they don't - have you ever seen native Korean or Chinese codepages?
Problems with your header based approach are self-evident in a sense
that there is no single sane way to deal with it on cross-locale basis
(that you simply ignore as noted below).
This just shows you don't care for multilingual stuff at all. Imagine
any language tutor/translator/dictionary on the Web. For instance most
languages need to intersperse ASCII (also keep in mind e.g. HTML
markup). Books often feature citations in native language (or e.g.
latin) along with translations.
This is a small segment of use and it would be handled fine by an
alternate encoding.
??? Simply makes no sense. There is no intersection between some legacy
encodings as of now. Or do you want to add N*(N-1) cross-encodings for
any combination of 2? What about 3 in one string?
Now also take into account math symbols, currency symbols and beyond.
Also these days cultures are mixing in wild combinations so you might
need to see the text even if you can't read it. Unicode is not only
"encode characters from all languages". It needs to address universal
representation of symbolics used in writing systems at large.
I take your point that it isn't just languages, but symbols also. I see
no reason why UTF-8 is a better encoding for that purpose than the kind
of simple encoding I've suggested.
We want monoculture! That is to understand each without all these
"par-le-vu-france?" and codepages of various complexity(insanity).
I hate monoculture, but then I haven't had to decipher some screwed-up
codepage in the middle of the night. ;)
So you never had trouble of internationalization? What languages do you
use (read/speak/etc.)?
That said, you could standardize
on UCS for your code space without using a bad encoding like UTF-8, as I
said above.
UCS is a myth as of ~5 years ago. Early adopters of Unicode fell into
that trap (Java, Windows NT). You shouldn't.
Want small - use compression schemes which are perfectly fine and get
to the precious 1byte per codepoint with exceptional speed.
http://www.unicode.org/reports/tr6/
Correct me if I'm wrong, but it seems like that compression scheme
simply adds a header and then uses a single-byte encoding, exactly what
I suggested! :)
This is it but it's far more flexible in a sense that it allows
multi-linguagal strings just fine and lone full-with unicode codepoints
as well.
But I get the impression that it's only for sending over
the wire, ie transmision, so all the processing issues that UTF-8
introduces would still be there.
Use mime-type etc. Standards are always a bit stringy and suboptimal,
their acceptance rate is one of chief advantages they have. Unicode has
horrifically large momentum now and not a single organization aside from
them tries to do this dirty work (=i18n).
And borrowing the arguments from from that rant: locale is borked shit
when it comes to encodings. Locales should be used for tweaking visual
like numbers, date display an so on.
Is that worse than every API simply assuming UTF-8, as he says? Broken
locale support in the past, as you and others complain about, doesn't
invalidate the concept.
It's combinatorial blowup and has some stone-walls to hit into. Consider
adding another encoding for "Tuva" for isntance. Now you have to add 2*n
conversion routines to match it to other codepages/locales.
Beyond that - there are many things to consider in internationalization
and you would have to special case them all by codepage.
If they're screwing up something so simple,
imagine how much worse everyone is screwing up something complex like
UTF-8?
UTF-8 is pretty darn simple. BTW all it does is map [0..10FFFF] to a
sequence of octets. It does it pretty well and compatible with ASCII,
even the little rant you posted acknowledged that. Now you are either
against Unicode as whole or what?
--
Dmitry Olshansky
