> Jeff Clites <[EMAIL PROTECTED]> wrote: > >>On Apr 9, 2004, at 7:19 AM, Leopold Toetsch wrote:
I'm replying for Jeff since I've been burned by the same questions over and over again :-) > >>So internally, strings don't have an associated encoding (or chartype >>or anything) > > > How do you handle EBCDIC? UTF8 for Ponie? All character sets (like EBCDIC) or encodings (like UTF-8) are "normalized" to the Unicode (character set) (and our own *internal* encoding, the 8/16/32 one.) > Not used *yet* - what about: > > use German; > print uc("i"); > use Turkish; > print uc("i"); That is implementable (and already implemented by ICU) but by something higher level than a "string". > And if one is working with two different language at a time? One becomes mad. As Jeff demonstrated, there is no silver bullet in there, one gets quickly to situations where there provably is NO correct solution. So we shouldn't try building the impossible to the lowest level of string implementation. > when comparing graphemes or letters. The latter might depend on the > language too. > > We'll basically need 4 levels of string support: > > ,--[ Larry Wall ]-------------------------------------------------------- > | level 0 byte == character, "use bytes" basically > | level 1 codepoint == character, what we seem to be aiming for, vaguely > | level 2 grapheme == character, what the user usually wants > | level 3 letter == character, what the "current language" wants > `------------------------------------------------------------------------ Jeff's solution gives us level 1, and I assume that level 0 is trivially deductible from that. Note, however, that not all string operations (especially such a rich set of string ops as Perl has) can even be defined for all those levels: e.g. bitstring boolean bit ops are rather insane at levels higher than zero. > The N-th character depends on the level. Above examples C<.length> gives > either 2 or 1, when the user queries at level 1 or 2. The same problem > arises with positions. The current level depends on the scope were the > string was coming from too. (s. example WRT turkish letter "i") The levels 2 and 3 depend on something higher level, like the higher levels of ICU. I believe we have everything we need (and even more) in ICU. Let's get the levels 0 and 1 working first. >>>- What's the plan towards all the transcode opcodes? (And leaving these >>> as a noop would have been simpler) > > >>Basically there's no need for a transcode op on a string--it no longer >>makes sense, there's nothing to transcode. > > > I can't imagine that. I've an ASCII string and want to convert it to UTF8 > and UTF16 and write it into a file. How do I do that? IIUC the old "transcoding" stuff was doing transcoding in run-time so that two encoding-marked strings could be compared. The new scheme "normalizes" (not to be confused with Unicode normalization) all strings to Unicode. If you want to do transformations like you describe above you either call an explicit transcoding interface (which ICU no doubt has) or your I/O layers do that implicitly (this functionality PIO does not yet have, if I understood Jeff correctly). Maybe it's good to refresh on the 'character hierarchy' as defined by Unicode (and IETF, and W3C). ACR - Abstract Character Repertoire: an unordered collection of abstract characters, like "UPPERCASE A" or "LOWERCASE B" or "DECIMAL DIGIT SEVEN". CCS - Coded Character Set: an ordered (numbered) list of characters, like 65 -> "UPPERCASE A". For example: ASCII and EBCDIC. CEF - Character Encoding Form: mapping the numbers of the CCS character codoes to platform-specific numbers like bytes or integers. CES - Character Encoding Scheme: mapping the CEF numbers to serialized bytes, possibly adding synchronization metadata like shift codes or byte order markers. Why the great confusion exists is mostly because in the old way (like ASCII or Latin-1) all these four levels were conflated into one. ISO 8859-1 (which is a CCS) has an eight-bit CEF. UTF-8 is both a CEF and a CES. UTF-16 is a CEF, while UTF-16LE is a CES. ISO 2022-{JP,KR} are CES. (Outside of Unicode) there is TES (Transfer Encoding Syntax), too, which is application-level encoding like base64 or gzip.