Sounds to me like we need a lazy Data.Text variation that allows UTF-8 and UTF-16 "segments" in it list of strict text elements :) Then big chunks of western text will be encoded efficiently, and same with CJK! Not sure what to do about strict Data.Text though :)
On Tue, Aug 17, 2010 at 1:40 PM, Ketil Malde <[email protected]> wrote: > Michael Snoyman <[email protected]> writes: > > > As far as space usage, you are correct that CJK data will take up more > > memory in UTF-8 than UTF-16. > > With the danger of sounding ... alphabetist? as well as belaboring a > point I agree is irrelevant (the storage format): > > I'd point out that it seems at least as unfair to optimize for CJK at > the cost of Western languages. UTF-16 uses two bytes for (most) CJK > ideograms, and (all, I think) characters in Western and other phonetic > scripts. UTF-8 uses one to two bytes for a lot of Western alphabets, > but three for CJK ideograms. > > Now, CJK has about 20K ideograms, which is almost 15 bits per ideogram, > while an ASCII letter is about six bits. Thus, the information density > of CJK and ASCII is about equal for UTF-8, 5/8 vs 6/8 - compared to > 15/16 vs 6/16 for UTF-16. In other words a given document translated > between Chinese and English should occupy roughly the same space in > UTF-8, but be 2.5 times longer in English for UTF-16. > > -k > -- > If I haven't seen further, it is by standing in the footprints of giants > _______________________________________________ > Haskell-Cafe mailing list > [email protected] > http://www.haskell.org/mailman/listinfo/haskell-cafe >
_______________________________________________ Haskell-Cafe mailing list [email protected] http://www.haskell.org/mailman/listinfo/haskell-cafe
