> Which Unicode encoding scheme? UTF-8? UTF-16? UTF-32? > http://www.unicode.org/standard/principles.html
Don't forget about UCS-2. UCS-2 is a subset of UTF-16, because it can only represent code points (unicode characters) in the range 0 - 0xFFFF, whereas UTF-16 can represent all the unicode characters, 0 - 0x10FFFF (by making use of surrogate code points). To represent a unicode character greater than 0xFFFF in UTF-16, it uses two 16-bit numbers (4 bytes) in UTF-16. So a UTF-16 decoder can read UCS-2 just fine, but not vice-verse. Older versions of Windows couldn't really support UTF-16 completely, just UCS-2. See this page: http://blogs.msdn.com/michkap/archive/2005/05/11/416552.aspx Also, don't forget about UCS-4, which is pretty much identical to UTF-32. Each code point is represented by 4 bytes. Final thought: UTF-16 and UTF-32 can be encoded with big-endian or little-endian order, so really we have these encoding schemes: UCS-2 (subset of UTF-16) UCS-4 (virtually the same as UTF-32) UTF-8 UTF-16BE UTF-16LE UTF-32BE UTF-32LE I think Java can handle at least these types: UTF-8, UTF-16BE, UTF16-LE, and UTF-16 One funny thing about Microsoft is that they always try to equate Unicode with UTF-16, and a lot of people I have talked to have adopted the same habit. Everyone thinks that Unicode == two byte characters. Hello, it depends on how you encode it! <deep breath>ok, I'm glad I got that off my chest</deep breath> In notepad, for instance, when you save a text file you can pick one of these encodings: Unicode Unicode big endian UTF-8 Obviously, what MS should have said was this: UTF-16 little endian UTF-16 big endian UTF-8 Or if they want to put the word Unicode in there, fine, but don't leave UTF-8 out! Unicode UTF-16 little endian Unicode UTF-16 big endian Unicode UTF-8 > Unfortunately I have no idea what encoding assumptions that > BufferedReader makes about the character-input stream, or whether it > attempts to convert UTF-8 into UTF-16. I've not played with it enough to > know. I think BufferedReader is smart enough to recognize a BOM (byte order mark) at the beginning of a UTF-16 stream, and use the correct endianness, but I don't think it's smart enough to use the BOM to distinguish between UTF-8 and UTF-16. You can find out more about BOM here: http://www.unicode.org/faq/utf_bom.html#22 Phillip P.S. <puzzling question>Why did I stay up so late just to write a dumb e-mail about Unicode?</puzzling question> -- Phillip Hellewell <phillip AT hellewell.homeip.net> -------------------- BYU Unix Users Group http://uug.byu.edu/ The opinions expressed in this message are the responsibility of their author. They are not endorsed by BYU, the BYU CS Department or BYU-UUG. ___________________________________________________________________ List Info: http://uug.byu.edu/cgi-bin/mailman/listinfo/uug-list
