Hi Juan,
I took the liberty of answering to the list. I hope you don't mind.
Juan Lang wrote:
I most certainly didn't say that. I may have mentioned UCS4, but to the best of my knowledge at the time, Windows uses UTF-16.Hi Schachar, something you said earlier this year at wineconf caught my ear, and I think I'm finally understanding enough to ask about it. You said Windows uses UCS2-LE (I think you did, anyway) rather than UTF16. Is that true?
As a result of your email, I tried to check it again, however. The results are somewhat inconclusive. I generated a file (attached) containing two musical symbols in Unicode. These are from the codepoints above U10000, and are therefor unrepresentable in UCS2.
Opening this file up in notepad reveals a mixed result. On the one hand, they appear as two unknown characters (which seems to suggest that at least Notepad understands this is UTF-16LE). On the other hand, I did not manage to find any font that has any characters with codepoints above U10000.
I'm hoping one of our far-eastern speakers jumps in with more insight (Mike?). Are surrogates used on Windows? How common are they?
Does that then imply that
Windows really can't handle characters outside the
Basic Multilingual Plane? Does that also imply that
WCHARs are in fact fixed-width in Windows?
As far as I know, they are not. Sorry.
I'm not sure what you are aiming at achieving. Are you trying to hit the beginning of the last character of the string? If so, then you do, indeed, have a problem here.I'm planning to write a tool to detect the following problematic bit of code: char str[] = "hi", *p = str + sizeof(str) - 1; p--; At least, it's problematic when str contains double-byte characters.
In the past I have written programs that had to do MBCS (the non-unicode Japanese encoding). This is an encoding in which some characters are one byte, and some two. The best I could come up with was to build a wrapper around std::string that had two bytes per character internally. When you loaded a string, it would check character by character for whether it's a double byte, and then have each string location contain exactly one character. This allowed random access, as well as both forward AND backwards scanning.If the type of char and p were WCHAR * instead, and WCHARs are fixed width, there isn't a problem.
If you have thoughts on similar programming bugs
relating to internationalization (use of strchr and
strrchr, for example) I'd be interested to hear them.
Fortunately, UTF is much better than MBCS. Given a byte in either UTF-8 or UTF-16, it's fairly easy to figure out whether it's part of a surrogate, and what part. If you have assurance that the string you are handling is a well formed one, you can do backward scans of a UTF string fairly easily.
Thanks, --Juan
_______________________________
Do you Yahoo!?
Declare Yourself - Register online to vote today!
http://vote.yahoo.com
Do you want a gmail account?
Shachar
-- Shachar Shemesh Lingnu Open Source Consulting ltd. http://www.lingnu.com/
