On Sep 22 17:12, Andy Koppe wrote: > 2009/9/22 Corinna Vinschen: > >> Therefore, when converting a UTF-16 Windows filename to the current > >> charset, 0xDC?? words should be treated like any other UTF-16 word > >> that can't be represented in the current charset: it should be encoded > >> as a ^N sequence. > > > > How? Just like the incoming multibyte character didn't represent a valid > > UTF-8 char, a single U+DCxx value does not represent a valid UTF-16 char. > > Therefore, the ^N conversion will fail since U+DCxx can't be converted > > to valid UTF-8. > > True, but that's an implementation issue rather than a design issue, > i.e. the ^N conversion needs to do the UTF-8 conversion itself rather > than invoke the __utf8 functions. Shall I look into creating a patch?
Well, sure I'm interested to see that patch (lazy me), but please note that we need a snail mailed copyright assignment per http://cygwin.com/assign.txt from you before we can apply any significant patches. Sorry for the hassle. Hmm... maybe it's not that complicated. The ^N case checks for a valid UTF-8 lead byte right now. The U+DCxx case could be handled by generating (in sys_cp_wcstombs) and recognizing (in sys_cp_mbstowcs) a non-valid lead byte, like 0xff. > >> This won't work correctly, because different POSIX filenames will map > >> to the same Windows filename. For example, the filenames "\xC3\xA4" > >> (valid UTF-8 for a-umlaut) and "\xC4" (invalid UTF-8 sequence that > >> represents a-umlaut in 8859-1), will both map to Windows filename > >> "U+00C4", i.e a-umlaut in UTF-16. Furthermore, after creating a file > >> called "\xC4", a readdir() would show that file as "\xC3\xA4". > > > > Right, but using your above suggestion will also lead to another filename > > in readdir, it would just be \x0e\xsome\xthing. > > I don't think the suggestion above is directly relevant to the problem > I tried to highlight here. > > Currently, with UTF-8 filename encodings, "\xC3xA4" turns into U+00C4 > on disk, while "\xC4" turns into U+DCC4, and converting back yields > the original separate filenames. Well, right now it doesn't exactly. > If I understand your proposal > correctly, both "\xC3\xA4" and "\xC4" would turn into U+00C4, hence > converting back would yield "\xC3\xA4" for both. This is wrong. Those > filenames shouldn't be clobbering each other, and a filename shouldn't > change between open() and readdir(), certainly not without switching > charset inbetween. I see your point. I was more thinking along the lines of how likely that clobbering is, apart from pathological testcases. > Having said that, if you did switch charset from UTF-8 e.g. to > ISO-8859-1, the on-disk U+DCC4 would indeed turn into > "\x0E\xsome\xthing". However, that issue applies to any UTF-16 You don't have to switch the charset. Assume you're using any non-singlebyte charset in which \xC4 is the start of a double- or multibyte sequence. open ("\xC4"); close; readdir(); will return "\x0E\xsome\xthing" on readdir. Only singlebyte charsets are off the hook. So, your proposal to switch to the default ANSI codepage for the C locale would be good for most western languages, but it would still leave the eastern language users with double-byte charsets behind. Note that I'm not as opposed to your proposal to use the ANSI codepage as before this discussion. But I would like to see that the solution works for most eastern language users as well. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple