On Tue, 23 Dec 2003, Jarkko Hietaniemi wrote: > > It works because it relies > > on iconv(3) to convert between the current locale codeset and UTF-16 > > (used internally by Mozilla) if/wherever possible. 'wc*to*mb/mb*to*wc' > > is only used only where iconv(3) is not available. Anyway, yes, that's > > possible.
> Note that I'm not *opposed* to someone fixing e.g. Win32 being able to > acces Unicode names in NTFS/VFAT. What I'm opposed to is anyone > thinking there are (a) easy (b) portable solutions. We are talking > always of very OS and FS specific solutions. OK. I'm sorry if I misunderstood you. You're absolutely right that we're talking about very OS/FS-dependent issues. > Win32 and Mac OS X are probably the most "well-off". For (other) UNIXy > systems, I don't know. I guess BeOS is in the same league as Win2k/XP [1] and Mac OS X. There, everything should be in UTF-8. > If one is happy > with just using UTF-8 filenames, Perl 5.8 already can work fine. If one I wish everybody were :-) on Unix. Fortunately, UTF-8 seems to be catching on judging from the 'emergence' of two 'file system conversion' tools. See, for instance, <http://osx.freshmeat.net/releases/144059/>. > > If a user mixes multiple encodings/code sets in her/his file > > system, that's not Perl's problem but her/his problem so that I don't > > think that's a valid reason for not doing something reasonable. > wants to use locales and especially some non 8-bit locales, well, Perl > currently most definitely does not switch its "filename encoding" based > on locales. Personally I think that's a daft idea... at least without > a new specific (say) LC_FILENAME control-- overloading the poor LC_CTYPE > sounds dangerous. I don't see how introducing a new LC_* would help here. Whether it's LC_CTYPE or LC_FILENAME, the problem is still there. Perhaps, we need a pragma to indicate which of the following is to be assumed about the file system character encoding, 'locale', 'native', 'unicode', 'user-specified'. On Unix, 'locale' and 'native' would be identical both meaning that Perl should convert its internal Unicode to and from the codeset returned by 'nl_langinfo(CODESET)'. Directly inspecting LC_CTYPE or other environment variables is a BAD idea and should be used as a fallback only where nl_langinfo(CODESET) is not supported. When converting to and from 'native' encoding, it should rely on iconv(3)' available on the system instead of its internal 'encoding' converter. However, there's a problem here. A lot of system admins on commericial Unix install only the minimal set of iconv(3) modules. See <http://bugzilla.mozilla.org/show_bug.cgi?id=202747#c18>. Therefore, perhaps, we first try iconv(3) and then fall back to using Perl's 'encoding'. There are other problems when using iconv(3) (e.g. <http://bugzilla.mozilla.org/show_bug.cgi?id=197051). 'unicode' on Unix means 'utf8'. 'user-specified' means whatever a user wants to use. On Windows, 'locale' means using the code page of the current system locale. 'native' is UTF-16LE (but on Win 9x/ME, the character repertoire would be limited to that of the system codepage). The same is true of 'unicode'. On Mac OS X, locale, native and unicode would mean all the same (UTF-8). As for 'normalization', I have to think more about it. And so on...... I've been just thinking aloud so that you have to bear with some incoherency. Jungshik
