On 14/12/2004 11:32, Arcane Jill wrote:
I've been following this thread for a while, and I've pretty much got the hang of the issues here. To summarize:
I haven't followed everything, but here is my 2 cents worth.
I note that there is a real problem. I have had significant problems in Windows with files copied from other language systems. Sometimes for example these files are listed fine in Explorer but when I try to copy or delete them they are not found, presumably because the filename is being corrupted somewhere in the system and doesn't match.
This is not in fact a new concept. Some octet sequences which are valid filenames are invalid in a Latin-1 locale - for example, those which include octets in the range 0x80-0x9F, if "Latin-1" means ISO 8859-1. Some of these octets are of course defined in Windows CP1252 etc, so a Unix Latin-1 system may have some interpretation for some of them; but others e.g. 0x81 have no interpretation in any flavour of Latin-1 as far as I know. So there is by no means a guarantee that every non-Unicode Unix locale has an interpretation of every octet, which implies that other octets are invalid.
Unix filenames consist of an arbitrary sequence of octets, excluding 0x00 and 0x2F. How they are /displayed/ to any given user depends on that user's locale setting. In this scenario, two users with different locale settings will see different filenames for the same file, but they will still be able to access the file via the filename that they see. These two filenames will be spelt identically in terms of octets, but (apparently) differently when viewed in terms of characters.
At least, that's how it was until the UTF-8 locale came along. If we consider only one-byte-per-character encodings, then any octet sequence is "valid" in any locale. But UTF-8 introduces the possibility that an octet sequence might be "invalid" - a new concept for Unix. So if you change your locale to UTF-8, then suddenly, some files created by other users might appear to you to have invalid filenames (though they would still appear valid when viewed by the file's creator).
Now no doubt many Unix filename handling utilities ignore the fact that some octets are invalid or uninterpretable in the locale, because they handle filenames as octet strings (with 0x00 and 0x2F having special interpretations) rather than as locale-dependent character strings. But these routines should continue to work in a UTF-8 locale, as they make no attempt to interpret any octets other than 0x00 and 0x2F.
A specific example: if a file F is accessed by two different users, A and B, of whom A has set their locale to Latin-1, and B has set their locale to UTF-8, then the filename may appear to be valid to user A, but invalid to user B.
Lars is saying (and he's probably right, because he knows more about Unix than I) that user B does not necessarily have the right to change the actual octet sequence which is the filename of F, just to make it appear valid to user B, because doing so would stop a lot of things working for user A (for instance, A might have created the file, the filename might be hardcoded in a script, etc.). So Lars takes a Unix-like approach, saying "retain the actual octet sequence, but feel free to try to display and manipulate it as if it were some UTF-8-like encoding in which all octet sequences are valid". And all this seems to work fine for him, until he tries to roundtrip to UTF-16 and back.
I think the problem here is that a Unix filename is a string of octets, not of characters. And so it should not be converted into another encoding form as if it is characters; it should be processed at a quite different level of interpretation.
Of course a system is free to do what it wants internally.
All of this is ingenious, and may be useful for internal processing within a Unix system, and perhaps even for interaction between cooperating systems. But NOT-Unicode is not Unicode (!) and so Unicode should not be expected to standardise it.
I'm not sure why anyone's arguing about this though - Phillipe's suggestion seems to be the perfect solution which keeps everyone happy. So...
...allow me to construct a specific example of what Phillipe suggested only generally:
...
This would appear to solve Lars' problem, and because the three encodings, NOT-UTF-8, NOT-UTF-16 and NOT-UTF-32, don't claim to be UTFs, no-one need get upset.
I can see that there may be a need for a protocol for open exchange of Unix-like filenames. But these filenames should be treated as binary data (which may or may not be interpretable in any one locale) and encoded as such, rather than forced into the mould of Unicode characters which it does not fit.
-- Peter Kirk [EMAIL PROTECTED] (personal) [EMAIL PROTECTED] (work) http://www.qaya.org/