On approximately 4/29/2009 12:29 AM, came the following characters from
the keyboard of Martin v. Löwis:
C. File on disk with the invalid surrogate code, accessed via the str
interface, no decoding happens, matches in memory the file on disk
with
the byte that translates to the same surrogate, accessed via the bytes
interface. Ambiguity.
Is that an alternative to A and B?
I guess it is an adjunct to case B, the current PEP.
It is what happens when using the PEP on a system that provides both
bytes and str interfaces, and both get used.
Your formulation is a bit too stenographic to me, but please trust me
that there is *no* ambiguity in the case you construct.
No Martin, the point of reviewing the PEP is to _not_ trust you, even
though you are generally very knowledgeable and very trustworthy. It is
much easier to find problems before something is released, or even
coded, than it is afterwards.
Sure. However, that requires you to provide meaningful, reproducible
counter-examples, rather than a stenographic formulation that might
hint some problem you apparently see (which I believe is just not
there).
You assumed, and maybe I wasn't clear in my statement.
By "accessed via the str interface" I mean that (on Windows) the wide
string interface would be used to obtain a file name.
What does that mean? What specific interface are you referring to to
obtain file names? Most of the time, file names are obtained by the
user entering them on the keyboard. GUI applications are completely
out of the scope of the PEP.
Now, suppose that
the file name returned contains "abc" followed by the half-surrogate
U+DC10 -- four 16-bit codes.
Ok, so perhaps you might be talking about os.listdir here. Communication
would be much easier if I would not need to guess what you may mean.
os.listdir("")
Also, why is U+DC10 four 16-bit codes?
It isn't.
First 16-bit code is U+0061
Second 16-bit code is U+0062
Third 16-bit code is U+0063
Fourth 16-bit code is U+DC10
Then, ask for the same filename via the bytes interface, using UTF-8
encoding.
How do you do that on Windows? You cannot just pick an encoding, such
as UTF-8, and pass that to the byte interface, and expect it to work.
If you use the byte interface, you need to encode in the file system
encoding, of course.
Also, what do you mean by "ask for"?????? WHAT INTERFACE ARE YOU
USING???? Please use specific python code.
os.listdir(b"")
I find that on my Windows system, with all ASCII path file names, that I
get quite different results when I pass os.listdir an empty str vs an
empty bytes.
Rather than keep you guessing, I get the root directory contents from
the empty str, and the current directory contents from an empty bytes.
That is rather unexpected.
So I guess I'd better suggest that a specific, equivalent directory name
be passed in either bytes or str form.
The PEP says that the above name would get translated to
"abc" followed by 3 half-surrogates, corresponding to the 3 UTF-8 bytes
used to represent the half-surrogate that is actually in the file name,
specifically U+DCED U+DCB0 U+DC90. This means that one name on disk can
be seen as two different names in memory.
You are relying on false assumptions here, namely that the UTF-8
encoding would play any role.
What would happen instead is that the "mbcs" encoding would be used. The
"mbcs" encoding, by design from Microsoft, will never report an error,
so the error handler will not be invoked at all.
So what you are saying here is that Python doesn't use the "A" forms of
the Windows APIs for filenames, but only the "W" forms, and uses lossy
decoding (from MS) to the current code page (which can never be UTF-8 on
Windows).
You are further saying that Python doesn't give the programmer control
over the codec that is used to convert from W results to bytes, so that
on Windows, it is impossible to obtain a bytes result containing UTF-8
from os.listdir, even though sys.setfilesystemencoding exists, and
sys.getfilesystemencoding is affected by it, and the latter is
documented as returning "mbcs", and as returning the codec that should
be used by the application to convert str to bytes for filenames.
(Python 3.0.1).
While I can hear a "that is outside the scope of the PEP" coming, this
documentation is confusing, to say the least.
Now posit another file which, when accessed via the str interface, has
the name "abc" followed by U+DCED U+DCB0 U+DC90.
Looks ambiguous to me. Now if you have a scheme for handling this case,
fine, but I don't understand it from what is written in the PEP.
You were just making false assumptions in your reasoning, assumptions
that are way beyond the scope of the PEP.
Absolutely correct. I was making what seemed to be reasonable
assumptions about Python internals on Windows, and several of them are
false, including misleading documentation for listdir (which doesn't
specify that bytes and str parameters affect whether or not the current
directory is honored), and sys.getfilesystemencoding (which reflects the
result of sys.setfilesystemencoding, rather than returning, on Windows,
the "mbcs" used by Python to create bytes forms of filenames from W
forms of filenames even after sys.setfilesystemencoding is called.
Things are a little clearer in the documentation for
sys.setfilesystemencoding, which does say the encoding isn't used by
Windows -- so why is it permitted to change it, if it has no effect?).
--
Glenn -- http://nevcal.com/
===========================
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
