[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode

Inada Naoki added the comment: Mercurial still use it. https://www.mercurial-scm.org/repo/hg-stable/file/tip/mercurial/pycompat.py#l113 Mercurial has plan to move filesystem name from ANSI Code Page to UTF-8, but I don't know about its progress.

[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode

Irit Katriel added the comment: With 3.6 being over, is _enablelegacywindowsfsencoding still needed or is it time to deprecate it? -- nosy: +iritkatriel ___ Python tracker

[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode

Steve Dower added the comment: I took another look at this and it's still unclear whether it's worth the performance loss. Perhaps moving fsencode and fsdecode (almost) entirely into C would be a better approach? That shouldn't send us backwards at all, and all they

[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode

Steve Dower added the comment: How? -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode

STINNER Victor added the comment: Can't we just update the cache when the function changes the encoding? -- ___ Python tracker ___

[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode

Steve Dower added the comment: Thanks for checking that. I don't think it's worth retaining the cache on Windows in the face of the broken behaviour. Any real-world case where a lot of paths are being encoded or decoded is also likely to involve file-system access which will dwarf the

[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode

JGoutin added the comment: A little encoding cache benchmark. Current Code: = import sys def _fscodec(): encoding = sys.getfilesystemencoding() errors = sys.getfilesystemencodeerrors() def fsencode(filename): filename = fspath(filename) # Does type-checking

[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode

JGoutin added the comment: Yes, I reported this encoding issue to some of them. But, there is still some problems : - Some libraries are not updated frequently (Or not still maintained), and still use fsencode. - Tests and CI don't see this problem if they don't have a test case for filename

[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode

STINNER Victor added the comment: Hum, it was long time ago since I worked on Windows. Well, Python has a "mbcs" codec which uses the ANSI code page which exists like "forever". These libraries should be patched to use "mbcs" instead of sys.getfilesystemencoding(). --

[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode

STINNER Victor added the comment: > Temporary fixing issues with some third party libraries which use C code for > files I/O (With filename as "mbcs" encoded bytes internally). > > Theses libraries generally call > "filename.encode(sys.getfilesystemencoding())" or "os.fsencode(filename)" >

[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode

JGoutin added the comment: Personally, I call "sys._enablelegacywindowsfsencoding()" for only one reason : Temporary fixing issues with some third party libraries which use C code for files I/O (With filename as "mbcs" encoded bytes internally). Theses libraries generally call

[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode

Steve Dower added the comment: Windows doesn't use the fs encoding at all until Python code requests/provides something in bytes. Except for the caching in fsencode/fsdecode, there's no problem setting it once at the start of your program (and it can only be set once - there's no parameter

[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode

STINNER Victor added the comment: My experience with changing the Python "filesystem encoding" (sys.getfilesystemencoding()) at runtime: it doesn't work. The filesystem encoding must be set as soon as possible and must never change later. As soon as possible: before the first call to

[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode

Marc-Andre Lemburg added the comment: Adding Victor, who implemented the fs codec. AFAIK, it's not possible to change the encoding after interpreter initialization, since it will have been already used for many different things by the time you get to executing code. -- nosy: +haypo,

[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode

Steve Dower added the comment: Then we do in fact need to make os.fsencode/fsdecode either stop caching the encoding completely, or figure out a way to reset the cache when that function is called. -- ___ Python tracker

[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode

JGoutin added the comment: import sys # Force the use of legacy encoding like versions of Python prior to 3.6. sys._enablelegacywindowsfsencoding() # Show actual file system encoding encoding = sys.getfilesystemencoding() print('File system encoding:', encoding) # os.fsencode(filename) VS

[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode

Steve Dower added the comment: If you import os first then that's acceptable and we should document it more clearly. Try calling enable before importing os. I wouldn't be surprised if os is imported automatically, in which case we need to figure out some alternate caching mechanism that can

[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode

New submission from JGoutin: The doc say that calling "sys._enablelegacywindowsfsencoding()" is equivalent to use "PYTHONLEGACYWINDOWSFSENCODING" environment variable. In fact, this no apply to "os.fsencode" and "os.fsdecode". Example with Python 3.6 64Bits on Windows 7 64 bits : EXAMPLE