[issue37834] readlink on Windows cannot read app exec links

Change by Łukasz Langa : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___ P

[issue37834] readlink on Windows cannot read app exec links

Łukasz Langa added the comment: New changeset 25a044ee6ce50a9172478cc61d914644778455f6 by Łukasz Langa in branch '3.8': [3.8] bpo-37834: Prevent shutil.rmtree exception (GH-15602) (#15603) https://github.com/python/cpython/commit/25a044ee6ce50a9172478cc61d914644778455f6 -- _

[issue37834] readlink on Windows cannot read app exec links

Change by Łukasz Langa : -- pull_requests: +15280 pull_request: https://github.com/python/cpython/pull/15603 ___ Python tracker ___ _

[issue37834] readlink on Windows cannot read app exec links

Łukasz Langa added the comment: New changeset 7fcc2088a50a4ecb80e5644cd195bee209c9f979 by Łukasz Langa (Ned Deily) in branch 'master': bpo-37834: Prevent shutil.rmtree exception (GH-15602) https://github.com/python/cpython/commit/7fcc2088a50a4ecb80e5644cd195bee209c9f979 -- _

[issue37834] readlink on Windows cannot read app exec links

Change by Ned Deily : -- pull_requests: +15279 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/15602 ___ Python tracker ___

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: Huh, didn't realise those were always defined in stat.py. Changing the test to "hasattr(... "st_file_attributes")" should be fine. I can get to it in a couple of hours if nobody else gets there first. -- ___ Python t

[issue37834] readlink on Windows cannot read app exec links

Ned Deily added the comment: ... and the other important difference is that older versions of macOS do not support fd functions so _use_fd_functions is false and the alternate path is taken in shutil.rmtree, the path that calls _rm_tree_islink which fails. >>> shutil.rmtree('a') Traceback (m

[issue37834] readlink on Windows cannot read app exec links

Ned Deily added the comment: One problem seems to be that the code added for this issue assumes that the documentation is correct in implying that the stat.FILE_ATTRIBUTE_* constants (like stat.FILE_ATTRIBUTE_REPARSE_POINT) are only present on Windows. But besides being conditionally create

[issue37834] readlink on Windows cannot read app exec links

Łukasz Langa added the comment: Unit tests didn't catch it since it fails on older macOS releases. -- ___ Python tracker ___ ___ Pyt

[issue37834] readlink on Windows cannot read app exec links

Łukasz Langa added the comment: There's a bug on macOS that is blocking the release regarding `stat.FILE_ATTRIBUTE_REPARSE_POINT` being used to check whether `os.stat_result` objects have the `st_file_attributes` attribute. -- nosy: +lukasz.langa resolution: fixed -> stage: resolved

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: That should be all the buildbot issues fixes, so I'm marking this resolved and will wait for the inevitable 3.8.0b4 feedback! -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python

[issue37834] readlink on Windows cannot read app exec links

miss-islington added the comment: New changeset 967d625a6df27fb490f035045ec8fe4675001d63 by Miss Islington (bot) in branch '3.8': bpo-37834: Fix test on Windows 7 (GH-15377) https://github.com/python/cpython/commit/967d625a6df27fb490f035045ec8fe4675001d63 -- nosy: +miss-islington _

[issue37834] readlink on Windows cannot read app exec links

Change by miss-islington : -- pull_requests: +15089 pull_request: https://github.com/python/cpython/pull/15378 ___ Python tracker ___ __

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: New changeset 374be59b8e479afa8c7a8ae6e77e98915e2f6d45 by Steve Dower in branch 'master': bpo-37834: Fix test on Windows 7 (GH-15377) https://github.com/python/cpython/commit/374be59b8e479afa8c7a8ae6e77e98915e2f6d45 --

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: Adding a small fix for the Win7 buildbots in PR 15377 -- ___ Python tracker ___ ___ Python-bugs-list

[issue37834] readlink on Windows cannot read app exec links

Change by Steve Dower : -- pull_requests: +15088 pull_request: https://github.com/python/cpython/pull/15377 ___ Python tracker ___ _

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: New changeset 9eb3d5463976068900e94b860ced7e035885835c by Steve Dower in branch '3.8': bpo-37834: Normalise handling of reparse points on Windows (GH-15370) https://github.com/python/cpython/commit/9eb3d5463976068900e94b860ced7e035885835c -- _

[issue37834] readlink on Windows cannot read app exec links

Change by Steve Dower : -- pull_requests: +15081 stage: commit review -> patch review pull_request: https://github.com/python/cpython/pull/15370 ___ Python tracker ___

[issue37834] readlink on Windows cannot read app exec links

Change by Steve Dower : -- stage: patch review -> commit review ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: New changeset df2d4a6f3d5da2839c4fc11d31511c8e028daf2c by Steve Dower in branch 'master': bpo-37834: Normalise handling of reparse points on Windows (GH-15231) https://github.com/python/cpython/commit/df2d4a6f3d5da2839c4fc11d31511c8e028daf2c -- __

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: So my colleagues confirmed that they deliberately represent junction points as symlinks within WSL, including translating the target to the mounted location (assuming it is mounted) and letting the Linux code traverse it normally. They also said they haven't he

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: The latest PR also fixes issue1311 and issue20541 properly (os.path.exists("NUL") now returns True). -- ___ Python tracker ___

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: > So the order of the GetFileInformationByHandleEx and GetFileType blocks > actually needs to be flipped. I've done that now. And thanks for confirming. That was my suspicion, but I wasn't sure if you knew something here that I didn't (v. likely!). --

[issue37834] readlink on Windows cannot read app exec links

Eryk Sun added the comment: > Any particular reason you did GetFileAttributesW(path) in the > non-FILE_TYPE_DISK case when we have the hFile around? The point of getting the file attributes is to identify the root directory of the namedpipe and mailslot file systems. For example, os.listdir(

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: Thanks for the code snippet, that helped me a lot (and since you went to the trouble of fixing other bugs, I guess I'll have to merge it into my PR now). Any particular reason you did GetFileAttributesW(path) in the non-FILE_TYPE_DISK case when we have the hFil

[issue37834] readlink on Windows cannot read app exec links

Eryk Sun added the comment: Here are two additional differences between mount points and symlinks: (1) A mount point in a remote path is always evaluated on the server and restricted to devices that are local to the server. So if we handle a mount point as if it's a POSIX symlink that works

[issue37834] readlink on Windows cannot read app exec links

Eryk Sun added the comment: Here's the requested overview for the case where name-surrogate reparse points are handled as winlinks (Windows links), but S_IFLNK is reserved for IO_REPARSE_TAG_SYMLINK. I took the time this afternoon to write it up in C, which hopefully is clearer than my prose

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: > Why group all reparse points under the banner of 'link'? Because for the purposes of the list of changes beneath it, there wasn't any difference (e.g. "traverses any links supported by the OS" is more meaningful to most people, even though both of us would un

[issue37834] readlink on Windows cannot read app exec links

Eryk Sun added the comment: > the '/mnt/c/Document and Settings' junction... though now I try > it that those don't actually work...) The security on compatibility junctions denies everyone read-data (list) access, but in Windows they can still be traversed (e.g. "C:/Documents and Settings/

[issue37834] readlink on Windows cannot read app exec links

Eryk Sun added the comment: > Where "links" are the generic term for the set that includes > "reparse point", "symlink", "mount point", "junction", etc.) Why group all reparse points under the banner of 'link'? If we have a typical HSM reparse point (the vast majority of allocated tags), the

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: That was a long set of replies, so here's my summary proposed behaviour: (Where "links" are the generic term for the set that includes "reparse point", "symlink", "mount point", "junction", etc.) os.lstat(path): returns attributes for path without traversing an

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: >> I don't want to add any parameters - I want to have predictable and >> reasonable default behaviour. os.readlink() already exists for >> "open reparse point" behaviour. > > I'd appreciate a parameter to always open reparse points, even if a > filter-driver or

[issue37834] readlink on Windows cannot read app exec links

Eryk Sun added the comment: > So for an actual non-root mount point ntpath.ismount() returns True > and with IO_REPARSE_TAG_MOUNT_POINT included ntpath.islink() also > returns True. nt.readlink() returns the "\\?\Volume{GUID}\" path If islink() is true, then st_mode has S_IFLNK and not S_IFD

[issue37834] readlink on Windows cannot read app exec links

Eryk Sun added the comment: > # Always make the OS resolve "unknown" reparse points >ALLOWED_TO_TRAVERSE = {SYMLINK, MOUNT_POINT} >if !traverse and st.reparse_tag not in ALLOWED_TO_TRAVERSE: >return xstat(path, !traverse) To me the naming here makes sense as ALLOWED_TO_OPEN

[issue37834] readlink on Windows cannot read app exec links

Eryk Sun added the comment: > It also has a bug that a drive root is a mount point, even if the > drive doesn't exist. Also, it's wrong in not checking for junctions > in UNC paths. SMB supports opening reparse points over the wire. "It" in the above sentences is ntpath.ismount, not GetVolum

[issue37834] readlink on Windows cannot read app exec links

Eryk Sun added the comment: > Okay, I get it now. So we _do_ want to "upgrade" lstat() to stat() > when it's not a symlink. I don't see that as a behavior upgrade. It's just an implementation detail. lstat() is still following its mandate to not follow symlinks -- however you ultimately def

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: So for an actual non-root mount point, ntpath.ismount() returns True and with IO_REPARSE_TAG_MOUNT_POINT included ntpath.islink() also returns True. nt.readlink() returns the "\\?\Volume{GUID}\" path Root mount points ("C:\\", etc.) do not return true for islin

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: > So we _do_ want to "upgrade" lstat() to stat() when it's not a symlink. Except this bug came about because we want to _downgrade_ stat() to lstat() when it's an appexeclink, because the whole point of those is to use them without following them (and yeah, mos

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: > For example, if we've opened an HSM reparse point, we must reopen to let > the file-system filter driver implement its semantics to replace the > reparse point with the real file from auxiliary storage and complete the > request. That is the stat() result I want

[issue37834] readlink on Windows cannot read app exec links

Eryk Sun added the comment: > Unless your point is that we should _always_ traverse junctions? In > which case we have a traverse 'upgrade' scenario (calls to lstat() > become calls to stat() when we find out it's a junction). If we've opened the reparse point to test IO_REPARSE_TAG_SYMLINK,

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: Unless your point is that we should _always_ traverse junctions? In which case we have a traverse 'upgrade' scenario (calls to lstat() become calls to stat() when we find out it's a junction). Again, not sure why we'd want to hide the ability to manipulate the

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: [Quoting from the PR comments] > traverse is from follow_symlinks and only applies to symlinks. It does not > apply to other types of reparse points. I get your argument that junctions are not symlinks, but I disagree that we should try this hard to emulate PO

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: > I assume you're talking about realpath() here ... Yes, and so are you :) Let's move that discussion to issue9949 and/or PR 15287. > I think os.chdir should raise an exception when passed a device path. When the OS starts returning an error code for this case,

[issue37834] readlink on Windows cannot read app exec links

Eryk Sun added the comment: > I wish we could remove the MAX_PATH limit in this case. > > The problem is that we have to remove the limit in any case where the > resulting path might be used, which is what we're already trying to > encourage by supporting long paths. Maybe it's better to ign

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: And I just posted an update to PR 15231 with essentially a rewrite of stat() on Windows. Should be better than it was :) -- ___ Python tracker

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: > Perhaps the best we can do is an additional test where we GetFinalPathName, > strip the prefix, reopen the file, GetFinalPathName again and if they match > then return it without the prefix. That should handle the both long path > settings as transparently as

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: > Given the only option here is follow_symlinks, then the first CreateFileW > call in win32_xstat_impl should only open a reparse point if follow_symlinks > is false. In this case, if it happens to open a reparse point that's not a > symlink, it should try to r

[issue37834] readlink on Windows cannot read app exec links

Eryk Sun added the comment: > but suddenly adding "\\?\" to the paths breaks a lot of assumptions. The unwritten assumption has been that readlink() is reading symlinks that get created by CreateSymbolicLinkW, which sets the print name as the DOS path that's passed to the call. In this case,

[issue37834] readlink on Windows cannot read app exec links

Eryk Sun added the comment: > I really want a fix for this in 3.8, or else os.stat(sys.executable) > may fail I agree, but Python can support this without handling junctions as symlinks or limiting the reparse points that we can follow. There are reparse points for offline storage and proje

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: > I'm wary of trying to return it without the prefix. Me too, but suddenly adding "\\?\" to the paths breaks a lot of assumptions. > We would need a function that's shared with the proposed implementation of > realpath() to determine whether the given path (not

[issue37834] readlink on Windows cannot read app exec links

Eryk Sun added the comment: > Until then, I think it makes sense for os.readlink() to handle the > prefix and _getfinalpathname() call, but leave nt.readlink() as > returning the raw value. os.readlink() shouldn't resolve the final path or realpath(). It should simply return the link substit

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: I think we'll want issue9949 merged as well, so that ntpath.realpath() does its job. Certainly the tests would benefit from it. Until then, I think it makes sense for os.readlink() to handle the prefix and _getfinalpathname() call, but leave nt.readlink() as re

[issue37834] readlink on Windows cannot read app exec links

Eryk Sun added the comment: > Replacing "\??\" with "\\?\" in place is trivial though, as we start > with a mutable buffer. I'm just not clear that it's as simple as that, > though. If the path starts with "\\??\\" we can just change the first question mark to a backslash. For a symlink, fa

[issue37834] readlink on Windows cannot read app exec links

Eryk Sun added the comment: > I feel like that's more work than is worth us doing for something that > will be relatively rarely used, will live in the stdlib, and is > obviously something that will become outdated as Microsoft adds new > reparse points. Junctions (NT 5) and symlinks (NT 6)

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: Latest PR update uses GetFinalPathName to resolve SubstituteName, returning it unmodified on failure (e.g. symlink where the target file no longer exists). Replacing "\??\" with "\\?\" in place is trivial though, as we start with a mutable buffer. I'm just not

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: > If we support reading junctions, this should be using the substitute name > (with \??\ replaced by \\?\) instead of the print name. GetFinalPathName() does this conversion for us, any reason not to use that? (GetFullPathName() doesn't seem to recognize the \?

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: > I looked into this spawn problem. It's due to Cygwin's spawnve, which calls > NtOpenFile to open the file, and then memory-maps it and reads the image > header [1]. Great, that's roughly what I suspected. Unfortunately, I've been told that looking into Cygwi

[issue37834] readlink on Windows cannot read app exec links

Eryk Sun added the comment: > Honestly, the only real problem I've seen is in applications that use > a reimplementation of spawn() rather than the UCRT's version (which > handles the reparse point jsut fine). I looked into this spawn problem. It's due to Cygwin's spawnve, which calls NtOpenF

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: Thanks Eryk for your valuable response :) > readlink() and is_link() should be reserved for POSIX symlinks, i.e. only > IO_REPARSE_TAG_SYMLINK. I'm okay with that reasoning. Honestly, the only real problem I've seen is in applications that use a reimplementat

[issue37834] readlink on Windows cannot read app exec links

Eryk Sun added the comment: > How about adding a separate function such as nt.read_reparse_point() If we support reading junctions, this should be using the substitute name (with \??\ replaced by \\?\) instead of the print name. For both symlinks and junctions, [MS-FSCC] 2.1.2 states that th

[issue37834] readlink on Windows cannot read app exec links

Eryk Sun added the comment: Symlinks are specially interpreted by the file API, I/O manager, and network redirector. So I think they should remain a separate category. readlink() and is_link() should be reserved for POSIX symlinks, i.e. only IO_REPARSE_TAG_SYMLINK. These app-exec reparse p

[issue37834] readlink on Windows cannot read app exec links

Steve Dower added the comment: Added an update with a new stat field (st_reparse_tag) so that we can tell them apart, which also means I can add a test for readlink. -- assignee: -> steve.dower ___ Python tracker

[issue37834] readlink on Windows cannot read app exec links

Change by Steve Dower : -- keywords: +patch pull_requests: +14955 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15231 ___ Python tracker ___

[issue37834] readlink on Windows cannot read app exec links

Change by Steve Dower : -- nosy: +eryksun ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python

[issue37834] readlink on Windows cannot read app exec links

New submission from Steve Dower : The IO_REPARSE_TAG_APPEXECLINK was introduced for aliases to installed UWP apps that require activation before they can be executed. Currently these are in an unusual place as far as Python support goes - stat() fails where lstat() succeeds, but the lstat() r