[issue30044] shutil.copystat should (allow to) copy ownership, and other attributes

[Steve Dower]

Steve Dower  added the comment:

Just wanted to add that the sooner we can offer a wrapper around CopyFileEx on 
Windows (with no callbacks), the sooner we can take advantage of some 
significant optimisations that are being done to this function (which I can't 
share details of right now, but concrete work is being done).

Personally, I'm fine with it being copy2() and we take the slight behaviour 
change. But it should definitely be easy for users to access and use.

--
nosy: +steve.dower

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30044] shutil.copystat should (allow to) copy ownership, and other attributes

[Eryk Sun]

Eryk Sun  added the comment:

> For Windows though, I'm unsure.

If copystat() gains the ability to copy the file's owner and group in POSIX, it 
is not a priority to mirror this capability in Windows, which doesn't implement 
anything like the Unix owner-group-other permission model. The only security 
metadata that I expect to be copied in Windows is the file's resource 
attributes, which are named attributes set on an object -- and considered 
inherent to the object -- for use in conditional access-control entries. 
Normally, the rest of the security descriptor is not copied, including the 
owner, group, mandatory label (integrity level and read-up/write-up/execute-up 
access), discretionary access-control list, and system access-control list 
(audit entries). That level of deep copying is a backup and restore operation 
of the complete file context, not a normal file copy.

Regarding copy2(), what I've seen proposed a couple of times for Windows is to 
implement it via CopyFileExW(). The documentation of copy2() would have to be 
special cased to explain that the platform copy routine is used in Windows, 
which copies all named (alternate) data streams and settable file attributes, 
extended attributes, and resource attributes. It would also need to be 
emphasized that copy2() is not equivalent to copyfile() + copystat() in Windows.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30044] shutil.copystat should (allow to) copy ownership, and other attributes

[David Justo]

David Justo  added the comment:

Hi folks,
I'd be interested in contributing to this issue, since it seems "easy" enough, 
but I'm unsure that consensus was reached about the right solution.

>From what I gathered from Giampaolo's comments, we have solutions that should 
>work for Unix-y systems and MacOS. So that's clear.

For Windows though, I'm unsure. It appears, we'd need kernel-mode-enabled 
clients to do this and maybe that's enough to discourage this effort altogether.

So it appears we have 3 options then:
1. Implement a new function that copies ownership in linux+mac but not in 
Windows. Then to document that difference.
2. Implement a new function that copies ownership in linux+mac, then use the 
native NT API to achieve the same result in Windows. Document the limitations
3. Just document the existing limitations.

Do we have a preference? I do not, just excited to potentially contribute 
something :) 

Thanks!

--
nosy: +davidmrdavid

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30044] shutil.copystat should (allow to) copy ownership, and other attributes

[Rahul Jha]

Change by Rahul Jha :


--
nosy: +RJ722

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30044] shutil.copystat should (allow to) copy ownership, and other attributes

[Eryk Sun]

Eryk Sun  added the comment:

> Since the need to copy file ownership is common, I think there could
> be space for a new copy3() function which copies ownership + extended
> attributes (where possible).

FYI, Windows and POSIX have significantly different concepts about file 
(object) ownership. In Windows:

* Any type of SID can be set as the owner, such as a user, global
  group, local group, well-known group, domain, or logon session. All
  of these SID types, except for user SIDs, are commonly set in the
  groups of a token. Also, the token user is not limited to just users.
  It's commonly set to a well-known group such as SYSTEM, LOCAL
  SERVICE, or NETWORK SERVICE.

* The effective access token of a thread is granted owner rights to
  an object if the token user or any of the token's enabled groups is
  the owner of the object. For example, if an object is owned by the
  "BUILTIN\Users" local group, then all access tokens for standard-user
  logons will be granted owner rights as long as they have the
  "BUILTIN\Users" group enabled, which it is by default.
  
* If not set explicitly via "OWNER RIGHTS" (i.e. S-1-3-4), the 
  owner is implicitly granted the READ_CONTROL right to query the
  object security and the WRITE_DAC right to modify the object's
  resource attributes and discretionary access-control list. As 
  long as these rights are granted implicitly, they cannot be
  denied by deny access-control entries. However, implicit owner
  rights may be denied if an object has an implicit (by object 
  type) or explicit (by label) no-read-up or no-write-up mandatory
  policy, and the token's integrity level is less than that of the
  object.

* An explicit "OWNER RIGHTS" entry can be set in the discretionary
  access control list in order to override the implicit owner rights.
  This is not the same as setting owner rights in POSIX, since other
  ACL entries may grant or deny rights. Given the canonical priority 
  of deny access-control entries and also mandatory access control
  based on the integrity level of the object vs the token, granting
  explicit access to "OWNER RIGHTS" does not necessarily ensure the
  owner will even be granted at least the desired access in all
  contexts. Also, unlike the implicit case, if an "OWNER RIGHTS"
  entry grants READ_CONTROL and/or WRITE_DAC access, either right 
  may be denied by deny access-control entries.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30044] shutil.copystat should (allow to) copy ownership, and other attributes

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

Sorry, after re-reading Eryk's comment, it seems I'm not correct about 
CopyFileEx.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30044] shutil.copystat should (allow to) copy ownership, and other attributes

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

Since the need to copy file ownership is common, I think there could be space 
for a new copy3() function which copies ownership + extended attributes (where 
possible). In detail:

- on Windows this can be achieved by using CopyFileEx, which would also provide 
better overall performances than the current readinto() implementation. This 
was the original approach used in BPO-33671, discarded after Eryk pointed out 
that ACLs should not be copied.

- on macOS we can use fcopyfile() (which is already exposed privately) + 
COPYFILE_ALL, see http://www.manpagez.com/man/3/fcopyfile/

- on all other UNIX we can use the current shutil.copystat() (which already 
copies xattrs) + os.chown(). I am not entirely sure this would achieve a "full 
ACLs copy" though, or what would be needed exactly to achieve that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30044] shutil.copystat should (allow to) copy ownership, and other attributes

[Eryk Sun]

Eryk Sun  added the comment:

In Windows, I wouldn't expect shutil.copy2 to preserve the owner and ACLs. They 
change whenever a file gets copied via CopyFileExW [1]. Keeping them exactly as 
in the source file generally requires a privileged backup and restore 
operation, such as via BackupRead [2] and BackupWrite [3]. Unless the caller 
has SeRestorePrivilege, the owner can only be set to one of the SIDs in the 
caller's groups that are flagged as SE_GROUP_OWNER, which is usually just the 
user's SID or, for an admin, the Administrators SID. Also, for copying the 
system ACL, adding or removing audit and scoped-policy-identifier entries 
requires SeSecurityPrivilege.

CopyFileExW copies all data streams in a file, which is typically just the 
anonymous data stream, but an NTFS/ReFS file can have multiple named data 
streams. For metadata, it copies the change and modify timestamps (but not the 
create and access timestamps), file attributes (readonly, hidden, system, 
archive, temporary, not-content-indexed), extended attributes, and resource 
attributes [4]. 

Separating this functionality into shutil.copy and shutil.copystat would be 
fairly involved. These functions could be left as is and just document the 
discrepancy in shutil.copy2, or new functions could be implemented in the nt or 
_winapi module to list the data streams in a file and get/set file attributes 
and system resource attributes. Supporting extended attributes would require 
the native NT API, and for little benefit since they're mostly used for 
"$Kernel." prefixed attributes that can only be set by kernel-mode callers such 
as drivers.

---

[1]: 
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-copyfileexw
[2]: 
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-backupread
[3]: 
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-backupwrite
[4]: Resource attributes are like extended attributes, but a named resource 
attribute is a tuple of one or more items with a given data type (integer, 
string, or bytes) that's stored as an entry in the file's system ACL. Keeping 
them in the SACL allows conditional access/audit entries to reference them in 
an access check or access audit. Unlike audit entries in the SACL, reading and 
writing resource attributes doesn't require SeSecurityPrivilege.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30044] shutil.copystat should (allow to) copy ownership, and other attributes

[Christian Heimes]

Christian Heimes  added the comment:

POSIX ACLs and SELinux context information are stored in extended file 
attributes. The information is copied from source to destination. POSIX ACLs 
are stored in xattr "system.posix_acl_access" and SELinux context in xattr 
"security.selinux".

--
keywords: +easy
nosy: +christian.heimes
stage:  -> needs patch
versions: +Python 3.10 -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30044] shutil.copystat should (allow to) copy ownership, and other attributes

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

This could be achieved at least on Windows with CopyFileEx [1] and on OSX with 
copyfile(3) + COPYFILE_ALL which copies ACLs (but not users/groups). These are 
(were, in case of CopyFileEx) exposed in 
https://github.com/python/cpython/pull/7160/. Such a new functionality would 
probably deserve a separate copy3() function but it would serve OSX and Windows 
only. I'm not sure how things would work on other POSIX platforms. 

[1] 
https://msdn.microsoft.com/en-us/library/windows/desktop/aa363852(v=vs.85).aspx
[2] http://www.manpagez.com/man/3/copyfile/

--
nosy: +giampaolo.rodola

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30044] shutil.copystat should (allow to) copy ownership, and other attributes

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
versions:  -Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30044] shutil.copystat should (allow to) copy ownership, and other attributes

[Florent Coriat]

New submission from Florent Coriat:

shutil.copystat() copies permissions, timestamps and even flags and xattrs (if 
supported), but not ownership.
Furthermore, shutil.copy2() documentation until 2.7 used to say it behaves like 
cp -p, which preserves ownership, and not xattr nor flags. (On my system it 
silently fails to copy ownership when not root).

It may not be related, but comments in source code for the except 
NotImplementedError block concerning chmod mistakenly mentions chown-related 
functions.

I think copystat (and copy2) should at least provide an option to preserve 
ownership.
I do not know if it currently preserves SELinux context and ACL, but if not, it 
may also allow it.
It would be really useful for replication or backup applications to have a 
function that copies everything it can.

--
components: Library (Lib)
messages: 291499
nosy: noctiflore
priority: normal
severity: normal
status: open
title: shutil.copystat should (allow to) copy ownership, and other attributes
type: enhancement
versions: Python 3.5, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com