[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Éric Araujo added the comment: New changeset b08c48e61745666df9aeee97d8bddbf1e5550627 by Anthony Sottile in branch 'main': bpo-33671 fix orphaned comment in shutil.copyfileobj (GH-27516) https://github.com/python/cpython/commit/b08c48e61745666df9aeee97d8bddbf1e5550627 -- nosy:

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Change by Anthony Sottile : -- nosy: +Anthony Sottile nosy_count: 19.0 -> 20.0 pull_requests: +26032 pull_request: https://github.com/python/cpython/pull/27516 ___ Python tracker

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

STINNER Victor added the comment: > Yes, it's deliberate, see PR-12016. "I decided not to document `shutil.COPY_BUFSIZE` because I consider it a corner case." Ok. I have no opinion on that, I just wanted to ask the question :-) -- ___ Python

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Giampaolo Rodola' added the comment: Yes, it's deliberate, see PR-12016. -- ___ Python tracker ___ ___ Python-bugs-list mailing

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

STINNER Victor added the comment: shutil.COPY_BUFSIZE isn't documented. Is it a deliberate choice? https://docs.python.org/dev/library/shutil.html -- ___ Python tracker ___

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Inada Naoki added the comment: New changeset 3b0abb019662e42070f1d6f7e74440afb1808f03 by Inada Naoki (Giampaolo Rodola) in branch 'master': bpo-33671: allow setting shutil.copyfile() bufsize globally (GH-12016) https://github.com/python/cpython/commit/3b0abb019662e42070f1d6f7e74440afb1808f03

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Change by Giampaolo Rodola' : -- pull_requests: -7529 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Change by Giampaolo Rodola' : -- pull_requests: -7536 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Change by Giampaolo Rodola' : -- pull_requests: -7528 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Change by Giampaolo Rodola' : -- pull_requests: +12047 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Change by desbma : -- nosy: -desbma ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Change by hervé : -- pull_requests: +7536 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Change by STINNER Victor : -- pull_requests: +7528 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Change by STINNER Victor : -- pull_requests: +7529 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

STINNER Victor added the comment: New changeset 937ee9e745d7ff3c2010b927903c0e2a83623324 by Victor Stinner in branch 'master': Revert "bpo-33671: Add support.MS_WINDOWS and support.MACOS (GH-7800)" (GH-7919) https://github.com/python/cpython/commit/937ee9e745d7ff3c2010b927903c0e2a83623324

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Change by Giampaolo Rodola' : -- pull_requests: -7403 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Change by Giampaolo Rodola' : -- pull_requests: -7525 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Change by Giampaolo Rodola' : -- pull_requests: -7482 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Change by STINNER Victor : -- pull_requests: +7525 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Change by Roundup Robot : -- pull_requests: +7482 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

STINNER Victor added the comment: New changeset 8fbbdf0c3107c3052659e166f73990b466eacbb0 by Victor Stinner in branch 'master': bpo-33671: Add support.MS_WINDOWS and support.MACOS (GH-7800) https://github.com/python/cpython/commit/8fbbdf0c3107c3052659e166f73990b466eacbb0 --

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Change by STINNER Victor : -- pull_requests: +7403 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Giampaolo Rodola' added the comment: New changeset c7f02a965936f197354d7f4e6360f4cfc86817ed by Giampaolo Rodola in branch 'master': bpo-33671 / shutil.copyfile: use memoryview() with dynamic size on Windows (#7681)

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

STINNER Victor added the comment: > Thanks Gianpaolo for pushing for this. Great job. I concur: great job! Cool optimization. -- ___ Python tracker ___

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Marcos Dione added the comment: Thanks Gianpaolo for pushing for this. Great job. -- ___ Python tracker ___ ___ Python-bugs-list

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Change by Giampaolo Rodola' : -- pull_requests: +7293 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Change by Yury Selivanov : -- nosy: -yselivanov ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Giampaolo Rodola' added the comment: For future reference, as per https://github.com/python/cpython/pull/7160 discussion, we decided not to use CopyFileEx on Windows and instead increase read() buffer size from 16KB to 1MB (Windows only) resulting in a 40.8% speedup (instead of 48%). Also

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Giampaolo Rodola' added the comment: New changeset 4a172ccc739065bb658c75e8929774a8e94af9e9 by Giampaolo Rodola in branch 'master': bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) (#7160)

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Giampaolo Rodola' added the comment: Yes, I tried changing benchmarks order and zero-copy variants are always faster. As for instantaneous CoW copy, it is debatable. E.g. "cp" command does not do it by default:

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Марк Коренберг added the comment: http://man7.org/linux/man-pages/man2/ioctl_ficlonerange.2.html That possibly should be used under Linux in order to really acheive zero-copying. Just like modern cp command. -- nosy: +socketpair ___ Python tracker

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Stefan Behnel added the comment: Regarding the benchmarks, just to be sure, did you try reversing the run order to make sure you don't get unfair caching effects for the later runs? -- ___ Python tracker

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Stefan Behnel added the comment: Nice, I really like this. Apart from the usual bit of minor style issues, I couldn't see anything inherently wrong with the PR, but I'll leave the detailed reviews to those who'd have to maintain the code in the future. :) -- nosy: +scoder

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Change by Giampaolo Rodola' : -- nosy: +SilentGhost, StyXman, asvetlov, desbma, facundobatista, gps, josh.r, martin.panter, ncoghlan, neologix, petr.viktorin, pitrou, python-dev, r.david.murray, tarek, vstinner, yselivanov ___

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Giampaolo Rodola' added the comment: PR: https://github.com/python/cpython/pull/7160 -- ___ Python tracker ___

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Change by Giampaolo Rodola' : -- pull_requests: +6795 ___ Python tracker ___ ___

[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)

Change by Giampaolo Rodola' : -- title: Efficient efficient zero-copy syscalls for shutil.copy* functions (Linux, OSX and Win) -> Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) ___ Python tracker