[issue26826] Expose new copy_file_range() syscall in os module.

Change by Benjamin Peterson : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue26826] Expose new copy_file_range() syscall in os module.

Giampaolo Rodola' added the comment: > Please open a new issue to discuss how it can used in shutil ;-) Use copy_file_range() in shutil.copyfile(): https://bugs.python.org/issue37159 -- ___ Python tracker

[issue26826] Expose new copy_file_range() syscall in os module.

STINNER Victor added the comment: I created bpo-37157: "shutil: add reflink=False to file copy functions to control clone/CoW copies (use copy_file_range)". -- The new os.copy_file_range() should be documented at: https://docs.python.org/dev/whatsnew/3.8.html#os --

[issue26826] Expose new copy_file_range() syscall in os module.

STINNER Victor added the comment: Please open a new issue to discuss how it can used in shutil ;-) -- ___ Python tracker ___ ___

[issue26826] Expose new copy_file_range() syscall in os module.

Giampaolo Rodola' added the comment: Actually "man copy_file_range" claims it can do server-side copy, meaning no network traffic between client and server if *src* and *dst* live on the same network fs. So I agree copy_file_range() should be preferred over sendfile() after all. =) I have a

[issue26826] Expose new copy_file_range() syscall in os module.

Giampaolo Rodola' added the comment: I think data deduplication / CoW / reflink copy is better implemented via FICLONE. "cp --reflink" uses it, I presume because it's older than copy_file_range(). I have a working patch adding CoW copy support for Linux and OSX (but not Windows). I think

[issue26826] Expose new copy_file_range() syscall in os module.

STINNER Victor added the comment: Giampaolo Rodola': > shutil.copyfile() already uses sendfile() which basically provides the same > performances. sendfile() should be preferred though because it’s supported > since Linux 2.6.33. Pablo Galindo Salgado: > But copy_file_rane can leverage more

[issue26826] Expose new copy_file_range() syscall in os module.

Pablo Galindo Salgado added the comment: But copy_file_rane can leverage more filesystem features like deduplication and copy offload stuff. -- ___ Python tracker ___

[issue26826] Expose new copy_file_range() syscall in os module.

Giampaolo Rodola' added the comment: shutil.copyfile() already uses sendfile() which basically provides the same performances. sendfile() should be preferred though because it’s supported since Linux 2.6.33. -- ___ Python tracker

[issue26826] Expose new copy_file_range() syscall in os module.

STINNER Victor added the comment: shutil copy functions would definitively benefit of using copy_file_range() if available. Can someone please open a separated issue for shutil? -- ___ Python tracker

[issue26826] Expose new copy_file_range() syscall in os module.

Pablo Galindo Salgado added the comment: New changeset aac4d0342c3e692731c189d003dbd73a8c681a34 by Pablo Galindo in branch 'master': bpo-26826: Expose copy_file_range in the os module (GH-7255) https://github.com/python/cpython/commit/aac4d0342c3e692731c189d003dbd73a8c681a34 --

[issue26826] Expose new copy_file_range() syscall in os module.

Giampaolo Rodola' added the comment: Little update about this. According to: http://man7.org/linux/man-pages/man2/copy_file_range.2.html ...it seems glibc 2.27 released on 2018-02-01 includes copy_file_range(). I'm not the best candidate for giving advice on C syscalls

[issue26826] Expose new copy_file_range() syscall in os module.

Change by Pablo Galindo Salgado : -- pull_requests: +6884 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue26826] Expose new copy_file_range() syscall in os module.

Giampaolo Rodola' added the comment: Check for availability in configure.ac appears to be broken. -- ___ Python tracker ___

[issue26826] Expose new copy_file_range() syscall in os module.

Marcos Dione added the comment: I'm really sorry, but I don't have time to continue with this (new daughter!). Can someone else pick it up? -- ___ Python tracker

[issue26826] Expose new copy_file_range() syscall in os module.

STINNER Victor added the comment: > As for the patch attached to this ticket, is there anything missing in order > to push it forward? IMHO the next step would be to create a pull request on GitHub. -- ___ Python tracker

[issue26826] Expose new copy_file_range() syscall in os module.

Giampaolo Rodola' added the comment: This is a great addition. I have a working patch adding sendfile() support for shutil.copyfileobj() which speeds it up by a factor of 1.3x on Linux. According to this

[issue26826] Expose new copy_file_range() syscall in os module.

Change by Giampaolo Rodola' : -- nosy: +giampaolo.rodola ___ Python tracker ___ ___

[issue26826] Expose new copy_file_range() syscall in os module.

Changes by desbma : -- nosy: +desbma ___ Python tracker ___ ___ Python-bugs-list mailing

[issue26826] Expose new copy_file_range() syscall in os module.

Nick Coghlan added the comment: Victor, due to the mention of getrandom() early on, this came up when I was looking for issues that could potentially be closed based on the PEP 524 implementation landing. Any chance you or someone else could take another look during the pre-beta sprint this

[issue26826] Expose new copy_file_range() syscall in os module.

Marcos Dione added the comment: Fixed extra space and semicolon (?!?!). Also, I'm getting 500 errors from riedvelt when I try to reply to comments. -- Added file: http://bugs.python.org/file43688/copy_file_range.diff ___ Python tracker

[issue26826] Expose new copy_file_range() syscall in os module.

Marcos Dione added the comment: Another version: * Changed availability to kernel type, version and date. -- Added file: http://bugs.python.org/file43640/copy_file_range.diff ___ Python tracker

[issue26826] Expose new copy_file_range() syscall in os module.

Marcos Dione added the comment: New version: * Adds a new test for offset parameters. -- Added file: http://bugs.python.org/file43639/copy_file_range.diff ___ Python tracker

[issue26826] Expose new copy_file_range() syscall in os module.

Facundo Batista added the comment: It looked ok to me (I couldn't try it, as I still have 4.4 kernel). One thing to the be done is to improve the test coverage (trying the usage of all the parameters, at least). -- nosy: +facundobatista ___ Python

[issue26826] Expose new copy_file_range() syscall in os module.

Marcos Dione added the comment: * Updated the patch to latest version. * PEP-8'ed the tests. * Dropped flags from the API but not the internal function. -- Added file: http://bugs.python.org/file43624/copy_file_range.diff ___ Python tracker

[issue26826] Expose new copy_file_range() syscall in os module.

Changes by Christian Heimes : -- nosy: -christian.heimes ___ Python tracker ___ ___

[issue26826] Expose new copy_file_range() syscall in os module.

Marcos Dione added the comment: ENOSYS catching fixed. -- Added file: http://bugs.python.org/file43319/copy_file_range.diff ___ Python tracker ___

[issue26826] Expose new copy_file_range() syscall in os module.

Martin Panter added the comment: It’s a bit ugly, but I would write the test so that it is recorded as skipped: try: os.copy_file_range(...) except OSError as err: if err.errno != ENOSYS: raise # We get to see the full exception details self.skipTest(err) # Test is

[issue26826] Expose new copy_file_range() syscall in os module.

Marcos Dione added the comment: Fixed the last comments, including comparing what was written to the original data, but only to the length of what was actually written. I'm just not sure if the way to handle the syscall not existing is ok. -- Added file:

[issue26826] Expose new copy_file_range() syscall in os module.

Marcos Dione added the comment: I added a couple of unit tests, which lead me to fix a couple of bugs (yay!). I discarded the idea of removing any reference to flags. -- Added file: http://bugs.python.org/file43310/copy_file_range.diff ___ Python

[issue26826] Expose new copy_file_range() syscall in os module.

Martin Panter added the comment: Another option could be to use Serhiy’s proposed partial keywords support in Issue 26282. It is not yet committed, but it looks like it will go ahead, and then you could make “in” and “out” positional-only parameters, and the rest keywords. But IMO “src” and

[issue26826] Expose new copy_file_range() syscall in os module.

Marcos Dione added the comment: I settled for s/in/src/ and s/out/dst/, fixed typos, made sure the docs are in sync (parameters in the docstring were out of order), rephrased paragraph about flags parameter and included the configure.ac code for detecting availability of the syscall. I'm also

[issue26826] Expose new copy_file_range() syscall in os module.

Martin Panter added the comment: There’s still something funny about your patches: the last one has a bit of configure script at the end of the posixmodule.c diff. One other thing I thought of: “in” is not a practical keyword argument name in Python, because it is a reserved word. Yes,

[issue26826] Expose new copy_file_range() syscall in os module.

Marcos Dione added the comment: Sorry for the delay. Based on suggestions in the mailing list, I changed the *count* handling as if it were a ssize_t, and I added a note about ignored output parameters. -- title: Expose new copy_file_range() syscal in os module. -> Expose new