New submission from Giampaolo Rodola' <g.rod...@gmail.com>: This is a follow up of #25063 and similar to socket.sendfile() (#17552). It provides a 20/25% speedup when copying files with shutil.copyfile(), shutil.copy() and shutil.copy2(). Differently from #25063 this is used for filesystem files only and copyfileobj() is left alone.
Unmerged #26826 is also related to this. I applied #26826 patch and built a wrapper around copy_file_range() and the speedup is basically the same. Nevertheless, even when #26826 gets merged it probably makes sense to rely on sendfile() in case copy_file_range() is not available (it was introduced in 2016) or the UNIX platform supports file-to-file copy via sendfile(2) (even though I'm not aware of any, so this would basically be Linux-only). Some benchmarks: $ dd if=/dev/urandom of=/tmp/f1 bs=1K count=128 $ time ./python -m timeit -s 'import shutil; p1 = "/tmp/f1"; p2 = "/tmp/f2"' 'shutil.copyfile(p1, p2)' 128K copy ========= --- without patch: 2000 loops, best of 5: 160 usec per loop real 0m2.353s user 0m0.454s sys 0m1.435s --- with patch: 2000 loops, best of 5: 187 usec per loop real 0m2.724s user 0m0.627s sys 0m1.634s 8MB copy ======== $ dd if=/dev/urandom of=/tmp/f1 bs=1M count=8 --- without patch: 50 loops, best of 5: 9.51 msec per loop real 0m3.392s user 0m0.343s sys 0m2.478s --- with patch: 50 loops, best of 5: 7.75 msec per loop real 0m2.878s user 0m0.105s sys 0m2.187s 512MB copy ========== --- without patch: 1 loop, best of 5: 872 msec per loop real 0m5.574s user 0m0.402s sys 0m3.115s --- with patch: 1 loop, best of 5: 646 msec per loop real 0m5.475s user 0m0.037s sys 0m2.959s ---------- components: Library (Lib) messages: 317611 nosy: StyXman, desbma, facundobatista, giampaolo.rodola, martin.panter, ncoghlan, neologix, petr.viktorin, python-dev, r.david.murray, vstinner priority: normal severity: normal stage: patch review status: open title: Use high-performance os.sendfile() in shutil.copy* type: performance versions: Python 3.8 _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33639> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com