New submission from Giampaolo Rodola' <g.rod...@gmail.com>: Patch in attachment uses platform specific zero-copy syscalls on Linux and Solaris (os.sendfile(2)), Windows (CopyFileW) and OSX (fcopyfile(2)) speeding up shutil.copyfile() and other functions using it (copy(), copy2(), copytree(), move()).
Average speedup for a 512MB file copy is +24% on Linux, +50% on OSX and +48% on Windows by copying file on the same partition (SSD disk was used). Follows some benchmarks. Setup ===== Create 128K, 8M, 512M file: $ python -c "import os; f = open('f1', 'wb'); f.write(os.urandom(128 * 1024))" $ python -c "import os; f = open('f1', 'wb'); f.write(os.urandom(8 * 1024 * 1024))" $ python -c "import os; f = open('f1', 'wb'); f.write(os.urandom(512 * 1024 * 1024))" Benchmark: $ time ./python -m timeit -s 'import shutil; p1 = "f1"; p2 = "f2"' 'shutil.copyfile(p1, p2)' Linux ===== 128K copy (+13%): without patch: 1000 loops, best of 5: 228 usec per loop real 0m1.756s user 0m0.386s sys 0m1.116s with patch: 1000 loops, best of 5: 198 usec per loop real 0m1.464s user 0m0.281s sys 0m0.958s 8MB copy (+24%): without patch: 50 loops, best of 5: 10.1 msec per loop real 0m2.703s user 0m0.316s sys 0m1.847s with patch: 50 loops, best of 5: 7.78 msec per loop real 0m2.447s user 0m0.086s sys 0m1.682s 512MB copy (+26%): without patch: 1 loop, best of 5: 872 msec per loop real 0m5.574s user 0m0.402s sys 0m3.115s with patch: 1 loop, best of 5: 646 msec per loop real 0m5.475s user 0m0.037s sys 0m2.959s OSX === 128K copy (+8.5%): without patch: 500 loops, best of 5: 508 usec per loop real 0m2.971s user 0m0.442s sys 0m2.168s with patch: 500 loops, best of 5: 464 usec per loop real 0m2.798s user 0m0.379s sys 0m2.031s 8MB copy (+67%): without patch: 20 loops, best of 5: 32.8 msec per loop real 0m3.672s user 0m0.357s sys 0m1.434s with patch: 20 loops, best of 5: 10.8 msec per loop real 0m1.860s user 0m0.079s sys 0m0.719s 512MB copy (+50%): without patch: 1 loop, best of 5: 953 msec per loop real 0m5.930s user 0m1.021s sys 0m4.835s with patch: 1 loop, best of 5: 480 msec per loop real 0m3.150s user 0m0.067s sys 0m2.740s Windows ======= 128K copy (+69%): without patch: 50 loops, best of 5: 6.45 msec per loop with patch: 50 loops, best of 5: 1.99 msec per loop 8M copy (+64%): without patch: 10 loops, best of 5: 22.6 msec per loop with patch: 50 loops, best of 5: 7.95 msec per loop 512M copy (+48%): without patch: 1 loop, best of 5: 1.21 sec per loop with patch: 1 loop, best of 5: 629 msec per loop ---------- components: Library (Lib) files: shutil-zero-copy.diff keywords: needs review, patch messages: 317878 nosy: giampaolo.rodola priority: normal severity: normal stage: patch review status: open title: Efficient efficient zero-copy syscalls for shutil.copy* functions (Linux, OSX and Win) type: performance versions: Python 3.8 Added file: https://bugs.python.org/file47621/shutil-zero-copy.diff _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33671> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com