Akira Li added the comment: > STINNER Victor added the comment: > >> scandir is slower on my machine: > > Please share more information about your config: OS, disk type (hard > drive, SSD, something else), filesystem, etc. > Ubuntu 14.04, SSD, ext4 filesystem. Results for different python versions are the same (scandir on pypy is even slower due to ctype usage).
What other information could be useful? > Ben Hoyt added the comment: > > Akira, note the "Using slower ctypes version of scandir" -- this is > the older, ctypes implementation of scandir. On Linux, depending on > file system etc, that can often be slower. You need to at least be > using the "fast C version" in _scandir.c, which is then half C, half > Python. But ideally you'd use the all-C version that I've provided as > a CPython 3.5 patch. Yes. I've noticed it, that is why I asked :) *"What commands should I run to benchmark the attached patch (scandir-2.patch)?"* I've read benchmark.py's source; It happens that it already supports benchmarking of os.scandir. In my python checkout: cpython$ hg import --no-commit https://bugs.python.org/file36963/scandir-2.patch cpython$ make && cd ../scandir/ scandir$ ../cpython/python benchmark.py -s /usr/ Using Python 3.5's builtin os.scandir() Comparing against builtin version of os.walk() Priming the system's cache... Benchmarking walks on /usr/, repeat 1/3... Benchmarking walks on /usr/, repeat 2/3... Benchmarking walks on /usr/, repeat 3/3... os.walk size 7925376343, scandir.walk size 5534939617 -- NOT EQUAL! os.walk took 13.552s, scandir.walk took 6.032s -- 2.2x as fast It seems os.walk and scandir.walk do a different work here. I've written get_tree_size_listdir_fd() that mimics get_tree_size() (see get_tree_size_listdir.diff patch) to get the same size: scandir$ ../cpython/python benchmark.py -s /usr Using Python 3.5's builtin os.scandir() Comparing against builtin version of os.walk() Comparing against get_tree_size_listdir_fd Priming the system's cache... Benchmarking walks on /usr, repeat 1/3... Benchmarking walks on /usr, repeat 2/3... Benchmarking walks on /usr, repeat 3/3... os.walk size 5534939617, scandir.walk size 5534939617 -- equal os.walk took 5.697s, scandir.walk took 5.621s -- 1.0x as fast scandir is not noticeably faster but scandir-based code is much nicer here. ---------- Added file: http://bugs.python.org/file37284/get_tree_size_listdir.diff _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue22524> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
