[issue37935] Improve performance of pathlib.scandir()

Gregory P. Smith added the comment: New changeset 98a4a713d001cf2dfb04a9e318e4aea899bc8fbd by Gregory P. Smith (Miss Islington (bot)) in branch '3.8': bpo-37935: Added tests for os.walk(), glob.iglob() and Path.glob() (GH-15956) (GH-16043)

[issue37935] Improve performance of pathlib.scandir()

Change by miss-islington : -- pull_requests: +15665 pull_request: https://github.com/python/cpython/pull/16043 ___ Python tracker ___

[issue37935] Improve performance of pathlib.scandir()

Gregory P. Smith added the comment: New changeset f9dc2ad89032201427ed5f08061c703794627ad9 by Gregory P. Smith (Serhiy Storchaka) in branch 'master': bpo-37935: Added tests for os.walk(), glob.iglob() and Path.glob() (GH-15956)

[issue37935] Improve performance of pathlib.scandir()

Serhiy Storchaka added the comment: Yes, I am the author of the code that uses os.scandir() in os.walk(), os.fwalk(), glob.iglob() and Path.glob() (see issue23605, issue25996, issue25596, issue26032). And it was always in mind to not keep many file descriptors open when traverse directories

[issue37935] Improve performance of pathlib.scandir()

Change by Serhiy Storchaka : -- keywords: +patch pull_requests: +15591 pull_request: https://github.com/python/cpython/pull/15956 ___ Python tracker ___

[issue37935] Improve performance of pathlib.scandir()

Gregory P. Smith added the comment: For some context here, the pathlib scandir code was written by Serhiy in https://bugs.python.org/issue26032 and related commit https://github.com/python/cpython/commit/680cb152c5d220a74321fa905d4fc91bdec40fbb. > Any optimization can be accepted only when

[issue37935] Improve performance of pathlib.scandir()

Serhiy Storchaka added the comment: Any optimization can be accepted only when we have any prove that the change actually has measurable effect, and that it is large enough. Avoiding calling list() on the output of scandir() may be not harmless. For example it will left open a file

[issue37935] Improve performance of pathlib.scandir()

Serhiy Storchaka added the comment: The title is misleading. There is no pathlib.scandir(). -- ___ Python tracker ___ ___

[issue37935] Improve performance of pathlib.scandir()

Gregory P. Smith added the comment: Shai, please open the 'Your Details' link in the bugs.python.org sidebar and make sure you have your github username filled in. it needs to say ShaiAvr for our CLA automation to understand. Avoiding calling list() on the output of scandir() is always

[issue37935] Improve performance of pathlib.scandir()

Serhiy Storchaka added the comment: I think using the timeit module is enough. For more precise benchmarking you may need to use the pyperf module, but I think this is not a case. For example, something like: ./python -m timeit -s "from pathlib import Path; p = Patch('...')" "for x in

[issue37935] Improve performance of pathlib.scandir()

Shai added the comment: I'm new to contributing here. I've never done benchmarking before. I'd appreciate it if you could provide a guide to benchmarking. You could look at the changes I made in the pull request (PR 15331). They're easy to follow and I think that removing a useless call to

[issue37935] Improve performance of pathlib.scandir()

Serhiy Storchaka added the comment: Could you please provide any microbenchmarks that show the performance improvement? -- nosy: +serhiy.storchaka ___ Python tracker ___

[issue37935] Improve performance of pathlib.scandir()

Shai added the comment: >From the docs (https://docs.python.org/3/library/os.html#os.scandir.close): "This is called automatically when the iterator is exhausted or garbage collected, or when an error happens during iterating. However it is advisable to call it explicitly or use the with

[issue37935] Improve performance of pathlib.scandir()

hongweipeng added the comment: Scandir() will be close when it iteration is over.You can see ScandirIterator_iternext: ``` static PyObject * ScandirIterator_iternext(ScandirIterator *iterator) { while (1) { ... } /* Error or no more files */

[issue37935] Improve performance of pathlib.scandir()

Change by Brett Cannon : -- nosy: +brett.cannon ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue37935] Improve performance of pathlib.scandir()

Change by SilentGhost : -- nosy: +pitrou stage: -> patch review ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue37935] Improve performance of pathlib.scandir()

New submission from Shai : I recently have taken a look in the source code of the pathlib module, and I saw something weird there: when the module used the scandir function, it first converted its iterator into a list and then used it in a for loop. The list wasn't used anywhere else, so I