[issue39907] `pathlib.Path.iterdir()` wastes memory by using `os.listdir()` rather than `os.scandir()`

Barney Gale added the comment: Thanks Rémi and Serhiy! Closing this ticket as the patch doesn't provide any sort of improvement. -- ___ Python tracker ___

[issue39907] `pathlib.Path.iterdir()` wastes memory by using `os.listdir()` rather than `os.scandir()`

Change by Barney Gale : -- resolution: -> not a bug stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue39907] `pathlib.Path.iterdir()` wastes memory by using `os.listdir()` rather than `os.scandir()`

Serhiy Storchaka added the comment: > Less reliable how? See issue39916. > so it looks like scandir as a small overhead when accumulating all results > and not using the extra info it returns. Try with larger directories. The difference may be not so small. $ python3 -m timeit -s 'from os

[issue39907] `pathlib.Path.iterdir()` wastes memory by using `os.listdir()` rather than `os.scandir()`

Rémi Lapeyre added the comment: This is not how timeit works, you just measured the time taken by an empty loop, you can look at `python3 -m timeit -h` to get help how to call it. I think a correct invocation would be: (venv) ➜ ~ python3 -m timeit -s 'from os import scandir'

[issue39907] `pathlib.Path.iterdir()` wastes memory by using `os.listdir()` rather than `os.scandir()`

Barney Gale added the comment: Less reliable how? Doesn't appear any slower: barney.gale@heilbron:~$ python3 -m timeit -s "import os; os.listdir('/usr/local')" 1 loops, best of 3: 0.0108 usec per loop barney.gale@heilbron:~$ python3 -m timeit -s "import os;

[issue39907] `pathlib.Path.iterdir()` wastes memory by using `os.listdir()` rather than `os.scandir()`

Serhiy Storchaka added the comment: It would be slower and less reliable implementation of os.listdir(). -- ___ Python tracker ___

[issue39907] `pathlib.Path.iterdir()` wastes memory by using `os.listdir()` rather than `os.scandir()`

Barney Gale added the comment: Ah, right you are! The globbing helpers call `list(os.scandir(...))` - perhaps we should do the same here? -- ___ Python tracker ___

[issue39907] `pathlib.Path.iterdir()` wastes memory by using `os.listdir()` rather than `os.scandir()`

Serhiy Storchaka added the comment: It is not so easy. There was reason why it was not done earlier. scandir() wastes more limited resource than memory -- file descriptors. It should also be properly closed and do not depend on the garbage collector. Consider the example: def

[issue39907] `pathlib.Path.iterdir()` wastes memory by using `os.listdir()` rather than `os.scandir()`

Karthikeyan Singaravelan added the comment: This optimisation was also hinted at https://bugs.python.org/issue26032#msg257653 -- nosy: +pitrou, serhiy.storchaka, xtreak ___ Python tracker

[issue39907] `pathlib.Path.iterdir()` wastes memory by using `os.listdir()` rather than `os.scandir()`

Change by Barney Gale : -- keywords: +patch pull_requests: +18223 stage: -> patch review pull_request: https://github.com/python/cpython/pull/18865 ___ Python tracker ___

[issue39907] `pathlib.Path.iterdir()` wastes memory by using `os.listdir()` rather than `os.scandir()`

New submission from Barney Gale : `pathlib.Path.iterdir()` uses `os.listdir()` rather than `os.scandir()`. I think this has a small performance cost, per PEP 471: > It returns a generator instead of a list, so that scandir acts as a true > iterator instead of returning the full list