Re: [Python-Dev] os.walk() is going to be *fast* with scandir

2014-08-18 Thread Robert Collins
Indeed - my suggestion is applicable to people using the library -Rob On 10 Aug 2014 18:21, "Larry Hastings" wrote: > On 08/09/2014 10:40 PM, Robert Collins wrote: > > A small tip from my bzr days - cd into the directory before scanning it > > > I doubt that's permissible for a library function

Re: [Python-Dev] os.walk() is going to be *fast* with scandir

2014-08-11 Thread Ben Hoyt
> Victor Stinner suggested [1] to allow scandir(fd) but I don't see it > being mentioned in the pep 471 [2]: it neither supports nor rejects the > idea. > > [1] https://mail.python.org/pipermail/python-dev/2014-July/135283.html > [2] http://legacy.python.org/dev/peps/pep-0471/ Yes, listdir() suppo

Re: [Python-Dev] os.walk() is going to be *fast* with scandir

2014-08-11 Thread Akira Li
Armin Rigo writes: > On 10 August 2014 08:11, Larry Hastings wrote: >>> A small tip from my bzr days - cd into the directory before scanning it >> >> I doubt that's permissible for a library function like os.scandir(). > > Indeed, chdir() is notably not compatible with multithreading. There > w

Re: [Python-Dev] os.walk() is going to be *fast* with scandir

2014-08-10 Thread R. David Murray
On Sun, 10 Aug 2014 13:57:36 +1000, Nick Coghlan wrote: > On 10 August 2014 13:20, Antoine Pitrou wrote: > > Le 09/08/2014 12:43, Ben Hoyt a écrit : > > > >> Just thought I'd share some of my excitement about how fast the all-C > >> version [1] of os.scandir() is turning out to be. > >> > >> Bel

Re: [Python-Dev] os.walk() is going to be *fast* with scandir

2014-08-10 Thread Armin Rigo
Hi Larry, On 10 August 2014 08:11, Larry Hastings wrote: >> A small tip from my bzr days - cd into the directory before scanning it > > I doubt that's permissible for a library function like os.scandir(). Indeed, chdir() is notably not compatible with multithreading. There would be a non-portab

Re: [Python-Dev] os.walk() is going to be *fast* with scandir

2014-08-09 Thread Larry Hastings
On 08/09/2014 10:40 PM, Robert Collins wrote: A small tip from my bzr days - cd into the directory before scanning it I doubt that's permissible for a library function like os.scandir(). //arry/ ___ Python-Dev mailing list Python-Dev@python.org http

Re: [Python-Dev] os.walk() is going to be *fast* with scandir

2014-08-09 Thread Robert Collins
A small tip from my bzr days - cd into the directory before scanning it - especially if you'll end up statting more than a fraction of the files, or are recursing - otherwise the VFS does a traversal for each path you directly stat / recurse into. This can become a dominating factor in some workloa

Re: [Python-Dev] os.walk() is going to be *fast* with scandir

2014-08-09 Thread Nick Coghlan
On 10 August 2014 13:20, Antoine Pitrou wrote: > Le 09/08/2014 12:43, Ben Hoyt a écrit : > >> Just thought I'd share some of my excitement about how fast the all-C >> version [1] of os.scandir() is turning out to be. >> >> Below are the results of my scandir / walk benchmark run with three >> diff

Re: [Python-Dev] os.walk() is going to be *fast* with scandir

2014-08-09 Thread Antoine Pitrou
Le 09/08/2014 12:43, Ben Hoyt a écrit : Just thought I'd share some of my excitement about how fast the all-C version [1] of os.scandir() is turning out to be. Below are the results of my scandir / walk benchmark run with three different versions. I'm using an SSD, which seems to make it especia

[Python-Dev] os.walk() is going to be *fast* with scandir

2014-08-09 Thread Ben Hoyt
Just thought I'd share some of my excitement about how fast the all-C version [1] of os.scandir() is turning out to be. Below are the results of my scandir / walk benchmark run with three different versions. I'm using an SSD, which seems to make it especially faster than listdir / walk. Note that