On 17 December 2012 15:28, Gilles Lenfant <gilles.lenf...@gmail.com> wrote: > I have googled but did not find an efficient solution to my problem. My > customer provides a directory with a huuuuge list of files (flat, potentially > 100000+) and I cannot reasonably use os.listdir(this_path) unless creating a > big memory footprint. > > So I'm looking for an iterator that yields the file names of a directory and > does not make a giant list of what's in. > > i.e : > > for filename in enumerate_files(some_directory): > # My cooking...
In the last couple of months there has been a lot of discussion (on python-list or python-dev - not sure) about creating a library to more efficiently iterate over the files in a directory. The result so far is this library on github: https://github.com/benhoyt/betterwalk It says there that """ Somewhat relatedly, many people have also asked for a version of os.listdir() that yields filenames as it iterates instead of returning them as one big list. So as well as a faster walk(), BetterWalk adds iterdir_stat() and iterdir(). They're pretty easy to use, but see below for the full API docs. """ Does that code work for you? If so, I imagine the author would be interested to get some feedback on how well it works. Alternatively, perhaps consider calling an external utility. Oscar -- http://mail.python.org/mailman/listinfo/python-list
