[issue10261] tarfile iterator without members caching

Lars Gustäbel added the comment: Closing after six years of inactivity. -- resolution: -> wont fix stage: -> resolved status: open -> closed ___ Python tracker

[issue10261] tarfile iterator without members caching

Lars Gustäbel l...@gustaebel.de added the comment: There is no trivial or backwards-compatible solution to this problem. The way it is now, there is no alternative to storing all TarInfo objects: there is no central table of contents in an archive we could use, so we must create our own. In

[issue10261] tarfile iterator without members caching

New submission from Karsten Wolf karste...@web.de: It would be helpful to have a tarfile iterator that does not cache every archive member encountered. This makes it nearly impossible to iterate over an archive with millions of files. -- components: Library (Lib) messages: 120041

[issue10261] tarfile iterator without members caching

Lars Gustäbel l...@gustaebel.de added the comment: I assume you're using Python 2.x. because tarfile's memory footprint was significantly reduced in Python 3.0, see the patch in issue2058 and r62337. This patch was not backported to the 2.x branch back then. As the 2.x branch has been closed

[issue10261] tarfile iterator without members caching

Karsten Wolf karste...@web.de added the comment: Yes, I'm on 2.6. I checked the Python 3.x tarfile just for this one line in TarFile.next(): self.members.append(tarinfo) to conclude it would have the same problem. Reducing 2.5gb memory usage as measured in my particular case by 60%, still

[issue10261] tarfile iterator without members caching

Changes by Antoine Pitrou pit...@free.fr: -- type: feature request - resource usage versions: +Python 2.7, Python 3.1, Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10261 ___