Re: How to process a very large (4Gb) tarfile from python?

On Thu, Jul 17, 2008 at 11:41:50PM -0700, Uwe Schmitt wrote: > On 17 Jul., 22:21, Lars Gustäbel <[EMAIL PROTECTED]> wrote: > > > > > Maybe we should post this issue to python-dev mailing list. > > > Parsing large tar-files is not uncommon. > > > > This issue is known and was fixed for Python 3.0,

Re: How to process a very large (4Gb) tarfile from python?

Due to the discussion above I wrote a python module for scanning of large tarfiles. You can get it from http://www.procoders.net/wp-content/tarfile_scanner.zip Greetings, Uwe -- http://mail.python.org/mailman/listinfo/python-list

Re: How to process a very large (4Gb) tarfile from python?

On 17 Jul., 22:21, Lars Gustäbel <[EMAIL PROTECTED]> wrote: > > > Maybe we should post this issue to python-dev mailing list. > > Parsing large tar-files is not uncommon. > > This issue is known and was fixed for Python 3.0, > seehttp://bugs.python.org/issue2058. The proposed patch does not avoid

Re: How to process a very large (4Gb) tarfile from python?

On Thu, Jul 17, 2008 at 10:39:23AM -0700, Uwe Schmitt wrote: > On 17 Jul., 17:55, Terry Carroll <[EMAIL PROTECTED]> wrote: > > On Thu, 17 Jul 2008 06:14:45 -0700 (PDT), Uwe Schmitt > > > > <[EMAIL PROTECTED]> wrote: > > >I had a look at tarfile.py in my current Python 2.5 installations > > >lib pat

Re: How to process a very large (4Gb) tarfile from python?

On 17 Jul., 17:55, Terry Carroll <[EMAIL PROTECTED]> wrote: > On Thu, 17 Jul 2008 06:14:45 -0700 (PDT), Uwe Schmitt > > <[EMAIL PROTECTED]> wrote: > >I had a look at tarfile.py in my current Python 2.5 installations > >lib path. The iterator caches TarInfo objects in a list > >tf.members . If you o

Re: How to process a very large (4Gb) tarfile from python?

On Thu, 17 Jul 2008 06:14:45 -0700 (PDT), Uwe Schmitt <[EMAIL PROTECTED]> wrote: >I had a look at tarfile.py in my current Python 2.5 installations >lib path. The iterator caches TarInfo objects in a list >tf.members . If you only want to iterate and you are not interested >in more functionallity

Re: How to process a very large (4Gb) tarfile from python?

On 17 Jul., 10:01, Terry Carroll <[EMAIL PROTECTED]> wrote: > I am trying to do something with a very large tarfile from within > Python, and am running into memory constraints.  The tarfile in > question is a 4-gigabyte datafile from > freedb.org,http://ftp.freedb.org/pub/freedb/, and has about 2

How to process a very large (4Gb) tarfile from python?

I am trying to do something with a very large tarfile from within Python, and am running into memory constraints. The tarfile in question is a 4-gigabyte datafile from freedb.org, http://ftp.freedb.org/pub/freedb/ , and has about 2.5 million members in it. Here's a simple toy program that just go