Re: memory error with zipfile module

2006-06-21 Thread Hari Sekhon
On 20/05/06, Bruno Desthuilliers [EMAIL PROTECTED] wrote: Roger Miller a écrit : The basic problem is that the zipfile interface only reads and writes whole files, so it may perform poorly or fail on huge files. At one time I implemented a patch to allow reading files in chunks. However I believe

Re: memory error with zipfile module

2006-06-21 Thread Fredrik Lundh
Hari Sekhon wrote: Is it me or is having to use os.system() all the time symtomatic of a deficiency/things which are missing from python as a language? it's you. /F -- http://mail.python.org/mailman/listinfo/python-list

Re: memory error with zipfile module

2006-06-21 Thread Hari Sekhon
Fredrik Lundh wrote: Hari Sekhon wrote: Is it me or is having to use os.system() all the time symtomatic of a deficiency/things which are missing from python as a language? it's you. /F I take it that it's still a work in progress to be able to pythonify

Re: memory error with zipfile module

2006-06-21 Thread Fredrik Lundh
Hari Sekhon wrote: I take it that it's still a work in progress to be able to pythonify everything, and until then we're just gonna have to rely on shell and those great C coded coreutils and stuff like that. Ok, I'm rather fond of Bash+coreutils, highest ratio of code lines to work I've

Re: memory error with zipfile module

2006-06-21 Thread Hari Sekhon
Fredrik Lundh wrote: Hari Sekhon wrote: I take it that it's still a work in progress to be able to pythonify everything, and until then we're just gonna have to rely on shell and those great C coded coreutils and stuff like that. Ok, I'm rather fond of Bash+coreutils, highest

Re: memory error with zipfile module

2006-06-21 Thread Fredrik Lundh
Hari Sekhon wrote: I've seen people using everything from zip to touch, either out of laziness or out of the fact it wouldn't work very well in python, this zip case is a good example. so based on a limitation in one library, and some random code you've seen on the internet, you're making

memory error with zipfile module

2006-05-19 Thread Hari Sekhon
I do import zipfile zip=zipfile.ZipFile('d:\somepath\cdimage.zip') zip.namelist() ['someimage.iso'] then either of the two: A) file('someimage.iso','w').write(zip.read('someimage.iso')) or B) content=zip.read('someimage.iso') but both result in the same error: Traceback (most recent call

Re: memory error with zipfile module

2006-05-19 Thread Ganesan Rajagopal
Hari Sekhon [EMAIL PROTECTED] writes: Traceback (most recent call last): File stdin, line 1, in ? File D:\u\Python24\lib\zipfile.py, line 357, in read bytes = dc.decompress(bytes) MemoryError Looks like the .iso file is huge. Even if it's only a CD image (approx 650MB), reading it

Re: memory error with zipfile module

2006-05-19 Thread bruno at modulix
Hari Sekhon wrote: I do import zipfile zip=zipfile.ZipFile('d:\somepath\cdimage.zip') zip.namelist() ['someimage.iso'] then either of the two: A) file('someimage.iso','w').write(zip.read('someimage.iso')) or B) content=zip.read('someimage.iso') but both result in the same

Re: memory error with zipfile module

2006-05-19 Thread [EMAIL PROTECTED]
Take a look at the pywin32 extension, which I believe has some lower level memory allocation and file capabilities that might help you in this situation. If I'm completely wrong, someone please tell me XD. Of course, you could just make the read() a step process, reading, O lets say 8192 bytes at

Re: memory error with zipfile module

2006-05-19 Thread bruno at modulix
[EMAIL PROTECTED] wrote: Take a look at the pywin32 extension, which I believe has some lower level memory allocation and file capabilities that might help you in this situation. But then the solution would not be portable, which would be a shame since the zlib module (on which ZipFile relies

Re: memory error with zipfile module

2006-05-19 Thread Sion Arrowsmith
Hari Sekhon [EMAIL PROTECTED] wrote: import zipfile zip=zipfile.ZipFile('d:\somepath\cdimage.zip') zip.namelist() ['someimage.iso'] [ ... ] B) content=zip.read('someimage.iso') Traceback (most recent call last): File stdin, line 1, in ? File D:\u\Python24\lib\zipfile.py, line 357, in read

Re: memory error with zipfile module

2006-05-19 Thread Sion Arrowsmith
bruno at modulix [EMAIL PROTECTED] wrote: http://mail.zope.org/pipermail/zope/2004-October/153882.html MemoryError is raised by Python when an underlying (OS-level) allocation fails. (...) Normally this would mean that you were out of even virtual memory (swap), but it could also be a symptom of

Re: memory error with zipfile module

2006-05-19 Thread bruno at modulix
Sion Arrowsmith wrote: Hari Sekhon [EMAIL PROTECTED] wrote: (snip) The python zipfile module is obviously broken... This isn't at all obvious to me. zipfile.read() does not seem to take full advantage of zlib's decompressobj's features. This could perhaps be improved (left as an exercice to

Re: memory error with zipfile module

2006-05-19 Thread Roger Miller
The basic problem is that the zipfile interface only reads and writes whole files, so it may perform poorly or fail on huge files. At one time I implemented a patch to allow reading files in chunks. However I believe that the current interface has too many problems to solve by incremental

Re: memory error with zipfile module

2006-05-19 Thread Bruno Desthuilliers
Roger Miller a écrit : The basic problem is that the zipfile interface only reads and writes whole files, so it may perform poorly or fail on huge files. At one time I implemented a patch to allow reading files in chunks. However I believe that the current interface has too many problems to