Claudio Grondi wrote: > What started as a simple test if it is better to load uncompressed data > directly from the harddisk or > load compressed data and uncompress it (Windows XP SP 2, Pentium4 3.0 GHz > system with 3 GByte RAM) > seems to show that none of the in Python available compression libraries > really works for large sized > (i.e. 500 MByte) strings. > > Test the provided code and see yourself. > > At least on my system: > zlib fails to decompress raising a memory error > pylzma fails to decompress running endlessly consuming 99% of CPU time > bz2 fails to compress running endlessly consuming 99% of CPU time > > The same works with a 10 MByte string without any problem. > > So what? Is there no compression support for large sized strings in Python?
you're probably measuring windows' memory managment rather than the com- pression libraries themselves (Python delegates all memory allocations >256 bytes to the system). I suggest using incremental (streaming) processing instead; from what I can tell, all three libraries support that. </F> -- http://mail.python.org/mailman/listinfo/python-list