New submission from asnakelover <a3277...@uggsrock.com>:
It's not a big problem because we can just shell to zcat but compare
these two:
$ time python ./b.py >/dev/null
real 0m10.977s
user 0m7.128s
sys 0m0.888s
$ time python ./a.py >/dev/null
real 1m19.015s
user 1m18.185s
sys 0m0.072s
$ # Notice that the gzip module (a.py) had the benefit of the files
being in a disk cache by too now...
$ cat a.py
import gzip
import os
apt_cache_dir = "/var/cache/apt/apt-file"
for apt_cache_file in os.listdir(apt_cache_dir):
if not apt_cache_file.endswith(".gz"):
continue
f = gzip.open(os.path.join(apt_cache_dir, apt_cache_file))
for line in f:
print line
$ cat b.py
import os
import subprocess
from cStringIO import StringIO
apt_cache_dir = "/var/cache/apt/apt-file"
for apt_cache_file in os.listdir(apt_cache_dir):
if not apt_cache_file.endswith(".gz"):
continue
p = subprocess.Popen(["zcat", os.path.join(apt_cache_dir,
apt_cache_file)],
stdout = subprocess.PIPE)
f = StringIO(p.communicate()[0])
assert p.returncode == 0
for line in f:
print line
Also tried this one just for "completeness":
$ cat c.py
import gzip
import os
from cStringIO import StringIO
apt_cache_dir = "/var/cache/apt/apt-file"
for apt_cache_file in os.listdir(apt_cache_dir):
if not apt_cache_file.endswith(".gz"):
continue
f = gzip.open(os.path.join(apt_cache_dir, apt_cache_file))
f = StringIO(f.read())
for line in f:
print line
But after it had ran (with some thrashing) for 3 and a half minutes I
killed it.
----------
components: Library (Lib)
messages: 96204
nosy: asnakelover
severity: normal
status: open
title: gzip module too slow
type: performance
versions: Python 2.6
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue7471>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
