I'm converting a Perl system to Python, and have run into a severe performance problem with pickle.
One facet of the system involves scanning and loading into memory a couple of parallel directory trees containing OTO 10^4 files. The trees don't change during development/testing and the scan takes 30-40 seconds, so to save time I cache the loaded tree structure to disk, in Perl with module Storable, and in Python with pickle. In Perl, the save operation produces a file of about 3MB, and both save and restore take a second or two. In Python, pickle.dump() produces a similar-size file but takes 20 seconds, and pickle.load() takes 45 seconds, which is actually LONGER than the time required to scan the directory trees. Is there anything I can do to speed up pickle.load() to get performance comparable to Perl's Storable? -- http://mail.python.org/mailman/listinfo/python-list