On Sun, Dec 27, 2009 at 11:30, [email protected] <[email protected]> wrote: > How hard would it be to set up a cron to tar up a daily snapshot so that the > initial download was quick (no API calls), then you'd only need an update > from the last snapshot?
Not hard, I think. I haven't done a complete download, but the script I made simply read in all the data in memory and then dumped the huge dictionary to a pickle. Updates would be done by loading the dictionary into memory again, and then updating that dictionary with whatever happened since the last time, and dumping it to a pickle again. :) The biggest problem with that technique is the memory usage, but I didn't see it go up significantly during my one-hour test run, so I think it is in fact feasible. If not I guess each package could be dumped into the pickle separately, but that would make updating more complicated. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 _______________________________________________ Distutils-SIG maillist - [email protected] http://mail.python.org/mailman/listinfo/distutils-sig
