Tempo wrote: > Why do you say that the bottleneck of the crawler will always be > downloading the page? Is it becasue there isn't already a modual to do > this and I will have to start from scratch? Or a bandwidth issue?
Because of bandwidth - not necessarily yours directly, but the maximum flow between your uplink and the site in question. It will always take at least a fractioin of a second up to several seconds until the data is there - in that time, lots of python code can run. Diez -- http://mail.python.org/mailman/listinfo/python-list