> Using a FIFO queue instead of a LIFO stack for parsed URLs could solve the > problem. But I haven't looked at the source - I'm not really comfortable with > Python ;-)
FIFO also makes sense with the way people read documents. This way, a maximum db size restriction could be added to stop the spider process once a certain amount of data has been gathered. The first links on the page would be added but the rest would be inaccessable. This would be useful for novels which link to chapters and especially for all those news sites that list the articles in reverse-chronological order. Hmm.. but then again, there is no maximum db size restriction :) Maybe I should just go back to sleep and enjoy the remaining 2 hours of nighttime. It was just a thought anyway -Greg _______________________________________________ plucker-list mailing list [EMAIL PROTECTED] http://lists.rubberchicken.org/mailman/listinfo/plucker-list
