Hi, how much work would it take to convert wget into a fully fledged asynchronous webcrawler?
I was thinking something like using select. Ideally, I want to be able to supply wget with a list of starting point URLs and then for wget to crawl the web from those starting points in an asynchronous fashion. James
