Hi Does anyone here have a good recommendation for an open source crawler that I could get my hands on? It doesn't have to be python based. I am interested in learning how crawling works. I think python based crawlers will ensure a high degree of flexibility but at the same time I am also torn between looking for open source crawlers in python vs C ++ because the latter is much more efficient(or so I heard. I will be crawling on very cheap hardware.)
I am definitely open to suggestions. Thx -- http://mail.python.org/mailman/listinfo/python-list
