On Apr 14, 2006, at 4:26 AM, J Cook wrote:

Plucene maybe? Its up on CPAN.

I'm intimately acquainted with Plucene. I actually spent a week or two hacking on it last August before deciding that its performance issues could not be resolved without a complete overhaul which would break the API.

   http://www.rectangular.com/kinosearch/benchmarks.html

KinoSearch, like Plucene, is a text search engine library. In order to write an industrial-strength spider a la Nutch, you need a lot more than that: HTML::Parser, HTML::LinkExtor, LWP::RobotUA... I've now discovered WWW::RobotRules::AnyDBM_File, which is going to be very helpful. But there are a lot of other problems to be solved. Check-summing page content to eliminate duplicate documents available via multiple URLs. Managing crawl depth so that a spider doesn't venture too deep into one domain and forget about all the others. Eventually, if you want to get fancy, link analysis and pagerank.

LWP::Parallel::RobotUA looks interesting. There's a bunch of stuff under Bundle::LinkController, but it hasn't been updated in a while. What else?

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

Reply via email to