No - you won't be able to crawl this page. Nutch will follow robots directive of the domain - see http://search.yahoo.com/robots.txt.
-Devang. -----Original Message----- From: Kim Theng Chong [mailto:kimthe...@yahoo.com] Sent: Tuesday, March 30, 2010 10:00 PM To: nutch-user@lucene.apache.org Subject: Crawl yahoo search result page Hi all, Can Nutch crawl Yahoo search result page? eg : http://search.yahoo.com/search?rd=&fp_ip=my&p=ontology&toggle=1&cop=mss&ei=U TF-8&fr=yfp-t-892 (put as seed url) . I was not able to fetch the results in this page. Can someone guide me on this? Thank you. Best regards, Kim
