Anyone have insight on the following message
attempt_201003301923_0007_m_00_0: -activeThreads=100, spinWaiting=0,
fetchQueues.totalSize=4998
attempt_201003301923_0007_m_00_0: -activeThreads=100, spinWaiting=0,
fetchQueues.totalSize=4998
attempt_201003301923_0007_m_00_0: Aborting
it is not allowed for robots.
http://search.yahoo.com/robots.txt
User-agent: *
Disallow: /search
Disallow: /bin
Disallow: /myweb
Disallow: /myresults
Disallow: /language
Kim Theng Chong schrieb:
Hi all,
Can Nutch crawl Yahoo search result page? eg :
Iam trying to crawl a seed list of 5000. it was working fine, but at the end
of fetching at depth 1 the process failed showing message like this. can any
one suggest what may be problem..
attempt_201003311259_0003_m_03_2: fetching
http://www.law.louisville.edu/news-events/admissions/feed