I have succesfully implemented NUTCH as crawler for SOLR index on http://szukaj.ug.edu.pl http://szukaj.ug.edu.pl site. But there is some problem with HTTP REFERER. Nutch is not sending referer header when crawling sites.
Is it possible to order NUTCH to send referer header on request? Scenario: 1. Nutch open www.domain.pl 2. Nutch founds www.domain.pl/abcd.pdf link. 3. Nutch requested www.domain.pl/abcd.pdf but without HTTP_REFERER=www.domain.pl -- View this message in context: http://lucene.472066.n3.nabble.com/HTTP-REFERER-is-missing-tp3987959.html Sent from the Nutch - Agent mailing list archive at Nabble.com.
