Hi, The first item on http://wiki.apache.org/nutch/OptimizingCrawls is DNS caching. Is this still something people regularly do? Even when running in EC2, which I assume has nameservers that are relatively close to instances doing crawling and nameserver lookups?
If so, are there any recommendations for the best DNS caching server/config to use? Thanks, Otis -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/
