Otis - we tried local DNS caching when we did very large scale crawls but decided to get rid of it as soon as possible because it got us too much overhead. Instead, we relied on an, apparently, powerful DNS server put available by the ISP in the network center. If the server is fast and has a lot of RAM the mapper won't quickly overwhelm it.
Markus -----Original message----- > From:Otis Gospodnetić <otis.gospodne...@gmail.com> > Sent: Sunday 31st January 2016 23:36 > To: Nutch User List <nutch-u...@lucene.apache.org> > Subject: DNS caching best practices > > Hi, > > The first item on http://wiki.apache.org/nutch/OptimizingCrawls is DNS > caching. Is this still something people regularly do? Even when running > in EC2, which I assume has nameservers that are relatively close to > instances doing crawling and nameserver lookups? > > If so, are there any recommendations for the best DNS caching server/config > to use? > > Thanks, > Otis > -- > Monitoring - Log Management - Alerting - Anomaly Detection > Solr & Elasticsearch Consulting Support Training - http://sematext.com/ >
