Hello - use the -deleteRobotsNoIndex flag. I also believe the meta tag should be:
<meta name="robots" value='noindex' /> It should be relatively easy to have the indexer listen to the real robots name instead. Markus -----Original message----- > From:Megha Bhandari <mbhanda...@sapient.com> > Sent: Tuesday 19th April 2016 10:54 > To: user@nutch.apache.org > Subject: Nutch 1.11 : meta directive noindex not honored > > Hi > > We are using Nutch 1.11 and are using the following meta directive > > <meta name="bot-name-set-in-site-default-xml" value='noindex' /> > > However Nutch is still indexing the page into Solr(5.5) . > > Any idea what settings are being missed over here? > > Thanks for any help. > > Megha >
