[ https://issues.apache.org/jira/browse/NUTCH-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356694#comment-15356694 ]
Hudson commented on NUTCH-1553: ------------------------------- FAILURE: Integrated in Nutch-trunk #3377 (See [https://builds.apache.org/job/Nutch-trunk/3377/]) NUTCH-1553 Property 'indexer.delete.robots.noindex' not working when (snagel: rev cb6fbae51a56587c30d15b8f170ebbf470851168) * src/plugin/parse-html/src/java/org/apache/nutch/parse/html/HtmlParser.java > Property 'indexer.delete.robots.noindex' not working when using parser-html. > ---------------------------------------------------------------------------- > > Key: NUTCH-1553 > URL: https://issues.apache.org/jira/browse/NUTCH-1553 > Project: Nutch > Issue Type: Bug > Components: indexer, parser > Affects Versions: 1.6 > Reporter: Alfonso Presa > Assignee: Sebastian Nagel > Priority: Minor > Fix For: 1.13 > > Attachments: NUTCH-1553-trunk-1.patch > > > May be I'm doing something wrong, but it seems to me that +NUTCH-1434+ patch > only works when using tika's parser. When using parser-html, "robots" metatag > is only populated if parse-metatags plugin is enabled and it's done with the > prefix "metatag.". So parseData.getMeta("robots") returns nothing if not > using tika. > I guess the simplest solution would be to provide a fallback in case > parseData.getMeta("robots") is null and then get > parseData.getMeta("metatag.robots") in that case. > Also dependency of this property with parse-metadata plugin when using > parse-html would be something interesting to document somewhere... > (nutch-default.xml?) > Thanks! -- This message was sent by Atlassian JIRA (v6.3.4#6332)