[ https://issues.apache.org/jira/browse/NUTCH-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193019#comment-13193019 ]
Markus Jelsma commented on NUTCH-1258: -------------------------------------- Ah, the Content-Type detected by Tika is never added to ParseMeta in the first place! I've modified TikaParser with nutchMetadata.add("Content-Type", mimeType);. In cases where at first i had a bad Content-Type in ParseMeta (but a good one in Content-Meta) i now have good old text/html. The problem is with Content-Types already added to the MetaData by the parser. In that case both the good and bad Content-Types are present in ParseMeta. Just as commented in the code we now have a problem with multi values fields. {code} // populate Nutch metadata with Tika metadata String[] TikaMDNames = tikamd.names(); for (String tikaMDName : TikaMDNames) { if (tikaMDName.equalsIgnoreCase(Metadata.TITLE)) continue; // TODO what if multivalued? nutchMetadata.add(tikaMDName, tikamd.get(tikaMDName)); } {code} This needs another issue opened but some comments are more than appreciated first. Thanks > MoreIndexingFilter should be able to read Content-Type from both parse > metadata and content metadata > ---------------------------------------------------------------------------------------------------- > > Key: NUTCH-1258 > URL: https://issues.apache.org/jira/browse/NUTCH-1258 > Project: Nutch > Issue Type: Improvement > Components: indexer > Affects Versions: 1.4 > Reporter: Markus Jelsma > Assignee: Markus Jelsma > Fix For: 1.5 > > Attachments: NUTCH-1258-1.5-1.patch > > > The MoreIndexingFilter reads the Content-Type from parse metadata. However, > this usually contains a lot of crap because web developers can set it to > anything they like. The filter must be able to read the Content-Type field > from content metadata as well because that contains the type detected by > Tika's Detector. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira