[
https://issues.apache.org/jira/browse/NUTCH-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14307974#comment-14307974
]
Jorge Luis Betancourt Gonzalez commented on NUTCH-1928:
-------------------------------------------------------
[~lewismc] I've updated the patch:
* Actually I was generating the patch from our internal SVN repository, and we
keep our plugins separated from the rest of the Nutch distribution, so the
previous patch couldn't be applied from the $NUTCH_HOME. I've generated the
patch from the 1.9 $NUTCH_HOME (sources).
* As usual you were right, I was using the deprecated syntax of the JUnit
tests, sorry for that.
As usual really useful feedback!
> Indexing filter of documents by the MIME type
> ---------------------------------------------
>
> Key: NUTCH-1928
> URL: https://issues.apache.org/jira/browse/NUTCH-1928
> Project: Nutch
> Issue Type: Improvement
> Components: indexer, plugin
> Reporter: Jorge Luis Betancourt Gonzalez
> Assignee: Jorge Luis Betancourt Gonzalez
> Labels: filter, mime-type, plugin
> Fix For: 1.10
>
> Attachments: mimetype-patch-v3.patch
>
>
> This allows to filter the indexed documents by the MIME type property of the
> crawled content. Basically this will allow you to restrict the MIME type of
> the contents that will be stored in Solr/Elasticsearch index without the need
> to restrict the crawling/parsing process, so no need to use URLFilter plugin
> family. Also this address one particular corner case when certain URLs
> doesn't have any format to filter such as some RSS feeds
> (http://www.awesomesite.com/feed) and it will end in your index mixed with
> all your HTML content.
> A configuration can file specified on the {{mimetype.filter.file}} property
> in the {{nutch-site.xml}}. This file use the same format as the
> {{urlfilter-suffix}} plugin. If no {{mimetype.filter.file}} key is found an
> {{allow all}} policy is used instead, so all your crawled documents will be
> indexed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
