[
https://issues.apache.org/jira/browse/NUTCH-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15936522#comment-15936522
]
Sebastian Nagel commented on NUTCH-2335:
----------------------------------------
Rebased pull-request, tested in production.
> Injector not to filter and normalize existing URLs in CrawlDb
> -------------------------------------------------------------
>
> Key: NUTCH-2335
> URL: https://issues.apache.org/jira/browse/NUTCH-2335
> Project: Nutch
> Issue Type: Improvement
> Components: crawldb, injector
> Affects Versions: 1.12
> Reporter: Sebastian Nagel
> Fix For: 1.13
>
>
> With NUTCH-1712 the behavior of the Injector has changed in case new URLs are
> added to an existing CrawlDb:
> - before only injected URLs were filtered and normalized
> - now filters and normalizers are applied to all URLs including those already
> in the CrawlDb
> The default should be as before not to filter existing URLs. Filtering and
> normalizing may take long for large CrawlDbs and/or complex URL filters. If
> URL filter or normalizer rules are not changed there is no need to apply them
> anew every time new URLs are added. Of course, injected URLs should be
> filtered and normalized by default.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
