[ https://issues.apache.org/jira/browse/NUTCH-1732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1732: --------------------------------- Priority: Critical (was: Major) > IndexerMapReduce to delete explicitly not indexable documents > ------------------------------------------------------------- > > Key: NUTCH-1732 > URL: https://issues.apache.org/jira/browse/NUTCH-1732 > Project: Nutch > Issue Type: Bug > Components: indexer > Affects Versions: 1.8 > Reporter: Sebastian Nagel > Priority: Critical > Fix For: 1.9 > > > In a continuous crawl a previously successfully indexed document (identified > by a URL) can become "not indexable" for a couple of reasons and must then > explicitly deleted from the index. Some cases are handled in IndexerMapReduce > (duplicates, gone documents or redirects, cf. NUTCH-1139) but others are not: > * failed to parse (but previously successfully parsed): e.g., the document > became larger and is now truncated > * rejected by indexing filter (but previously accepted) > In both cases (maybe there are more) the document should be explicitly > deleted (if {{-deleteGone}} is set). Note that this cannot be done in > CleaningJob because data from segments is required. > We should also update/add a description for {{-deleteGone}}: it does not only > trigger deletion of gone documents but also of redirects and duplicates (and > unparseable and skipped docs). -- This message was sent by Atlassian JIRA (v6.2#6252)