[
https://issues.apache.org/jira/browse/NUTCH-1732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18080809#comment-18080809
]
ASF GitHub Bot commented on NUTCH-1732:
---------------------------------------
lewismc commented on PR #891:
URL: https://github.com/apache/nutch/pull/891#issuecomment-4447361508
Hi @igiguere it looks like [Yetus failed the
CI](https://github.com/apache/nutch/actions/runs/25608554052/job/75925123276?pr=891)
```
conf/nutch-default.xml:1811: <description>Boolean value for whether we
should delete a page from the index when parsing the page fails.
conf/nutch-default.xml:1812: By default this property is deactivated,
because it will delete an existing page from the index, where a
src/java/org/apache/nutch/crawl/CrawlDbReducer.java:133:
src/java/org/apache/nutch/crawl/CrawlDbReducer.java:330:
src/java/org/apache/nutch/parse/ParseSegment.java:132: return;
src/java/org/apache/nutch/parse/ParseStatus.java:191:
src/test/org/apache/nutch/crawl/CrawlDBTestUtil.java:503:
src/test/org/apache/nutch/crawl/CrawlDBTestUtil.java:506: *
src/test/org/apache/nutch/crawl/CrawlDBTestUtil.java:511: * @param
resourceHandler
src/test/org/apache/nutch/crawl/TestCrawlDbStates.java:374:
src/test/org/apache/nutch/crawl/TestCrawlDbStates.java:788:
src/test/org/apache/nutch/crawl/TestCrawlDbStates.java:797:
src/test/org/apache/nutch/crawl/TestCrawlDbStates.java:810:
LOG.info("expect fetch success");
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:52:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:54:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:57:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:64:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:67:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:70:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:71:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:75:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:79:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:89:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:112:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:117:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:132:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:153:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:156:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:176:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:178:
urls.forEach(i ->
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:196:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:231:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:240:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:242:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:246:
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:248:
public void handle(String target, Request baseRequest, HttpServletRequest
request, HttpServletResponse response)
src/test/org/apache/nutch/fetcher/TestFetchWithParseFailures.java:258:
src/test/org/apache/nutch/indexer/TestIndexerMapReduce.java:109:
src/test/org/apache/nutch/indexer/TestIndexerMapReduce.java:127:
```
> IndexerMapReduce to delete explicitly not indexable documents
> -------------------------------------------------------------
>
> Key: NUTCH-1732
> URL: https://issues.apache.org/jira/browse/NUTCH-1732
> Project: Nutch
> Issue Type: Bug
> Components: indexer
> Affects Versions: 1.8
> Reporter: Sebastian Nagel
> Priority: Critical
> Fix For: 1.23
>
>
> In a continuous crawl a previously successfully indexed document (identified
> by a URL) can become "not indexable" for a couple of reasons and must then
> explicitly deleted from the index. Some cases are handled in IndexerMapReduce
> (duplicates, gone documents or redirects, cf. NUTCH-1139) but others are not:
> * failed to parse (but previously successfully parsed): e.g., the document
> became larger and is now truncated
> * rejected by indexing filter (but previously accepted)
> In both cases (maybe there are more) the document should be explicitly
> deleted (if {{-deleteGone}} is set). Note that this cannot be done in
> CleaningJob because data from segments is required.
> We should also update/add a description for {{-deleteGone}}: it does not only
> trigger deletion of gone documents but also of redirects and duplicates (and
> unparseable and skipped docs).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
