[
https://issues.apache.org/jira/browse/NUTCH-1708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13876288#comment-13876288
]
Markus Jelsma commented on NUTCH-1708:
--------------------------------------
Hi Sebastian - we've had issues with that before and tracked it down to the
representative URL being indexing in index-basic as well. We choose to
completely remove that from our custom indexing filter, in my opinion an URL
must be indexed by it's real URL, not some representative URL. Indexing
representative URL's also causes duplicates, which may or may not be removed by
Nutch' new deduplicating code because the signatures are usually not the same.
> use same id when indexing and deleting redirects
> ------------------------------------------------
>
> Key: NUTCH-1708
> URL: https://issues.apache.org/jira/browse/NUTCH-1708
> Project: Nutch
> Issue Type: Bug
> Components: indexer
> Affects Versions: 1.7
> Reporter: Sebastian Nagel
>
> Redirect targets are indexed using "representative URL"
> * in Fetcher repr URL is determined by URLUtil.chooseRepr() and stored in
> CrawlDatum (CrawlDb). Repr URL is either source or target URL of the redirect
> pair.
> * NutchField "url" is filled by basic indexing filter with repr URL
> * id field used as unique key is filled from url per solrindex-mapping.xml
> Deletion of redirects is done in IndexerMapReduce.reduce() by key which is
> the URL of the redirect source. If the source URL is chosen as repr URL a
> redirect target may get erroneously deleted.
> Test crawl with seed {{http://wiki.apache.org/nutch}} which redirects to
> {{http://wiki.apache.org/nutch/}}. DummyIndexWriter (NUTCH-1707) indicates
> that same URL is deleted and added:
> {code}
> delete http://wiki.apache.org/nutch
> add http://wiki.apache.org/nutch
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
