[
https://issues.apache.org/jira/browse/NUTCH-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12675315#action_12675315
]
Doğacan Güney commented on NUTCH-684:
-------------------------------------
I wasn't thinking of putting this in for 1.0, but if people want this feature I
will ready it for 1.0
bq. * there is a silent assumption that Solr schema uses "id" field as
unique key, and that this field contains the URL of the document. First,
shouldn't this be "url" field? Because as far as I can see the field name "id"
is not used anywhere in SolrIndexer/SolrWriter - please correct me if I missed
something. At least this assumption should be spelled out in javadocs, both on
the indexing side and on the dedup side. (Actually, we should have added an
example of the minimum required Solr schema when the original Nutch/Solr
integration was committed)
* field names should be constants and not magic literals, they should come
either from o.a.n.metadata.Nutch or be defined in SolrConstants.
This is something I have been thinking for a while. My assumption was that you
didn't have to use "url" field in
your solr server as the unique field so I added an extra "id" field (which in
NUTCH-442's schema.xml is copied from "url"). But I am no longer sure the extra
cost of a field is worth the flexibility.
I agree with you that we should have an solr schema xml somewhere in our
codebase that is officially blessed. I guess NUTCH-442's schema is a good
starting point for that but I am open to suggestions. I will create a new issue
for it.
bq. SolrServer.deleteById() creates and sends UpdateRequest containing just
this single id. This is inefficient, especially in our case where the number of
deletes may be significant. Perhaps this patch works sufficiently well for now,
but it should be improved (either here or in a separate issue) by using a
single UpdateRequest per reduce task, and calling
SolrServer.request(UpdateRequest) with the accumulated id-s.
Good point. I will send an improved patch.
> Dedup support for Solr
> ----------------------
>
> Key: NUTCH-684
> URL: https://issues.apache.org/jira/browse/NUTCH-684
> Project: Nutch
> Issue Type: New Feature
> Components: indexer
> Reporter: Doğacan Güney
> Assignee: Doğacan Güney
> Attachments: NUTCH-684_bin_nutch.patch, NUTCH-684_solrdedup_v2.patch,
> solrdedup.patch
>
>
> After NUTCH-442, nutch now can index to both solr and lucene. However,
> duplicate deletion feature (based on digests) is only available in lucene. It
> should also be available for solr.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
