[
https://issues.apache.org/jira/browse/SOLR-799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yonik Seeley resolved SOLR-799.
-------------------------------
Resolution: Fixed
Fix Version/s: 1.4
- fixed so that all values of a multi-valued field are included in the hash
- changed so that no string additions are done for performance
- moved HEX_CHARS to StrUtils
- changed "fields" to be a comma separated list (per the wiki documentation...
this may be more consistent if we allow this to be specified as a request
parameter later, but it's subjective for sure. we could always add support for
both arrays and comma separated lists).
- changed the hashcode generation to work with any sized hash (was previously
hardcoded to 16 bytes)
- added lookup3ycs
http://yonik.wordpress.com/2008/06/14/lookup3ycs-a-standard-high-performance-string-hash/
lookup3ycs can do hashes directly on strings (no need to convert to bytes
first). I used the 64 bit variant, which is more than enough to prevent false
collisions, and it resulted in a 27% speedup in total indexing time (after
removing other cruft from the schema such as copyFields and default values).
- tested with 10M documents to verify that no collisions occur with both MD5
and lookup3
- Committed! Thanks Mark! And thanks to everyone else for the great feedback.
> Add support for hash based exact/near duplicate document handling
> -----------------------------------------------------------------
>
> Key: SOLR-799
> URL: https://issues.apache.org/jira/browse/SOLR-799
> Project: Solr
> Issue Type: New Feature
> Components: update
> Reporter: Mark Miller
> Assignee: Yonik Seeley
> Priority: Minor
> Fix For: 1.4
>
> Attachments: SOLR-799.patch, SOLR-799.patch, SOLR-799.patch,
> SOLR-799.patch
>
>
> Hash based duplicate document detection is efficient and allows for blocking
> as well as field collapsing. Lets put it into solr.
> http://wiki.apache.org/solr/Deduplication
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
