[
https://issues.apache.org/jira/browse/LUCENE-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-5604:
---------------------------------------
Attachment: LUCENE-5604.patch
OK, new patch using simple linear probe.
For the benchmark, I switched to a single indexing thread,
NoMergePolicy, and set RAM buffer to 512 MB (so more entries in the
hash table before flushing, to provoke more conflict chances). I
index the first 5M docs of Wikipedia medium, and full Geonames corpus:
{noformat}
wiki
trunk: 402.8 sec, 194906771 conflicts
murmur3 + current probe: 410.8 sec, 136527391 conflicts
murmur3 + linear probe: 395.8 sec, 63403239 conflicts
geonames
trunk: 624.1 sec, 368781904 conflicts
murmur3 + current probe: 607.1 sec, 168310018 conflicts
murmur3 + linear probe: 606.6 sec, 176405531 conflicts
{noformat}
Curious how the linear probe was such a big win for Wikipedia in
reducing conflicts, but slightly higher conflicts for Geonames, but
net/net I think we should switch to linear probe.
> Should we switch BytesRefHash to MurmurHash3?
> ---------------------------------------------
>
> Key: LUCENE-5604
> URL: https://issues.apache.org/jira/browse/LUCENE-5604
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 4.9, 5.0
>
> Attachments: BytesRefHash.perturb.patch, LUCENE-5604.patch,
> LUCENE-5604.patch, LUCENE-5604.patch
>
>
> MurmurHash3 has better hashing distribution than the current hash function we
> use for BytesRefHash which is a simple multiplicative function with 31
> multiplier (same as Java's String.hashCode, but applied to bytes not chars).
> Maybe we should switch ...
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
