[
https://issues.apache.org/jira/browse/LUCENE-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15256114#comment-15256114
]
Andy Hind commented on LUCENE-6968:
-----------------------------------
The argument here says it is pretty much the same.
{code}
https://en.wikipedia.org/wiki/MinHash
{code}
The plan was to offer both options.
With respect to banding and finding docs related to some start document, the
number of hashes may depend on the start document.
Let's start with 5 word shingles, one hash and keep the minimum 100 hash
values. For a five word document we get one hash. For a 100 word doc where all
the shingles/words are the same we get one hash. For all different shingles we
get 96 hashes.
If we have 100 different hashes and keep the lowest one all the above cases end
up with 100 hashes.
So back to banding. With minimum sets, you need to look and see how many hashes
you really got and then do the banding. Comparing a small documents/snippet
(where we get 10 hashes in the fingerprint)with a much larger document (where
we get 100 hashes) is an interesting case to consider. Starting with the small
document there are fewer bits to match in the generated query. With 100 hashes
from the small document I think you end up in the roughly same place, except
for small snippets. Any given band is more likely to have the same shingle
hashed different ways.
There is also an argument for a winnowing approach. With a 100 hash
fingerprint, sampling for 100 words is great but not so great for 100,000
words. With a minimum set we have the option to generate a finger print related
to the document length and other features.
> LSH Filter
> ----------
>
> Key: LUCENE-6968
> URL: https://issues.apache.org/jira/browse/LUCENE-6968
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Cao Manh Dat
> Attachments: LUCENE-6968.patch, LUCENE-6968.patch, LUCENE-6968.patch
>
>
> I'm planning to implement LSH. Which support query like this
> {quote}
> Find similar documents that have 0.8 or higher similar score with a given
> document. Similarity measurement can be cosine, jaccard, euclid..
> {quote}
> For example. Given following corpus
> {quote}
> 1. Solr is an open source search engine based on Lucene
> 2. Solr is an open source enterprise search engine based on Lucene
> 3. Solr is an popular open source enterprise search engine based on Lucene
> 4. Apache Lucene is a high-performance, full-featured text search engine
> library written entirely in Java
> {quote}
> We wanna find documents that have 0.6 score in jaccard measurement with this
> doc
> {quote}
> Solr is an open source search engine
> {quote}
> It will return only docs 1,2 and 3 (MoreLikeThis will also return doc 4)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
