[
https://issues.apache.org/jira/browse/LUCENE-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15252743#comment-15252743
]
Andy Hind commented on LUCENE-6968:
-----------------------------------
Hi
It would be quite common to use min hashing after shingling. At this point the
number of possible word combinations vs the size of the hash is important. With
shingles of 5 words from 100,000 that is 10e25 combinations. Some naive
processing of the ~500k Enron emails (splitting on white space, case folding
and 5 word shingles) gives ~1e13 combinations. So a long hash would be better
at 1.8e19. I have not yet looked at a larger corpus.
The LSH query is neat. However the logic can give banding where the last band
is uneven. In the patch I think the last band would be dropped unless bands *
rows in band = # of hashes
The underlying state of the source filter may also be lost (if using shingling)
I do not believe the similarity is required at all. I think you can get Jaccard
distance using constant score queries and disabling coordination on the boolean
query.
I went for 128-bit hashes, or a 32 bit hash identifier + 96 bit hash with a bit
more flexibility allowing a minimum set of hash values for a bunch of hashes.
There is clearly some trade off for speed of hashing and over representing
short documents. The minimum set may be a solution to this. I think there is
some interesting research there.
I will add my patch inspired by the original .... and apologise for the mixed
formatting in advance ......
> LSH Filter
> ----------
>
> Key: LUCENE-6968
> URL: https://issues.apache.org/jira/browse/LUCENE-6968
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Cao Manh Dat
> Attachments: LUCENE-6968.patch, LUCENE-6968.patch
>
>
> I'm planning to implement LSH. Which support query like this
> {quote}
> Find similar documents that have 0.8 or higher similar score with a given
> document. Similarity measurement can be cosine, jaccard, euclid..
> {quote}
> For example. Given following corpus
> {quote}
> 1. Solr is an open source search engine based on Lucene
> 2. Solr is an open source enterprise search engine based on Lucene
> 3. Solr is an popular open source enterprise search engine based on Lucene
> 4. Apache Lucene is a high-performance, full-featured text search engine
> library written entirely in Java
> {quote}
> We wanna find documents that have 0.6 score in jaccard measurement with this
> doc
> {quote}
> Solr is an open source search engine
> {quote}
> It will return only docs 1,2 and 3 (MoreLikeThis will also return doc 4)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
