[ https://issues.apache.org/jira/browse/LUCENE-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15252743#comment-15252743 ]
Andy Hind commented on LUCENE-6968: ----------------------------------- Hi It would be quite common to use min hashing after shingling. At this point the number of possible word combinations vs the size of the hash is important. With shingles of 5 words from 100,000 that is 10e25 combinations. Some naive processing of the ~500k Enron emails (splitting on white space, case folding and 5 word shingles) gives ~1e13 combinations. So a long hash would be better at 1.8e19. I have not yet looked at a larger corpus. The LSH query is neat. However the logic can give banding where the last band is uneven. In the patch I think the last band would be dropped unless bands * rows in band = # of hashes The underlying state of the source filter may also be lost (if using shingling) I do not believe the similarity is required at all. I think you can get Jaccard distance using constant score queries and disabling coordination on the boolean query. I went for 128-bit hashes, or a 32 bit hash identifier + 96 bit hash with a bit more flexibility allowing a minimum set of hash values for a bunch of hashes. There is clearly some trade off for speed of hashing and over representing short documents. The minimum set may be a solution to this. I think there is some interesting research there. I will add my patch inspired by the original .... and apologise for the mixed formatting in advance ...... > LSH Filter > ---------- > > Key: LUCENE-6968 > URL: https://issues.apache.org/jira/browse/LUCENE-6968 > Project: Lucene - Core > Issue Type: Improvement > Reporter: Cao Manh Dat > Attachments: LUCENE-6968.patch, LUCENE-6968.patch > > > I'm planning to implement LSH. Which support query like this > {quote} > Find similar documents that have 0.8 or higher similar score with a given > document. Similarity measurement can be cosine, jaccard, euclid.. > {quote} > For example. Given following corpus > {quote} > 1. Solr is an open source search engine based on Lucene > 2. Solr is an open source enterprise search engine based on Lucene > 3. Solr is an popular open source enterprise search engine based on Lucene > 4. Apache Lucene is a high-performance, full-featured text search engine > library written entirely in Java > {quote} > We wanna find documents that have 0.6 score in jaccard measurement with this > doc > {quote} > Solr is an open source search engine > {quote} > It will return only docs 1,2 and 3 (MoreLikeThis will also return doc 4) -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org