On 30/07/14 08:55, jim ferenczi wrote:
Hi Per,
First of all the BloomFilter implementation in Lucene is not exactly a
bloom filter. It uses only one hash function and you cannot set the false
positive ratio beforehand. ElasticSearch has its own bloom filter
implementation (using "guava like" BloomFilter), you should take a look at
their implementation if you really need this feature.
Yes, I am looking into what Lucene can do and how to use it through
Solr. If it does not fit our needs I will enhance it - potentially with
inspiration from ES implementation. Thanks
What is your use-case ? If your index fits in RAM the bloom filter won't
help (and it may have a negative impact if you have a lot of segments). In
fact the only use case where the bloom filter can help is when your term
dictionary does not fit in RAM which is rarely the case.
We have so many documents that it will never fit in memory. We use
optimistic locking (our own implementation) to do correct concurrent
assembly of documents and to do duplicate control. This require a lot of
finding docs from their id, and most of the time the document is not
there, but to be sure we need to check both transactionlog and the
actual index (UpdateLog). We would like to use Bloom Filter to quickly
tell that a document with a particular id is NOT present.
Regards,
Jim
Regards, Per Steffensen
