Hi
I am not sure exactly what LUCENE-5675 does, but reading the description
it seems to me that it would help finding out that there is no document
(having an id-field) where version-field is less than <some-version>. As
far as I can see this will not help finding out if a document with
id=<some-id> exists. We want to ask "does a document with id <some-id>
exist", without knowing the value of its version-field (if it actually
exists). You do not know if it ever existed, either.
Please elaborate. Thanks!
Regarding " The only other choice today is bloom filters, which use up
huge amounts of memory", I guess a bloom filter only takes as much space
(disk or memory) as you want it to. The more space you allow it to use
the more it gives you a false positive (saying "this doc might exist" in
cases where the doc actually does not exist). So the space you need to
use for the bloom filter depends on how frequently you can live with
false positives (where you have to actually look it up in the real index).
Regards, Per Steffensen
On 30/07/14 10:05, Shalin Shekhar Mangar wrote:
Hi Per,
There's LUCENE-5675 which has added a new postings format for IDs. Trying
it out in Solr is in my todo list but maybe you can get to it before me.
https://issues.apache.org/jira/browse/LUCENE-5675
On Wed, Jul 30, 2014 at 12:57 PM, Per Steffensen <st...@designware.dk>
wrote:
On 30/07/14 08:55, jim ferenczi wrote:
Hi Per,
First of all the BloomFilter implementation in Lucene is not exactly a
bloom filter. It uses only one hash function and you cannot set the false
positive ratio beforehand. ElasticSearch has its own bloom filter
implementation (using "guava like" BloomFilter), you should take a look at
their implementation if you really need this feature.
Yes, I am looking into what Lucene can do and how to use it through Solr.
If it does not fit our needs I will enhance it - potentially with
inspiration from ES implementation. Thanks
What is your use-case ? If your index fits in RAM the bloom filter won't
help (and it may have a negative impact if you have a lot of segments). In
fact the only use case where the bloom filter can help is when your term
dictionary does not fit in RAM which is rarely the case.
We have so many documents that it will never fit in memory. We use
optimistic locking (our own implementation) to do correct concurrent
assembly of documents and to do duplicate control. This require a lot of
finding docs from their id, and most of the time the document is not there,
but to be sure we need to check both transactionlog and the actual index
(UpdateLog). We would like to use Bloom Filter to quickly tell that a
document with a particular id is NOT present.
Regards,
Jim
Regards, Per Steffensen
