[
https://issues.apache.org/jira/browse/LUCENE-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410145#comment-13410145
]
Mark Harwood commented on LUCENE-4069:
--------------------------------------
bq. So now we are close to 1M lookups/sec for a single thread!
Cool!
bq. I wonder if somehow we can do a better job picking the right sized bit
vector up front?
bq. You basically need to know up front how many unique terms will be in the
given field for this segment right?
Yes - the job of anticipating the number of unique keys probably has 2
different contexts:
1) Net new segments e.g. guessing up front how many docs/keys a user is likely
to generate in a new segment before the flush settings kick in.
2) Merged segments e.g. guessing how many unique keys survive a merge operation
Estimating key volumes in context 1 is probably hard without some additional
hints from the end user. Arguably the BloomFilterFactory.getSetForField()
method already represents where this setting can be controlled.
In context 2 where potentially large merges occur we could look at adding an
extra method to BloomFilterFactory to handle this different context e.g.
something like
FuzzySet getSetForMergeOpOnField(FieldInfo fi, OneMerge mergeContext)
Based on the size of the segments being merged and volumes of deletes a more
appropriate size of Bloom bitset could be allocated based on a worst-case
estimate.
Not sure how we get the OneMerge instance fed through the call stack - could
that be held somewhere on a ThreadLocal as generally useful context?
> Segment-level Bloom filters for a 2 x speed up on rare term searches
> --------------------------------------------------------------------
>
> Key: LUCENE-4069
> URL: https://issues.apache.org/jira/browse/LUCENE-4069
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 3.6, 4.0
> Reporter: Mark Harwood
> Priority: Minor
> Fix For: 4.0, 3.6.1
>
> Attachments: BloomFilterPostingsBranch4x.patch,
> LUCENE-4069-tryDeleteDocument.patch, LUCENE-4203.patch,
> MHBloomFilterOn3.6Branch.patch, PKLookupUpdatePerfTest.java,
> PKLookupUpdatePerfTest.java, PKLookupUpdatePerfTest.java,
> PKLookupUpdatePerfTest.java, PrimaryKeyPerfTest40.java
>
>
> An addition to each segment which stores a Bloom filter for selected fields
> in order to give fast-fail to term searches, helping avoid wasted disk access.
> Best suited for low-frequency fields e.g. primary keys on big indexes with
> many segments but also speeds up general searching in my tests.
> Overview slideshow here:
> http://www.slideshare.net/MarkHarwood/lucene-bloomfilteredsegments
> Benchmarks based on Wikipedia content here: http://goo.gl/X7QqU
> Patch based on 3.6 codebase attached.
> There are no 3.6 API changes currently - to play just add a field with "_blm"
> on the end of the name to invoke special indexing/querying capability.
> Clearly a new Field or schema declaration(!) would need adding to APIs to
> configure the service properly.
> Also, a patch for Lucene4.0 codebase introducing a new PostingsFormat
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
