[
https://issues.apache.org/jira/browse/LUCENE-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428839#comment-13428839
]
Michael McCandless commented on LUCENE-4069:
--------------------------------------------
Hmm in testing Bloom + Lucene40 PF on luceneutil's "id" field I don't see any
gains with what was committed (base=Lucene40 for id field, comp=Bloom +
Lucene40 for id field; all other fields are just Lucene40):
{noformat}
Task QPS base StdDev base QPS bloomStdDev bloom Pct
diff
PKLookup 196.71 9.29 192.94 5.40 -8% -
5%
HighSloppyPhrase 0.75 0.04 0.75 0.04 -10% -
9%
HighPhrase 1.78 0.02 1.76 0.01 -2% -
0%
MedPhrase 5.00 0.09 4.95 0.15 -5% -
3%
AndHighMed 74.66 3.52 74.02 4.50 -11% -
10%
AndHighHigh 16.94 0.82 16.83 0.78 -9% -
9%
MedSloppyPhrase 4.10 0.11 4.08 0.12 -5% -
4%
LowPhrase 13.35 0.19 13.31 0.24 -3% -
2%
LowSloppyPhrase 31.91 0.52 31.90 0.58 -3% -
3%
AndHighLow 1114.88 43.26 1121.52 33.72 -6% -
7%
LowSpanNear 9.67 0.22 9.76 0.26 -4% -
5%
LowTerm 421.55 14.96 426.55 5.23 -3% -
6%
MedTerm 176.45 7.66 179.40 2.84 -4% -
7%
HighSpanNear 1.43 0.03 1.45 0.05 -3% -
7%
MedSpanNear 2.51 0.05 2.56 0.09 -3% -
7%
HighTerm 15.52 0.95 15.90 0.24 -4% -
10%
Respell 69.77 3.20 71.65 3.61 -6% -
13%
Fuzzy1 88.19 2.49 91.04 3.06 -2% -
9%
Fuzzy2 36.18 1.26 37.50 1.44 -3% -
11%
Wildcard 45.33 1.52 47.04 1.10 -1% -
9%
Prefix3 65.65 2.53 68.26 1.99 -2% -
11%
OrHighLow 37.45 0.88 39.87 2.48 -2% -
15%
OrHighHigh 9.58 0.10 10.23 0.64 0% -
14%
OrHighMed 14.89 0.14 15.96 1.01 0% -
15%
IntNRQ 10.24 1.04 11.07 1.33 -13% -
34%
{noformat}
I just used the DefaultBloomFilterFactory, which should be good here since it
targets 10% fill based on doc count.
Maybe even slight loss (but could easily be noise). I confirmed that I'm
getting .blm files and that at search time the filter does in fact "work"
(quickly returns false from seekExact).
The differences vs. the dedicated benchmark on this issue is 1) luceneutil is
only testing lookups, 2) the "id" field is just one field (ie we also have eg
body/title/date for searching), 3) luceneutil is testing a bunch of other
queries in other threads in addition to PKLookup.
Out of curiosity I also tried Memory vs Bloom + Memory, and Bloom was ~15%
slower, I guess because computing the hash & doing the lookup is non-trivial
cost vs just walking the FST to find out the term doesn't exist.
> Segment-level Bloom filters for a 2 x speed up on rare term searches
> --------------------------------------------------------------------
>
> Key: LUCENE-4069
> URL: https://issues.apache.org/jira/browse/LUCENE-4069
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 3.6, 4.0-ALPHA
> Reporter: Mark Harwood
> Assignee: Mark Harwood
> Priority: Minor
> Fix For: 4.0, 5.0
>
> Attachments: 4069Failure.zip, BloomFilterPostingsBranch4x.patch,
> LUCENE-4069-tryDeleteDocument.patch, LUCENE-4203.patch,
> MHBloomFilterOn3.6Branch.patch, PKLookupUpdatePerfTest.java,
> PKLookupUpdatePerfTest.java, PKLookupUpdatePerfTest.java,
> PKLookupUpdatePerfTest.java, PrimaryKeyPerfTest40.java
>
>
> An addition to each segment which stores a Bloom filter for selected fields
> in order to give fast-fail to term searches, helping avoid wasted disk access.
> Best suited for low-frequency fields e.g. primary keys on big indexes with
> many segments but also speeds up general searching in my tests.
> Overview slideshow here:
> http://www.slideshare.net/MarkHarwood/lucene-bloomfilteredsegments
> Benchmarks based on Wikipedia content here: http://goo.gl/X7QqU
> Patch based on 3.6 codebase attached.
> There are no 3.6 API changes currently - to play just add a field with "_blm"
> on the end of the name to invoke special indexing/querying capability.
> Clearly a new Field or schema declaration(!) would need adding to APIs to
> configure the service properly.
> Also, a patch for Lucene4.0 codebase introducing a new PostingsFormat
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
