[jira] Updated: (LUCENE-855) MemoryCachedRangeFilter to boost performance of Range queries
[ https://issues.apache.org/jira/browse/LUCENE-855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Ericson updated LUCENE-855: Attachment: FieldCacheRangeFilter.patch Andy was correct the 2 performance tests were bogus as they did not call get() from the bit sets. And my code does all of the work int the get() call. I guess I should have looked a little closer at the tests before using it I changes his tests and mine to call and IndexSearcher.search(q,filter) and actually do the search Here are the results Using the MemoryCachedRangeFilter [junit] - Standard Output --- [junit] Start interval: Tue Apr 09 14:32:14 PDT 2002 [junit] End interval: Sun Apr 08 14:32:14 PDT 2007 [junit] Creating RAMDirectory index... [junit] Reader opened with 10 documents. Creating RangeFilters... [junit] Standard RangeFilter finished in 57533ms [junit] MemoryCachedRangeFilter inished in 905ms [junit] - --- Using FieldCacheRangeFilter [junit] - Standard Output --- [junit] Start interval: Tue Apr 09 14:30:29 PDT 2002 [junit] End interval: Sun Apr 08 14:30:29 PDT 2007 [junit] Creating RAMDirectory index... [junit] Reader opened with 10 documents. Creating RangeFilters... [junit] Standard RangeFilter finished in 58822ms [junit] FieldCacheRangeFilter inished in 102ms [junit] - --- They are much closer this time I have fixed my BitSets to allow a user to call nextClearBit or nextSetBit > MemoryCachedRangeFilter to boost performance of Range queries > - > > Key: LUCENE-855 > URL: https://issues.apache.org/jira/browse/LUCENE-855 > Project: Lucene - Java > Issue Type: Improvement > Components: Search >Affects Versions: 2.1 >Reporter: Andy Liu > Attachments: FieldCacheRangeFilter.patch, > FieldCacheRangeFilter.patch, MemoryCachedRangeFilter.patch, > MemoryCachedRangeFilter_1.4.patch > > > Currently RangeFilter uses TermEnum and TermDocs to find documents that fall > within the specified range. This requires iterating through every single > term in the index and can get rather slow for large document sets. > MemoryCachedRangeFilter reads all pairs of a given field, > sorts by value, and stores in a SortedFieldCache. During bits(), binary > searches are used to find the start and end indices of the lower and upper > bound values. The BitSet is populated by all the docId values that fall in > between the start and end indices. > TestMemoryCachedRangeFilterPerformance creates a 100K RAMDirectory-backed > index with random date values within a 5 year range. Executing bits() 1000 > times on standard RangeQuery using random date intervals took 63904ms. Using > MemoryCachedRangeFilter, it took 876ms. Performance increase is less > dramatic when you have less unique terms in a field or using less number of > documents. > Currently MemoryCachedRangeFilter only works with numeric values (values are > stored in a long[] array) but it can be easily changed to support Strings. A > side "benefit" of storing the values are stored as longs, is that there's no > longer the need to make the values lexographically comparable, i.e. padding > numeric values with zeros. > The downside of using MemoryCachedRangeFilter is there's a fairly significant > memory requirement. So it's designed to be used in situations where range > filter performance is critical and memory consumption is not an issue. The > memory requirements are: (sizeof(int) + sizeof(long)) * numDocs. > MemoryCachedRangeFilter also requires a warmup step which can take a while to > run in large datasets (it took 40s to run on a 3M document corpus). Warmup > can be called explicitly or is automatically called the first time > MemoryCachedRangeFilter is applied using a given field. > So in summery, MemoryCachedRangeFilter can be useful when: > - Performance is critical > - Memory is not an issue > - Field contains many unique numeric values > - Index contains large amount of documents -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-584) Decouple Filter from BitSet
[
https://issues.apache.org/jira/browse/LUCENE-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12487456
]
Doron Cohen commented on LUCENE-584:
...right, your diff-txt had the Match tasks - I missed that - checked it, it is
exactly what I did, so we're ok here.
When you rerun, you may want to use my alg - to compare the two approaches in
one run. You can run this by something like:
ant run-task -Dtask.mem=256M -Dtask.alg=conf\matcher-vs-bitset.alg
Also, to get cleaner results, add the line:
ResetSystemSoft
just in the beginning of the "search round" - this resets the (query) inputs
and also calls GC.
I tried like this twice, and got inconsistent results:
When the bitset searches preceded the match searches:
[java] Operation round runCnt recsPerRunrec/s
elapsedSecavgUsedMemavgTotalMem
[java] SrchBitsSamRdr_5000 - 10 5000706.4
70.78 7,511,219 16,573,645
[java] SrchMtchSamRdr_5000 - - - - 10 - - - 5000 - - 689.6 - -
72.50 - 8,223,005 - 11,926,323
[java] SrchBitsNewRdr_500 - 10 500152.5
32.8014,360,618 16,962,356
[java] SrchMtchNewRdr_500 - - - - - 10 - - - 500 - - 171.3 - -
29.19 - 15,150,797 - 17,395,712
When the match searches preceded the bitset searches:
[java] Operation round runCnt recsPerRunrec/s
elapsedSecavgUsedMemavgTotalMem
[java] SrchMtchSamRdr_5000 - 10 5000763.5
65.49 9,563,243 17,128,244
[java] SrchBitsSamRdr_5000 - - - - 10 - - - 5000 - - 729.3 - -
68.56 - 10,003,775 - 13,001,114
[java] SrchMtchNewRdr_500 - 10 500175.7
28.4612,068,559 17,524,326
[java] SrchBitsNewRdr_500 - - - - - 10 - - - 500 - - 183.7 - -
27.22 - 15,098,480 - 17,974,476
My conclusion from this is that the speed-up, if exists, is minor, at least for
the setup of this test.
There are only 15 unique queries in this test - also printed in the log - are
these the queries you would expect to save in?
I didn't follow this issue very closely so I don't know where the saving is
expected here. Both SearchTask and MatchTask now do nothing in collect, so no
difference at the actual collect() call.
Also, Scorer.score(HitCollector) and Matcher.match(MatchCollector) are very
similar:
public void score(HitCollector hc) throws IOException {
while (next()) {
hc.collect(doc(), score());
}
}
public void match(MatchCollector mc) throws IOException {
while (next()) {
mc.collect(doc());
}
}
Especially for the case that the collect() method is doing nothing, as in this
test.
I think there is a potential gain for large boolean OR queries, because score()
would have to call next() on all TermScorers and collect/sum their scores,
while match() could use skipTo(last+1) because any match encountered is a match
and there is no need to sum the individual scores for the same doc by other
scorers. However as far as I can tell, current match() implementation does not
take advantage of this, but I may be overlooking something?
> Decouple Filter from BitSet
> ---
>
> Key: LUCENE-584
> URL: https://issues.apache.org/jira/browse/LUCENE-584
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
>Affects Versions: 2.0.1
>Reporter: Peter Schäfer
>Priority: Minor
> Attachments: bench-diff.txt, bench-diff.txt, BitsMatcher.java,
> Filter-20060628.patch, HitCollector-20060628.patch,
> IndexSearcher-20060628.patch, MatchCollector.java, Matcher.java,
> Matcher20070226.patch, Scorer-20060628.patch, Searchable-20060628.patch,
> Searcher-20060628.patch, Some Matchers.zip, SortedVIntList.java,
> TestSortedVIntList.java
>
>
> {code}
> package org.apache.lucene.search;
> public abstract class Filter implements java.io.Serializable
> {
> public abstract AbstractBitSet bits(IndexReader reader) throws IOException;
> }
> public interface AbstractBitSet
> {
> public boolean get(int index);
> }
> {code}
> It would be useful if the method =Filter.bits()= returned an abstract
> interface, instead of =java.util.BitSet=.
> Use case: there is a very large index, and, depending on the user's
> privileges, only a small portion of the index is actually visible.
> Sparsely populated =java.util.BitSet=s are not efficient and waste lots of
> memory. It would be desirable to have an alternative BitSet implementation
> with smaller memory footprint.
> Though it _is_ possibly to derive classes from =java.util.BitSet=, it was
> obviously not designed for that purpose.
> That's why I propose to use an interface instead. The default implementation
> could stil
