[ http://issues.apache.org/jira/browse/LUCENE-328?page=all ]
paul.elschot updated LUCENE-328:
--------------------------------
Attachment: SkipFilter1.patch
This patches Filter.java and IndexSearcher.java .
Filter.java is modified to implement SkipFilter, to provide
a first step in a backward compatible way to slowly make Filter
independent of BitSet.
IndexSearcher.java is modified to test for a DocNrSkipper from
a given Filter, and to use that. In that case also skipTo() is used
on the scorer of the query being filtered.
This patch requires org.apache.lucene.util.DocNrSkipper, which
available at this issue.
Also required is org.apache.lucene.search.SkipFilter, which
is available at LUCENE-330.
The patch also contains some commented test code for Filter.java.
This test code always provides a DocNrSkipper (from the BitSet).
With and without this test code, all tests pass here.
When extending Filter in this way, SkipFilter may not be necessary
at all. I left it in to allow a path forward to complete independence
from BitSet.
In case SkipFilter stays, it would be good to add (a) new method(s)
to IndexSearcher allowing a SkipFilter to filter a query.
The DocNrSkipper interface contains only one method:
nextDocNr(int docNr).
It may be good for performance to also add a nextDocNr()
method without any argument, much like skipTo(target) and next()
on Scorer. IOW, I do not consider DocNrSkipper stable at this moment.
I don't think this patch should be added to release 2.0.
> Some utilities for a compact sparse filter
> ------------------------------------------
>
> Key: LUCENE-328
> URL: http://issues.apache.org/jira/browse/LUCENE-328
> Project: Lucene - Java
> Type: Improvement
> Components: Search
> Versions: CVS Nightly - Specify date in submission
> Environment: Operating System: other
> Platform: Other
> Reporter: paul.elschot
> Assignee: Lucene Developers
> Priority: Minor
> Attachments: AndDocNrSkipper.java, AndDocNrSkipper.java,
> BitSetSortedIntList.java, DocNrSkipper.java, DocNrSkipper.java,
> IntArraySortedIntList.java, IntArraySortedIntList.java, OrDocNrSkipper.java,
> OrDocNrSkipper.java, SkipFilter1.patch, SortedVIntList.java,
> SortedVIntList.java, SortedVIntList.java, TestDocNrSkippers.java,
> TestDocNrSkippers.java, TestSortedVIntList.java, TestSortedVIntList.java,
> TestSortedVIntList.java
>
> Two files are attached that might form the basis for an alternative
> filter implementation that is more memory efficient than one bit
> per doc when less than about 1/8 of the docs pass through the filter.
>
> The document numbers are stored in RAM as VInt's from the Lucene index
> format. These VInt's encode the difference between two successive
> document numbers, much like a PositionDelta in the Positions:
> http://jakarta.apache.org/lucene/docs/fileformats.html
>
> The getByteSize() method can be used to verify the compression
> once a SortedVIntList is constructed.
> The precise conditions under which this is more memory efficient than
> one bit per document are not easy to specify in advance.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
