[
https://issues.apache.org/jira/browse/LUCENE-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17573156#comment-17573156
]
ASF subversion and git services commented on LUCENE-10559:
----------------------------------------------------------
Commit 33d5ab96f266447b228833baee3489b12bdc3a68 in lucene's branch
refs/heads/branch_9x from Kaival Parikh
[ https://gitbox.apache.org/repos/asf?p=lucene.git;h=33d5ab96f26 ]
LUCENE-10559: Add Prefilter Option to KnnGraphTester (#932)
Added a `prefilter` and `filterSelectivity` argument to KnnGraphTester to be
able to compare pre and post-filtering benchmarks.
`filterSelectivity` expresses the selectivity of a filter as proportion of
passing docs that are randomly selected. We store these in a FixedBitSet and
use this to calculate true KNN as well as in HNSW search.
In case of post-filter, we over-select results as `topK / filterSelectivity` to
get final hits close to actual requested `topK`. For pre-filter, we wrap the
FixedBitSet in a query and pass it as prefilter argument to KnnVectorQuery.
> Add preFilter/postFilter options to KnnGraphTester
> --------------------------------------------------
>
> Key: LUCENE-10559
> URL: https://issues.apache.org/jira/browse/LUCENE-10559
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Michael Sokolov
> Priority: Major
> Fix For: 9.4
>
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
> We want to be able to test the efficacy of pre-filtering in KnnVectorQuery:
> if you (say) want the top K nearest neighbors subject to a constraint Q, are
> you better off over-selecting (say 2K) top hits and *then* filtering
> (post-filtering), or incorporating the filtering into the query
> (pre-filtering). How does it depend on the selectivity of the filter?
> I think we can get a reasonable testbed by generating a uniform random filter
> with some selectivity (that is consistent and repeatable). Possibly we'd also
> want to try filters that are correlated with index order, but it seems they'd
> be unlikely to be correlated with vector values in a way that the graph
> structure would notice, so random is a pretty good starting point for this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
