[
https://issues.apache.org/jira/browse/SOLR-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18018285#comment-18018285
]
Sonu Sharma edited comment on SOLR-17319 at 9/5/25 6:52 AM:
------------------------------------------------------------
Please understand that the document's relative ranking is lost the moment it is
sharded. And It is not going to help getting the same expected document as in
the case of non-sharded, either RRF done locally to a shard or RRF done on
incorrect ordering received per query from all shards.
So, any approach that computes a "score" based on merging the per-shard results
first and then doing RRF _per query_ to find the topN results is also wrong
IMHO, as I didn't find any definition of RRF (or any algorithm taking rank as a
signal) over multiple shards. Please help me with a one if you came across.
Let me dissect the Way 2 with a similar example:
Start with the same settings and add some query score, assuming Qx as lexical
and Qy as vector:
{noformat}
Rank(D,Q) & Score(D,Q) score(RRF(k=0))
Doc Qx QxScore Qy QyScore RRF
D1 1 127.3 2 0.96 1.5
D2 2 110.1 3 0.93 0.833
D3 3 105.1 4 0.9 0.583
D4 4 100.2 1 0.98 1.25
D5 5 98.4 8 0.67 0.325
D6 6 90.5 7 0.71 0.309
D7 7 88.6 6 0.75 0.309
D8 8 82.3 5 0.82 0.325
{noformat}
The correct final RRF reordering is still the same.
Now, let's distribute these documents across the shards. The documents are
distributed, and they have their absolute scores, causing the relative scores
to be lost. This assumes uneven vector distributions and the use of approximate
nearest neighbor (ANN) methods like HNSW.
{noformat}
shardA
Rank(D,Q) & Score(D,Q) score(RRF(k=0))
Doc Qx QxScore Qy QyScore
D1 1 105.2 2 0.91 1.5
D2 2 101.1 3 0.84 0.833
D3 3 97.3 4 0.80 0.583
D4 4 90.4 1 0.95 1.25
shardB
Rank(D,Q) & Score(D,Q) score(RRF(k=0))
Doc Qx QxScore Qy QyScore
D5 1 99.2 4 0.85 1.25
D6 2 95.1 3 0.88 0.833
D7 3 91.3 2 0.90 0.833
D8 4 88.4 1 0.93 1.25
{noformat}
Now, when you merge these results per query first, it looks something like:
{noformat}
Rank(D,Q) & Score(D,Q) score(RRF(k=0))
Doc Qx QxScore Qy QyScore RRF
D1 1 105.2 3 0.91 1.33
D2 2 101.1 7 0.84 0.642
D3 4 97.3 8 0.80 0.375
D4 7-tie 90.4 1 0.95 1.14
D5 3 99.2 6 0.85 0.5
D6 5 95.1 5 0.88 0.4
D7 6 91.3 4 0.90 0.416
D8 7-tie 90.4 2 0.93 0.642
{noformat}
Here, the RRF gets wrong as you are getting wrong relative scores than the
documents originally had.
Final RRF ranking, which I don't think matches 100% with expected either:
{noformat}
Way 2 Way 1 Expected
D1 D1 D1
D4 D2 D4
D2 (tie1) D4 D2
D8 (tie1) D3 (tie1) D3
D5 D5 (tie1) D5 (tie1)
D7 D6 (tie2) D8 (tie1)
D6 D7 (tie2) D6 (tie2)
D3 D8 (tie2) D7 (tie2){noformat}
The only difference here is RRF happening at the later state and combining the
docs per query across shards first.
Conclusively, I can say both method has some pitfalls when it comes to
multi-shard merging. I find "Way 1" relatively easier with a few limitations
(as compared to what David has defined in the Details section above) to
implement, given the Solr architecture of querying each shard first and then
merging them.
was (Author: ercsonu):
Please understand that the document's relative ranking is lost the moment it is
sharded. And It is not going to help getting the same expected document as in
the case of non-sharded, either RRF done locally to a shard or RRF done on
incorrect ordering received per query from all shards.
So, any approach that computes a "score" based on merging the per-shard results
first and then doing RRF _per query_ to find the topN results is also wrong
IMHO, as I didn't find any definition of RRF (or any algorithm taking rank as a
signal) over multiple shards. Please help me with a one if you came across.
Let me dissect the Way 2 with a similar example:
Start with the same settings and add some query score, assuming Qx as lexical
and Qy as vector:
{noformat}
Rank(D,Q) & Score(D,Q) score(RRF(k=0))
Doc Qx QxScore Qy QyScore RRF
D1 1 127.3 2 0.96 1.5
D2 2 110.1 3 0.93 0.833
D3 3 105.1 4 0.9 0.583
D4 4 100.2 1 0.98 1.25
D5 5 98.4 8 0.67 0.325
D6 6 90.5 7 0.71 0.309
D7 7 88.6 6 0.75 0.309
D8 8 82.3 5 0.82 0.325
{noformat}
The correct final RRF reordering is still the same.
Now, let's distribute these documents across the shards. The documents are
distributed, and they have their absolute scores, causing the relative scores
to be lost. This assumes uneven vector distributions and the use of approximate
nearest neighbor (ANN) methods like HNSW.
{noformat}
shardA
Rank(D,Q) & Score(D,Q) score(RRF(k=0))
Doc Qx QxScore Qy QyScore
D1 1 105.2 2 0.91 1.5
D2 2 101.1 3 0.84 0.833
D3 3 97.3 4 0.80 0.583
D4 4 90.4 1 0.95 1.25
shardB
Rank(D,Q) & Score(D,Q) score(RRF(k=0))
Doc Qx QxScore Qy QyScore
D5 1 99.2 4 0.85 1.25
D6 2 95.1 3 0.88 0.833
D7 3 91.3 2 0.90 0.833
D8 4 88.4 1 0.93 1.25
{noformat}
Now, when you merge these results per query first, it looks something like:
{noformat}
Rank(D,Q) & Score(D,Q) score(RRF(k=0))
Doc Qx QxScore Qy QyScore RRF
D1 1 105.2 3 0.91 1.33
D2 2 101.1 7 0.84 0.642
D3 4 97.3 8 0.80 0.375
D4 7-tie 90.4 1 0.95 1.14
D5 3 99.2 6 0.85 0.5
D6 5 95.1 5 0.88 0.4
D7 6 91.3 4 0.90 0.416
D8 7-tie 90.4 2 0.93 0.642
{noformat}
Here, the RRF gets wrong as you are getting wrong relative scores than the
documents originally had.
Final RRF ranking, which I don't think is correct either:
{noformat}
Way 2 Way 1 Expected
D1 D1 D1
D4 D2 D4
D2 (tie1) D4 D2
D8 (tie1) D3 (tie1) D3
D5 D5 (tie1) D5 (tie1)
D7 D6 (tie2) D8 (tie1)
D6 D7 (tie2) D6 (tie2)
D3 D8 (tie2) D7 (tie2){noformat}
The only difference here is RRF happening at the later state and combining the
docs per query across shards first.
Conclusively, I can say both method has some pitfalls when it comes to
multi-shard merging. I find "Way 1" relatively easier with a few limitations
(as compared to what David has defined in the Details section above) to
implement, given the Solr architecture of querying each shard first and then
merging them.
> Introduce support for Reciprocal Rank Fusion (combining queries)
> ----------------------------------------------------------------
>
> Key: SOLR-17319
> URL: https://issues.apache.org/jira/browse/SOLR-17319
> Project: Solr
> Issue Type: New Feature
> Components: vector-search
> Affects Versions: 9.6.1
> Reporter: Alessandro Benedetti
> Assignee: Alessandro Benedetti
> Priority: Major
> Labels: pull-request-available
> Time Spent: 23h 10m
> Remaining Estimate: 0h
>
> Reciprocal Rank Fusion (RRF) is an algorithm that takes in input multiple
> ranked lists to produce a unified result set.
> Examples of use cases where RRF can be used include hybrid search and
> multiple Knn vector queries executed concurrently.
> RRF is based on the concept of reciprocal rank, which is the inverse of the
> rank of a document in a ranked list of search results.
> The combination of search results happens taking into account the position of
> the items in the original rankings, and giving higher score to items that
> are ranked higher in multiple lists. RRF was introduced the first time by
> Cormack et al. in [1].
> The syntax proposed:
> JSON Request
> {code:json}
> {
> "queries": {
> "lexical1": {
> "lucene": {
> "query": "id:(10^=2 OR 2^=1 OR 4^=0.5)"
> }
> },
> "lexical2": {
> "lucene": {
> "query": "id:(2^=2 OR 4^=1 OR 3^=0.5)"
> }
> }
> },
> "limit": 10,
> "fields": "[id,score]",
> "params": {
> "combiner": true,
> "combiner.upTo": 5,
> "facet": true,
> "facet.field": "id",
> "facet.mincount": 1
> }
> }
> {code}
> [1] Cormack, Gordon V. et al. “Reciprocal rank fusion outperforms condorcet
> and individual rank learning methods.” Proceedings of the 32nd international
> ACM SIGIR conference on Research and development in information retrieval
> (2009)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
