Sure, we can make it work but in a distributed environment you have to run first each query distributed (aggregating all nodes) and then RRF on top of the aggregated ranked lists. Doing RRF per node first and then aggregate per shard won't return the same results I suspect. When I go back to working on the task I'll be able to elaborate more!
Cheers -------------------------- *Alessandro Benedetti* Director @ Sease Ltd. *Apache Lucene/Solr Committer* *Apache Solr PMC Member* e-mail: a.benede...@sease.io *Sease* - Information Retrieval Applied Consulting | Training | Open Source Website: Sease.io <http://sease.io/> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter <https://twitter.com/seaseltd> | Youtube <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github <https://github.com/seaseltd> On Mon, 13 May 2024 at 14:12, Adrien Grand <jpou...@gmail.com> wrote: > > Maybe Adrien Grand and others might also have some feedback :-) > > I'd suggest the signature to look something like `TopDocs TopDocs#rrf(int > topN, int k, TopDocs[] hits)` to be consistent with `TopDocs#merge`. > Internally, it should look at `ScoreDoc#shardId` and `ScoreDoc#doc` to > figure out which hits map to the same document. > > > Back in the day, I was reasoning on this and I didn't think Lucene was > the right place for an interleaving algorithm, given that Reciprocal Rank > Fusion is affected by distribution and it's not supposed to work per node. > > To me this is like `TopDocs#merge`. There are changes needed on the > application side to hook this call into the logic that combines hits that > come from multiple shards (multiple queries in the case of RRF), but Lucene > can still provide the merging logic. > > On Mon, May 13, 2024 at 1:41 PM Michael Wechner <michael.wech...@wyona.com> > wrote: > >> Thanks for your feedback Alessandro! >> >> I am using Lucene independent of Solr or OpenSearch, Elasticsearch, but >> would like to combine different result sets using RRF, therefore think that >> Lucene itself could be a good place actually. >> >> Looking forward to your additional elaboration! >> >> Thanks >> >> Michael >> >> >> >> >> Am 13.05.2024 um 12:34 schrieb Alessandro Benedetti <a.benede...@sease.io >> >: >> >> This is not strictly related to Lucene, but I'll give a talk at Berlin >> Buzzwords on how I am implementing Reciprocal Rank Fusion in Apache Solr. >> I'll resume my work on the contribution next week and have more to share >> later. >> >> Back in the day, I was reasoning on this and I didn't think Lucene was >> the right place for an interleaving algorithm, given that Reciprocal Rank >> Fusion is affected by distribution and it's not supposed to work per node. >> I think I evaluated the possibility of doing it as a Lucene query or a >> Lucene component but then ended up with a different approach. >> I'll elaborate more when I go back to the task! >> >> Cheers >> -------------------------- >> *Alessandro Benedetti* >> Director @ Sease Ltd. >> *Apache Lucene/Solr Committer* >> *Apache Solr PMC Member* >> >> e-mail: a.benede...@sease.io >> >> >> *Sease* - Information Retrieval Applied >> Consulting | Training | Open Source >> >> Website: Sease.io <http://sease.io/> >> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter >> <https://twitter.com/seaseltd> | Youtube >> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github >> <https://github.com/seaseltd> >> >> >> On Sat, 11 May 2024 at 09:10, Michael Wechner <michael.wech...@wyona.com> >> wrote: >> >>> sure, no problem! >>> >>> Maybe Adrien Grand and others might also have some feedback :-) >>> >>> Thanks >>> >>> Michael >>> >>> Am 10.05.24 um 23:03 schrieb Chang Hank: >>> >>> Thank you for these useful resources, please allow me to spend some time >>> look into it. >>> I’ll let you know asap!! >>> >>> Thanks >>> >>> Hank >>> >>> On May 10, 2024, at 12:34 PM, Michael Wechner >>> <michael.wech...@wyona.com> <michael.wech...@wyona.com> wrote: >>> >>> also we might want to consider how this relates to >>> >>> >>> https://lucene.apache.org/core/9_10_0/core/org/apache/lucene/search/Rescorer.html >>> >>> In vector search reranking has become quite popular, e.g. >>> >>> https://docs.cohere.com/docs/reranking >>> >>> IIUC LangChain (python) for example adds the reranker as an argument to >>> the searcher/retriever >>> >>> >>> https://python.langchain.com/v0.1/docs/integrations/retrievers/cohere-reranker/ >>> >>> So maybe the following might make sense as well >>> >>> TopDocs topDocsKeyword = keywordSearcher.search(keywordQuery, 10); >>> TopDocs topDocsVector = vectorSearcher.search(query, 50, new >>> CohereReranker()); >>> >>> TopDocs topDocs = TopDocs.merge(new RRFRanker(), topDocsKeyword, >>> topDocsVector); >>> >>> WDYT? >>> >>> Thanks >>> >>> Michael >>> >>> >>> Am 10.05.24 um 21:08 schrieb Michael Wechner: >>> >>> great, yes, let's get started :-) >>> >>> What about the following pseudo code, assuming that there might be >>> alternative ranking algorithms to RRF >>> >>> StoredFieldsKeyword storedFieldsKeyword = >>> indexReaderKeyword.storedFields(); >>> StoredFieldsVector storedFieldsVector = >>> indexReaderKeyword.storedFields(); >>> >>> TopDocs topDocsKeyword = keywordSearcher.search(keywordQuery, 10); >>> TopDocs topDocsVector = vectorSearcher.search(vectorQuery, 50); >>> >>> Ranker ranker = new RRFRanker(); >>> TopDocs topDocs = TopDocs.rank(ranker, topDocsKeyword, topDocsVector); >>> >>> for (ScoreDoc scoreDoc : topDocs.scoreDocs) { >>> Document docK = storedFieldsKeyword.document(scoreDoc.doc); >>> Document docV = storedFieldsVector.document(scoreDoc.doc); >>> .... >>> } >>> >>> whereas also see >>> >>> >>> https://lucene.apache.org/core/9_10_0/core/org/apache/lucene/search/TopDocs.html >>> https://www.elastic.co/guide/en/elasticsearch/reference/current/rrf.html >>> >>> WDYT? >>> >>> Thanks >>> >>> Michael >>> >>> >>> >>> >>> Am 10.05.24 um 20:01 schrieb Chang Hank: >>> >>> Hi Michael, >>> >>> Sounds good to me. >>> Let’s do it!! >>> >>> Cheers, >>> Hank >>> >>> On May 10, 2024, at 10:50 AM, Michael Wechner >>> <michael.wech...@wyona.com> <michael.wech...@wyona.com> wrote: >>> >>> Hi Hank >>> >>> Very cool! >>> >>> Adrien Grand suggested to implement it as a utility method on the >>> TopDocs class, and since Adrien worked for a decade on Lucene >>> https://www.elastic.co/de/blog/author/adrien-grand I guess it makes >>> sense to follow his advice :-) We could create a PR and work together on >>> it, WDYT? All the best Michael >>> Am 10.05.24 um 18:51 schrieb Chang Hank: >>> >>> Hi Michael, >>> >>> Thank you for the reply. >>> This is really a cool issue to work on, I’m happy to work on this with >>> you. I’ll try to do research on RRF first. >>> Also, are we going to implement this on the TopDocs class? >>> >>> Best, >>> Hank >>> >>> >>> On May 9, 2024, at 11:08 PM, Michael Wechner <michael.wech...@wyona.com> >>> <michael.wech...@wyona.com> wrote: >>> >>> Hi Hank >>> >>> Thanks for offering your help! >>> >>> I recently suggested to implement RRF (Reciprocal Rank Fusion) >>> >>> https://lists.apache.org/thread/vvwvjl0gk67okn8z1wg33ogyf9qm07sz >>> >>> but still have not found the time to really work on this. >>> >>> Maybe you would be interested to do this or that we work on it together >>> somehow? >>> >>> Thanks >>> >>> Michael >>> >>> >>> >>> Am 10.05.24 um 07:27 schrieb Chang Hank: >>> >>> Hi everyone, >>> >>> I’m Hank Chang, currently studying Information Retrieval topics. I’m >>> really interested in contributing to Apache Lucene and enhance my >>> understanding to the field. >>> I’ve reviewed several issues posted on the Github repository but haven’t >>> found a straightforward starting point. Could someone please recommend >>> suitable issues for a newcomer like me or suggest areas I could assist with? >>> >>> Thank you for your time and guidance. >>> >>> Best regards, >>> Hank Chang >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: dev-h...@lucene.apache.org >>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: dev-h...@lucene.apache.org >>> >>> >>> >>> >>> >>> >>> >>> >>> >> > > -- > Adrien >
