Re: Can we import an HNSW graph into lucene index ?

2024-06-14 Thread Michael Froh
Hi Anand, Interesting that you should bring this up! There was a talk just this week at Berlin Buzzwords talking about using cuVS with Lucene: https://www.youtube.com/watch?v=qiW7iIDFJC0 >From that talk, it sounds like the folks at SearchScale have managed to integrate cuVS as a custom codec

Re: Can we import an HNSW graph into lucene index ?

2024-06-14 Thread Atri Sharma
There is a module that integrates Lucene with CAGRA. Ishan should be able to point you to the link. On Fri, 14 Jun 2024 at 8:27 PM, Benjamin Trent wrote: > Anand, > > In short, I think it's feasible, but I don't think it's simple. I also > don't think Lucene should directly provide an

Re: Can we import an HNSW graph into lucene index ?

2024-06-14 Thread Benjamin Trent
Anand, In short, I think it's feasible, but I don't think it's simple. I also don't think Lucene should directly provide an interface to the format that says "Give me the graph". You could have a custom writer that does this however. All formats are nominally based, so if your GPU merge format

Can we import an HNSW graph into lucene index ?

2024-06-14 Thread Anand Kotriwal
Hi all, We extensively use Lucene and HNSW graph search capability for ANN searches. One issue we have been running into is long build times with higher dimensional vectors. To address this, we are exploring ways where we can build the hnsw index on the GPU and merge it into an existing Lucene