Re: Can we import an HNSW graph into lucene index ?

[Michael Froh]

Hi Anand,

Interesting that you should bring this up!

There was a talk just this week at Berlin Buzzwords talking about using
cuVS with Lucene: https://www.youtube.com/watch?v=qiW7iIDFJC0

>From that talk, it sounds like the folks at SearchScale have managed to
integrate cuVS as a custom codec under Lucene. There was also mention in
the talk that a CAGRA graph can be built using GPUs (to get some impressive
speedups), then they can be searched using CPU-based logic (so you don't
need to provision GPU hosts for your search fleet).

I wasn't able to find the code for the codec at
https://github.com/SearchScale/lucene-cuvs/tree/main, but Ishan and Noble
should be on this list and might be able to find it.

- Froh

On Fri, Jun 14, 2024 at 7:57 AM Benjamin Trent 
wrote:

> Anand,
>
> In short, I think it's feasible, but I don't think it's simple. I also
> don't think Lucene should directly provide an interface to the format
> that says "Give me the graph". You could have a custom writer that
> does this however.
>
> All formats are nominally based, so if your GPU merge format writes
> out the appropriate name and format, it should be readable.
>
> > One issue we have been running into is long build times with higher
> dimensional vectors.
>
> Are you building the graph with a single thread?
>
> What vector dimensions are you using?
>
> As an aside, building the graph via quantized vectors can help speed
> things up. Though I understand the desire to do graph building with a
> GPU.
>
> Very interesting ideas indeed Anand.
>
> Ben
>
> On Fri, Jun 14, 2024 at 4:49 AM Anand Kotriwal 
> wrote:
> >
> > Hi all,
> >
> > We extensively use Lucene and HNSW graph search capability for ANN
> searches.
> > One issue we have been running into is long build times with higher
> dimensional vectors. To address this, we are exploring ways where we can
> build the hnsw index on the GPU and merge it into an existing Lucene index
> to serve queries. For example, Nvidia's cuvs library supports building a
> CAGRA index and  transforming it into a hnswlib graph.
> >
> > My idea is - once the hnswgraph is built on the GPUs, we can import the
> graph. We need the graph vertices and their connections. We can then write
> it to a lucene compatible segment file format. We also map the docids to
> embeddings and update the fieldinfos.
> >
> > I would like feedback from the community on whether this sounds feasible
> and any implementation pointers you might have.
> >
> >
> > Thanks,
> > Anand Kotriwal
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Can we import an HNSW graph into lucene index ?

[Atri Sharma]

There is a module that integrates Lucene with CAGRA.

Ishan should be able to point you to the link.

On Fri, 14 Jun 2024 at 8:27 PM, Benjamin Trent 
wrote:

> Anand,
>
> In short, I think it's feasible, but I don't think it's simple. I also
> don't think Lucene should directly provide an interface to the format
> that says "Give me the graph". You could have a custom writer that
> does this however.
>
> All formats are nominally based, so if your GPU merge format writes
> out the appropriate name and format, it should be readable.
>
> > One issue we have been running into is long build times with higher
> dimensional vectors.
>
> Are you building the graph with a single thread?
>
> What vector dimensions are you using?
>
> As an aside, building the graph via quantized vectors can help speed
> things up. Though I understand the desire to do graph building with a
> GPU.
>
> Very interesting ideas indeed Anand.
>
> Ben
>
> On Fri, Jun 14, 2024 at 4:49 AM Anand Kotriwal 
> wrote:
> >
> > Hi all,
> >
> > We extensively use Lucene and HNSW graph search capability for ANN
> searches.
> > One issue we have been running into is long build times with higher
> dimensional vectors. To address this, we are exploring ways where we can
> build the hnsw index on the GPU and merge it into an existing Lucene index
> to serve queries. For example, Nvidia's cuvs library supports building a
> CAGRA index and  transforming it into a hnswlib graph.
> >
> > My idea is - once the hnswgraph is built on the GPUs, we can import the
> graph. We need the graph vertices and their connections. We can then write
> it to a lucene compatible segment file format. We also map the docids to
> embeddings and update the fieldinfos.
> >
> > I would like feedback from the community on whether this sounds feasible
> and any implementation pointers you might have.
> >
> >
> > Thanks,
> > Anand Kotriwal
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Can we import an HNSW graph into lucene index ?

[Benjamin Trent]

Anand,

In short, I think it's feasible, but I don't think it's simple. I also
don't think Lucene should directly provide an interface to the format
that says "Give me the graph". You could have a custom writer that
does this however.

All formats are nominally based, so if your GPU merge format writes
out the appropriate name and format, it should be readable.

> One issue we have been running into is long build times with higher 
> dimensional vectors.

Are you building the graph with a single thread?

What vector dimensions are you using?

As an aside, building the graph via quantized vectors can help speed
things up. Though I understand the desire to do graph building with a
GPU.

Very interesting ideas indeed Anand.

Ben

On Fri, Jun 14, 2024 at 4:49 AM Anand Kotriwal  wrote:
>
> Hi all,
>
> We extensively use Lucene and HNSW graph search capability for ANN searches.
> One issue we have been running into is long build times with higher 
> dimensional vectors. To address this, we are exploring ways where we can 
> build the hnsw index on the GPU and merge it into an existing Lucene index to 
> serve queries. For example, Nvidia's cuvs library supports building a CAGRA 
> index and  transforming it into a hnswlib graph.
>
> My idea is - once the hnswgraph is built on the GPUs, we can import the 
> graph. We need the graph vertices and their connections. We can then write it 
> to a lucene compatible segment file format. We also map the docids to 
> embeddings and update the fieldinfos.
>
> I would like feedback from the community on whether this sounds feasible and 
> any implementation pointers you might have.
>
>
> Thanks,
> Anand Kotriwal

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Can we import an HNSW graph into lucene index ?

[Anand Kotriwal]

Hi all,

We extensively use Lucene and HNSW graph search capability for ANN
searches.
One issue we have been running into is long build times with higher
dimensional vectors. To address this, we are exploring ways where we can
build the hnsw index on the GPU and merge it into an existing Lucene index
to serve queries. For example, Nvidia's cuvs library supports building a
CAGRA  index and  transforming it into a
hnswlib graph.

My idea is - once the hnswgraph is built on the GPUs, we can import the
graph. We need the graph vertices and their connections. We can then write
it to a lucene compatible segment file format. We also map the docids to
embeddings and update the fieldinfos.

I would like feedback from the community on whether this sounds feasible
and any implementation pointers you might have.


Thanks,
Anand Kotriwal