Re: Experience re OpenAI embeddings in combination with Lucene vector search

2022-02-12 Thread Michael Wechner
Here a concrete example where I combine OpenAI model "text-similarity-ada-001" with Lucene vector search INPUT sentence: "What is your age this year?" Result sentences 1) How old are you this year?    score '0.98860765' 2) What was your age last year?    score '0.97811764' 3) What is your ag

How to Increase max vector size?

2022-02-12 Thread Michael Wechner
Hi I just tried to test the OpenAI model "text-similarity-davinci-001" with 12288 dimensions and receive the following error java.lang.IllegalArgumentException: vector numDimensions must be <= VectorValues.MAX_DIMENSIONS (=1024); got 12288     at org.apache.lucene.document.FieldType.setV

Re: Experience re OpenAI embeddings in combination with Lucene vector search

2022-02-12 Thread Michael Wechner
Hi Alessandro I am mainly interested in detecting similarity, for example whether the following two sentences are similar resp. likely to mean the same thing "How old are you?" "What is your age?" and that the following two sentences are not similar, resp. do not mean the same thing "How o

Re: Experience re OpenAI embeddings in combination with Lucene vector search

2022-02-12 Thread Alessandro Benedetti
Hi Michael, experience to what extent? We have been exploring the area for a while given we contributed the first neural search milestone to Apache Solr. What is your curiosity? Performance? Relevance impact? How to integrate it? Regards On Fri, 11 Feb 2022, 22:38 Michael Wechner, wrote: > Hi >

Re: Monitor enhancements

2022-02-12 Thread Alan Woodward
These sound like nice improvements to me! I don’t know that anyone is actively working on Monitor stuff at the moment so please feel free open PRs or create patches. > On 11 Feb 2022, at 08:41, Niko Usai wrote: > > Hi all, > > I'm working on a project where I use very heavily Lucene Monitor p