I'm also in favor of raising this limit. We do see some datasets with
higher than 1024 dims. I also think we need to keep a limit. For example we
currently need to keep all the vectors in RAM while indexing and we want to
be able to support reasonable numbers of vectors in an index segment. Also
we don't know what innovations might come down the road. Maybe someday we
want to do product quantization and enforce that (k, m) both fit in a byte
-- we wouldn't be able to do that if a vector's dimension were to exceed
32K.
On Fri, Mar 31, 2023 at 11:57 AM Alessandro Benedetti <a.benede...@sease.io>
wrote:
> I am also curious what would be the worst-case scenario if we remove the
> constant at all (so automatically the limit becomes the Java
> Integer.MAX_VALUE).
> i.e.
> right now if you exceed the limit you get:
>
>> if (dimension > ByteVectorValues.MAX_DIMENSIONS) {
>> throw new IllegalArgumentException(
>> "cannot index vectors with dimension greater than " + ByteVectorValues.
>> MAX_DIMENSIONS);
>> }
>
>
> in relation to:
>
>> These limits allow us to
>> better tune our data structures, prevent overflows, help ensure we
>> have good test coverage, etc.
>
>
> I agree 100% especially for typing stuff properly and avoiding resource
> waste here and there, but I am not entirely sure this is the case for the
> current implementation i.e. do we have optimizations in place that assume
> the max dimension to be 1024?
> If I missed that (and I likely have), I of course suggest the contribution
> should not just blindly remove the limit, but do it appropriately.
> I am not in favor of just doubling it as suggested by some people, I would
> ideally prefer a solution that remains there to a decent extent, rather
> than having to modifying it anytime someone requires a higher limit.
>
> Cheers
>
> --------------------------
> *Alessandro Benedetti*
> Director @ Sease Ltd.
> *Apache Lucene/Solr Committer*
> *Apache Solr PMC Member*
>
> e-mail: a.benede...@sease.io
>
>
> *Sease* - Information Retrieval Applied
> Consulting | Training | Open Source
>
> Website: Sease.io <http://sease.io/>
> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
> <https://twitter.com/seaseltd> | Youtube
> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
> <https://github.com/seaseltd>
>
>
> On Fri, 31 Mar 2023 at 16:12, Michael Wechner <michael.wech...@wyona.com>
> wrote:
>
>> OpenAI reduced their size to 1536 dimensions
>>
>> https://openai.com/blog/new-and-improved-embedding-model
>>
>> so 2048 would work :-)
>>
>> but other services do provide also higher dimensions with sometimes
>> slightly better accuracy
>>
>> Thanks
>>
>> Michael
>>
>>
>> Am 31.03.23 um 14:45 schrieb Adrien Grand:
>> > I'm supportive of bumping the limit on the maximum dimension for
>> > vectors to something that is above what the majority of users need,
>> > but I'd like to keep a limit. We have limits for other things like the
>> > max number of docs per index, the max term length, the max number of
>> > dimensions of points, etc. and there are a few things that we don't
>> > have limits on that I wish we had limits on. These limits allow us to
>> > better tune our data structures, prevent overflows, help ensure we
>> > have good test coverage, etc.
>> >
>> > That said, these other limits we have in place are quite high. E.g.
>> > the 32kB term limit, nobody would ever type a 32kB term in a text box.
>> > Likewise for the max of 8 dimensions for points: a segment cannot
>> > possibly have 2 splits per dimension on average if it doesn't have
>> > 512*2^(8*2)=34M docs, a sizable dataset already, so more dimensions
>> > than 8 would likely defeat the point of indexing. In contrast, our
>> > limit on the number of dimensions of vectors seems to be under what
>> > some users would like, and while I understand the performance argument
>> > against bumping the limit, it doesn't feel to me like something that
>> > would be so bad that we need to prevent users from using numbers of
>> > dimensions in the low thousands, e.g. top-k KNN searches would still
>> > look at a very small subset of the full dataset.
>> >
>> > So overall, my vote would be to bump the limit to 2048 as suggested by
>> > Mayya on the issue that you linked.
>> >
>> > On Fri, Mar 31, 2023 at 2:38 PM Michael Wechner
>> > <michael.wech...@wyona.com> wrote:
>> >> Thanks Alessandro for summarizing the discussion below!
>> >>
>> >> I understand that there is no clear reasoning re what is the best
>> embedding size, whereas I think heuristic approaches like described by the
>> following link can be helpful
>> >>
>> >>
>> https://datascience.stackexchange.com/questions/51404/word2vec-how-to-choose-the-embedding-size-parameter
>> >>
>> >> Having said this, we see various embedding services providing higher
>> dimensions than 1024, like for example OpenAI, Cohere and Aleph Alpha.
>> >>
>> >> And it would be great if we could run benchmarks without having to
>> recompile Lucene ourselves.
>> >>
>> >> Therefore I would to suggest to either increase the limit or even
>> better to remove the limit and add a disclaimer, that people should be
>> aware of possible crashes etc.
>> >>
>> >> Thanks
>> >>
>> >> Michael
>> >>
>> >>
>> >>
>> >>
>> >> Am 31.03.23 um 11:43 schrieb Alessandro Benedetti:
>> >>
>> >>
>> >> I've been monitoring various discussions on Pull Requests about
>> changing the max number of dimensions allowed for Lucene HNSW vectors:
>> >>
>> >> https://github.com/apache/lucene/pull/12191
>> >>
>> >> https://github.com/apache/lucene/issues/11507
>> >>
>> >>
>> >> I would like to set up a discussion and potentially a vote about this.
>> >>
>> >> I have seen some strong opposition from a few people but a majority of
>> favor in this direction.
>> >>
>> >>
>> >> Motivation
>> >>
>> >> We were discussing in the Solr slack channel with Ishan
>> Chattopadhyaya, Marcus Eagan, and David Smiley about some neural search
>> integrations in Solr: https://github.com/openai/chatgpt-retrieval-plugin
>> >>
>> >>
>> >> Proposal
>> >>
>> >> No hard limit at all.
>> >>
>> >> As for many other Lucene areas, users will be allowed to push the
>> system to the limit of their resources and get terrible performances or
>> crashes if they want.
>> >>
>> >>
>> >> What we are NOT discussing
>> >>
>> >> - Quality and scalability of the HNSW algorithm
>> >>
>> >> - dimensionality reduction
>> >>
>> >> - strategies to fit in an arbitrary self-imposed limit
>> >>
>> >>
>> >> Benefits
>> >>
>> >> - users can use the models they want to generate vectors
>> >>
>> >> - removal of an arbitrary limit that blocks some integrations
>> >>
>> >>
>> >> Cons
>> >>
>> >> - if you go for vectors with high dimensions, there's no guarantee
>> you get acceptable performance for your use case
>> >>
>> >>
>> >>
>> >> I want to keep it simple, right now in many Lucene areas, you can push
>> the system to not acceptable performance/ crashes.
>> >>
>> >> For example, we don't limit the number of docs per index to an
>> arbitrary maximum of N, you push how many docs you like and if they are too
>> much for your system, you get terrible performance/crashes/whatever.
>> >>
>> >>
>> >> Limits caused by primitive java types will stay there behind the
>> scene, and that's acceptable, but I would prefer to not have arbitrary
>> hard-coded ones that may limit the software usability and integration which
>> is extremely important for a library.
>> >>
>> >>
>> >> I strongly encourage people to add benefits and cons, that I missed (I
>> am sure I missed some of them, but wanted to keep it simple)
>> >>
>> >>
>> >> Cheers
>> >>
>> >> --------------------------
>> >> Alessandro Benedetti
>> >> Director @ Sease Ltd.
>> >> Apache Lucene/Solr Committer
>> >> Apache Solr PMC Member
>> >>
>> >> e-mail: a.benede...@sease.io
>> >>
>> >>
>> >> Sease - Information Retrieval Applied
>> >> Consulting | Training | Open Source
>> >>
>> >> Website: Sease.io
>> >> LinkedIn | Twitter | Youtube | Github
>> >>
>> >>
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>
