[
https://issues.apache.org/jira/browse/LUCENE-10471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17579874#comment-17579874
]
Robert Muir commented on LUCENE-10471:
--------------------------------------
My main concern is that it can't be undone, as i mentioned. Nobody will be
willing to go backwards.
It impacts more than current implementation, it impacts future implementations
as well (different algorithms and data structures).
If something like 16k dimensions are allowed it may prevent even simple
optimizations (such as 8-bit width).
So its important to be very conservative.
This is why I make a big deal about it, because of the "one-way" nature of the
backwards compatibility associated with this change. It seems this is still not
yet understood or appreciated.
Historically, users fight against every limit we have in lucene, so when people
complain about this one, it doesn't bother me (esp when it seems related to one
or two bad models/bad decisions unrelated to this project). But these limits
are important, especially when features are in their infancy, without them,
there is less flexibility and you can find yourself easily "locked in" to a
particular implementation.
> Increase the number of dims for KNN vectors to 2048
> ---------------------------------------------------
>
> Key: LUCENE-10471
> URL: https://issues.apache.org/jira/browse/LUCENE-10471
> Project: Lucene - Core
> Issue Type: Wish
> Reporter: Mayya Sharipova
> Priority: Trivial
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The current maximum allowed number of dimensions is equal to 1024. But we see
> in practice a couple well-known models that produce vectors with > 1024
> dimensions (e.g
> [mobilenet_v2|https://tfhub.dev/google/imagenet/mobilenet_v2_035_224/feature_vector/1]
> uses 1280d vectors, OpenAI / GPT-3 Babbage uses 2048d vectors). Increasing
> max dims to `2048` will satisfy these use cases.
> I am wondering if anybody has strong objections against this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
