Both what Allesandro said and what Bruno said: make it configurable and
move it. Both are good and not mutually exclusive and could happen in any
order.
Marcus, are you against configurability? In particular, I propose a
simple Integer.getInteger("lucene.hnsw.maxDimensions", 1024). Your response
suggests trying to somehow perfect what the _default_ limit should be, but
I've not seen an argument _against_ configurability. Especially in this
way -- a toggle that doesn't bind Lucene's APIs in any way.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Thu, May 11, 2023 at 7:59 AM Uwe Schindler <u...@thetaphi.de> wrote:
> That's actually a good idea.
>
> +1
> Am 10.05.2023 um 09:22 schrieb Bruno Roustant:
>
> *Proposed option:* Move the max dimension limit lower level to a HNSW
> specific implementation. Once there, this limit would not bind any other
> potential vector engine alternative/evolution.
>
> *Motivation:* There seem to be contradictory performance interpretations
> about the current HNSW implementation. Some consider its performance ok,
> some not, and it depends on the target data set and use-case. Increasing
> the max dimension limit where it is currently (in top level
> FloatVectorValues) would not allow potential alternatives (e.g. for other
> use-cases) to be based on a lower limit.
>
> Bruno
>
> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremenhttps://www.thetaphi.de
> eMail: u...@thetaphi.de
>
>
