Instead of doing double math, new PR to limit vector components to smaller than 1E17 to prevent overflow: https://github.com/apache/lucene/pull/12373
On Fri, Jun 9, 2023 at 4:38 PM Jonathan Ellis <jbel...@gmail.com> wrote: > Hi all, > > I ran into a bug where the cosine of a large vector taken with itself > returned NaN. (Cosine of equal vectors should always be 1.) I put > together a PR to do the internal math of the cosine function with double, > before returning the result as a float: > https://github.com/apache/lucene/pull/12281 > > -- > Jonathan Ellis > co-founder, http://www.datastax.com > @spyced > -- Jonathan Ellis co-founder, http://www.datastax.com @spyced
