benwtrent commented on issue #13519: URL: https://github.com/apache/lucene/issues/13519#issuecomment-2208862938
@naveentatikonda this is a WIP :). > Why are we setting SIGNED_CORRECTION as 127 instead of 128 ? My logic here was scaling by what we will round by. But thinking about it, I think we should scale down by 128. > ...the min should be -128 I disagree. We want `-127, 127`. Otherwise we have to handle the exotic `-128*-128` case when doing vector comparisons and this disallows some nice SIMD optimizations down the road. > In this new code we are not adding this... This is a wip...I didn't figure out yet how to incorporate this rounding error as it assumes unsigned operations and has a knock-on effect on the buckets. So, simply adding the naive rounding error would reduce recall significantly. I am wondering how to apply it. Will report back if I can figure it out :). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
