naveentatikonda commented on issue #13519: URL: https://github.com/apache/lucene/issues/13519#issuecomment-2209645562
> I disagree. We want `-127, 127`. Otherwise we have to handle the exotic `-128*-128` case when doing vector comparisons and this disallows some nice SIMD optimizations down the road. > @benwtrent Sorry, didn't get it. Can you pls share more details or links related to this. > * Applies a modification of the rounding error compensation. The idea behind how I applied the different compensation now is that we need to handle the linear scaling as this can cause us to snap to different buckets. I am still not 100% sure my transformation is correct there. @MilindShyani Can you check and comment on this. `(dx - dxq) * (dxq - SIGNED_CORRECTION * alpha)` is the rounding error compensation added by Ben. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org