On 4/24/22 15:01, Paul Brook wrote:
The abs1 function in ops_sse.h only works sorrectly when the result fits
in a signed int. This is fine most of the time because we're only dealing
with byte sized values.
However pcmp_elen helper function uses abs1 to calculate the absolute value
of a cpu register. This incorrectly truncates to 32 bits, and will give
the wrong anser for the most negative value.
Fix by open coding the saturation check before taking the absolute value.
Signed-off-by: Paul Brook <p...@nowt.org>
---
target/i386/ops_sse.h | 20 +++++++++-----------
1 file changed, 9 insertions(+), 11 deletions(-)
This works, since the bound comes first, so
Reviewed-by: Richard Henderson <richard.hender...@linaro.org>
+ if ((val > limit) || (val < -limit)) {
+ return limit;
+ }
+ return abs1(val);
But you could also have used uabs64() for one fewer compare.
r~
