Phil Steitz (JIRA) a écrit :
I agree this is a corner case and the negative result is due to rounding. The question is, should we force the result to 0 when a negative value is returned by the computation?
Yes, I think this would be wiser than what I said in my previous comment. I forgot that people often compute the square root of such sums, mainly to compare them with some dimensioning threshold.
We really know that the value must be positive and that if it is negative, this is due to cancellation. Hence we could force the value in this case.
Just replace the return statement by : Math.amx(0, sumYY - sumXY * sumXY / sumXX); ... and fix my test case by replacing ther " < 1.0e-14" back to ">= 0". Sorry, I didn't reflect enough Luc --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
