On 7/4/06, Luc Maisonobe <[EMAIL PROTECTED]> wrote:
Phil Steitz (JIRA) a écrit :
> I agree this is a corner case and the negative result is due to rounding.
The question is, should we force the result to 0 when a negative value is returned
by the computation?
Yes, I think this would be wiser than what I said in my previous
comment. I forgot that people often compute the square root of such
sums, mainly to compare them with some dimensioning threshold.
We really know that the value must be positive and that if it is
negative, this is due to cancellation. Hence we could force the value in
this case.
Just replace the return statement by :
Math.amx(0, sumYY - sumXY * sumXY / sumXX);
Sounds good.
... and fix my test case by replacing ther " < 1.0e-14" back to ">= 0".
Yes.
Sorry, I didn't reflect enough
No worries. The important thing is finding the best resolution that
we can for the issue and documenting any special behavior. In this
case, if we were to leave it as is, we would have to warn that
negative results could be returned. In general, we try to respect
invariants implicit in definitions, like that a sum of squares or a
probability cannot be negative. I will go ahead and make the change
above. Thanks for looking into this.
Phil
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
