skambha commented on pull request #27627: URL: https://github.com/apache/spark/pull/27627#issuecomment-635730700
I wanted to summarize where we are at. The changes up to this point took care of whole stage enabled, disabled and ansi enabled/disabled scenarios etc consistently. But based on the discussion with @cloud-fan , we do not want to do the overflow checks for the intermediate results so the decision is to scale back the fix to - handle the case where it won’t return incorrect results when ansi enabled is false - for other codepaths that are not handled, we throw exceptions and not incorrect results which is similar behavior as before. - There is potential for non-deterministic behavior for sum when there is a overflow. Although not ideal, it is a step forward and it will not return incorrect results for some scenarios. Please see this pr thread https://github.com/skambha/spark/pull/1 for the discussion. I will push the following changes after this: - Merged @cloud-fan's changes at https://github.com/skambha/spark/pull/1 and fix for test failure. - The only changes on top of it is I have cleaned up the code comments to reflect the current state ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
