andygrove commented on issue #2021: URL: https://github.com/apache/datafusion-comet/issues/2021#issuecomment-3156059148
> Initial notes : > > * BinaryExpr (e.g., for addition, subtraction, and other arithmetic operations) includes a with_fail_on_overflow option, which eagerly returns an error upon detecting overflow. While this behavior is desirable in strict modes like ANSI, it limits our ability to handle both successful and overflow cases gracefully—unlike Spark, which returns NULL on overflow in ANSI mode. > * Based on recent discussions with the community on Discord, it appears that supporting ANSI mode behavior directly in native DataFusion (and Arrow-RS) is currently non-trivial or unsupported. > * Given these constraints, the most practical path forward seems to be creating a custom UDF (which is similar to BinaryExpr ) that uses checked_add, checked_div, etc., to return NULL on overflow. This would allow us to preserve DataFusion’s existing infrastructure while aligning more closely with Spark’s ANSI semantics. > * Another option would be to make changes in the DataFusion repo itself > > Would love to hear your thoughts / opinions on this direction, [@andygrove](https://github.com/andygrove) That sounds like a good plan @coderfender. Thanks for the detailed write-up on this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
