LukaZdravic commented on code in PR #59239:
URL: https://github.com/apache/spark/pull/59239#discussion_r4193742056
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/BinByExec.scala:
##########
Review Comment:
Should this be MathUtils.subtractExact? For a very long range this throws
the raw java overflow error never mentioning spark.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/BinByExec.scala:
##########
@@ -122,7 +122,8 @@ case class BinByExec(
} else if (rs == re) {
val (k, binStart) =
DateTimeUtils.timeBucketFromTimestampDTInterval(width, rs, origin,
zone)
- val binEnd = DateTimeUtils.timeBucketFromIndexDTInterval(width,
k + 1, origin, zone)
+ val nextK = MathUtils.addExact(k, 1L)
+ val binEnd = DateTimeUtils.timeBucketFromIndexDTInterval(width,
nextK, origin, zone)
Review Comment:
`ARITHMETIC_OVERFLOW` error doesn't precisely say what the issue is and it
recommends to set `spark.sql.ansi.enabled` to false to bypass this error, but
`MathUtils.addExact` throws the error no matter what this flag is.
My proposal is to wrap this into a try-catch block and catch the case of
`ArithmeticException` and call the helper that you would write in
`QueryExecutionErrors.scala` and that way to throw the `DATETIME_OVERFLOW`
error instead, with more precise issue.
Also wrap the bin start lookups on lines 123 and 137 since with `BIN WIDTH
INTERVAL` of 1 day it can overflow too.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/BinByExec.scala:
##########
Review Comment:
Maybe apply the same as above, for symmetry. `k+1` here can wrap too if the
`ALIGN TO` is before the 1970 with a 1 microsecond width; it throws the
`ARITHMETIC_OVERFLOW`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
