Ganesha S created SPARK-54836:
---------------------------------
Summary: Include timestamp value in ArithmeticException for
timestamp overflow errors
Key: SPARK-54836
URL: https://issues.apache.org/jira/browse/SPARK-54836
Project: Spark
Issue Type: Task
Components: SQL
Affects Versions: 4.1.0
Reporter: Ganesha S
Currently, when a timestamp overflow occurs during interval addition
operations, Spark throws a generic ArithmeticException with only "long
overflow" message, making it difficult for users to debug which timestamp
values caused the issue.
{code:java}
java.lang.ArithmeticException: long overflow
at java.base/java.lang.Math.addExact(Math.java:903)
at
org.apache.spark.sql.catalyst.util.SparkDateTimeUtils.instantToMicros(SparkDateTimeUtils.scala:144)
at
org.apache.spark.sql.catalyst.util.SparkDateTimeUtils.instantToMicros$(SparkDateTimeUtils.scala:137)
at
org.apache.spark.sql.catalyst.util.DateTimeUtils$.instantToMicros(DateTimeUtils.scala:41)
at
org.apache.spark.sql.catalyst.util.DateTimeUtils$.timestampAddInterval(DateTimeUtils.scala:319)
at
org.apache.spark.sql.catalyst.util.DateTimeUtils.timestampAddInterval(DateTimeUtils.scala)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificPredicate.subExpr_13$(Unknown
Source)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificPredicate.eval(Unknown
Source)
at
org.apache.spark.sql.execution.FilterEvaluatorFactory$FilterPartitionEvaluator.$anonfun$eval$1(FilterEvaluatorFactory.scala:42)
at
org.apache.spark.sql.execution.FilterEvaluatorFactory$FilterPartitionEvaluator.$anonfun$eval$1$adapted(FilterEvaluatorFactory.scala:41)
at scala.collection.Iterator$$anon$6.hasNext(Iterator.scala:479)
at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:583)
at
org.apache.spark.sql.execution.collect.UnsafeRowBatchUtils$.$anonfun$encodeUnsafeRows$5(UnsafeRowBatchUtils.scala:88)
{code}
We should enhance the error message to include a timestamp value in the
ArithmeticException for timestamp overflow errors, facilitating easier
debugging.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
