jiangxintong created SPARK-58235:
------------------------------------
Summary: [SQL] Fix Double/Float to Timestamp silent overflow
clamping in non-ANSI mode
Key: SPARK-58235
URL: https://issues.apache.org/jira/browse/SPARK-58235
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.0
Reporter: jiangxintong
h2. Background
In non-ANSI mode ({{spark.sql.ansi.enabled=false}}), casting a Double or Float
value that overflows the Long range to Timestamp silently clamps to
{{Long.MAX_VALUE}} or {{Long.MIN_VALUE}} instead of returning {{NULL}}.
The interpreted path ({{Cast.doubleToTimestamp}}) directly computes {{(d *
MICROS_PER_SECOND).toLong}} without overflow detection. The codegen path
similarly casts without checking: {{(long)($c * $MICROS_PER_SECOND)}}. When the
double multiplication exceeds Long range, the unchecked cast to long silently
produces a clamped value instead of NULL.
This is inconsistent with other overflow behavior in non-ANSI mode — for
example, casting Timestamp to Byte/Short/Int returns {{NULL}} on overflow, and
ANSI mode correctly throws {{CAST_OVERFLOW}} via {{DoubleExactNumeric.toLong}}.
h2. Repro
{code:sql}
SET spark.sql.ansi.enabled = false;
SELECT CAST(1e20 AS TIMESTAMP);
-- Before fix: +294247-01-10 04:00:54.775807 (silently clamped to
Long.MAX_VALUE)
-- After fix: NULL
SELECT CAST(-1e20 AS TIMESTAMP);
-- Before fix: -290308-12-21 19:59:05.224192 (silently clamped to
Long.MIN_VALUE)
-- After fix: NULL
{code}
Note: NaN and Infinity are already handled by the existing guard ({{d.isNaN ||
d.isInfinite}} in the interpreted path, JDK methods in codegen) and are
unaffected by this fix.
h2. Fix
Add overflow detection before converting to Long in both paths:
* *Interpreted*: check if the multiplication result exceeds Long range before
calling {{.toLong}}, then return NULL in non-ANSI mode (via {{errorOrNull}}) or
throw {{CAST_OVERFLOW}} in ANSI mode
* *Codegen*: same range check for Double and Float branches in
{{castToTimestampCode}}, then set {{$evNull = true}} in non-ANSI mode
The overflow conditions use the same pattern as other Spark overflow guards:
check the double result against Long range boundaries, then respect the ANSI
mode setting.
h2. Testing
* {{CastWithAnsiOffSuite}} (138 tests): non-ANSI overflow (positive/negative),
boundary values (9223372036854.0 safe / 9223372036855.0 overflow), NaN,
Infinity, normal values, zero
* {{CastWithAnsiOnSuite}} (136 tests): ANSI overflow (CAST_OVERFLOW), boundary
values, normal values
* Both Double and Float paths covered
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
