Linhong Liu created SPARK-60637:
-----------------------------------
Summary: Account for decimal coercion when deriving VALUES
nullability
Key: SPARK-60637
URL: https://issues.apache.org/jira/browse/SPARK-60637
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Linhong Liu
VALUES currently derives output nullability from the original expressions
rather than their coerced forms. With ANSI mode disabled and
spark.sql.legacy.decimal.retainFractionDigitsOnTruncate=true, combining a
38-digit integer with 0.1 can overflow the integer to NULL but leave the output
marked nonnullable. IS NULL and COUNT(column) can then return incorrect
results. Derive metadata from coerced expressions. For eagerly materialized
tables, retain nonnullable metadata for originally nonnullable columns when
every evaluated value is non-NULL, without evaluating expressions again. Cover
both eager evaluation settings and the runtime-expression Expand path.
CURRENT_LIKE-deferred tables may retain conservative nullability.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
