Josh Rosen created SPARK-58431:
----------------------------------
Summary: SimplifyExtractValueOps folds an out-of-bounds array
index to NULL under ANSI, suppressing INVALID_ARRAY_INDEX and corrupting the
result's nullability
Key: SPARK-58431
URL: https://issues.apache.org/jira/browse/SPARK-58431
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.0
Reporter: Josh Rosen
This is a report of a correctness bug reachable under default configuration on
Spark 4.x (ANSI mode is now enabled by default). A query that must raise
{{INVALID_ARRAY_INDEX}} instead returns a fabricated value of {{{}0{}}},
silently.
{{SimplifyExtractValueOps}} rewrites {{array(...)[i]}} when the index is a
literal ({{{}ComplexTypes.scala{}}}):
{code:java}
case ga @ GetArrayItem(CreateArray(elems, _), IntegerLiteral(idx), _) =>
if (idx >= 0 && idx < elems.size) {
elems(idx)
} else {
// out of bounds, mimic the runtime behavior and return null
Literal(null, ga.dataType)
} {code}
The comment states the intent: mimic the runtime behavior. That was accurate
when the rule was written (SPARK-18601, 2.2.0), but SPARK-33386 (3.1.1) changed
the runtime behavior: under ANSI, an out-of-bounds array access _raises_ rather
than returning null. The rule was never updated, and it does not consult
{{GetArrayItem.failOnError}} (which defaults to
{{{}SQLConf.get.ansiEnabled{}}}).
This causes two problems:
* The error is suppressed.
* Under ANSI {{GetArrayItem}} declares itself *non-nullable* (it throws
instead of returning null), so substituting a null literal places a NULL in a
slot the schema marks non-nullable. Depending on the downstream consumer, this
can return either NULL (incorrect under ANSI) or as the type's zero value (also
incorrect).
*Repro* (default configuration on 4.x; add {{SET spark.sql.ansi.enabled=true}}
on older versions):
{code:java}
SET spark.sql.ansi.enabled=true;
CREATE TABLE arr_src(c INT) USING parquet;
INSERT INTO arr_src VALUES (1);
SELECT array(c, 2, 3)[5] AS v FROM arr_src;
-- prints: NULL (expected under ANSI: INVALID_ARRAY_INDEX error)
SELECT count(*) FROM arr_src WHERE array(c, 2, 3)[5] IS NULL;
-- prints: 0 (the value that just printed NULL matches no IS NULL
filter)
SELECT array(c, 2, 3)[5] IS NULL AS n FROM arr_src;
-- prints: false
SELECT sum(array(c,2,3)[5]) AS s, count(array(c,2,3)[5]) AS n FROM arr_src;
-- prints: NULL, 1 (count treats it as one NON-null value; sum of it is
null) {code}
No consistent value can produce those four outputs together. {*}Which symptom
the corrupted slot surfaces as is read-path-dependent{*}: the spark-sql CLI's
row rendering honors the physical null bit and prints {{{}NULL{}}}, while
{{Dataset.collect()}} (Scala/Java/Python APIs) trusts the declared {{{}nullable
= false{}}}, elides the null check, and materializes a fabricated {{0.}}
Affected versions: the fold dates from 2.2.0, but it only became incorrect when
SPARK-33386 (3.1.1) made ANSI out-of-bounds access raise. So 3.1.1+ with ANSI
enabled, and {*}4.x by default{*}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
