Josh Rosen created SPARK-58484:
----------------------------------
Summary: CSV scan fails with ArrayIndexOutOfBoundsException when a
pushed filter has no column references and the required schema is empty
Key: SPARK-58484
URL: https://issues.apache.org/jira/browse/SPARK-58484
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.0
Reporter: Josh Rosen
The query
CREATE TABLE m(c0 INT) USING CSV;
INSERT INTO m VALUES (1);
SELECT COUNT(*) FROM m WHERE (SELECT BOOL_OR(true) FROM m);
crashes at execution time with
{code:java}
java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
at
org.apache.spark.sql.catalyst.OrderedFilters.$anonfun$predicates$3(OrderedFilters.scala:63)
at
org.apache.spark.sql.catalyst.OrderedFilters.$anonfun$predicates$3$adapted(OrderedFilters.scala:50)
at scala.collection.immutable.List.foreach(List.scala:323)
at
org.apache.spark.sql.catalyst.OrderedFilters.<init>(OrderedFilters.scala:50)
at
org.apache.spark.sql.catalyst.csv.UnivocityParser.<init>(UnivocityParser.scala:119)
[...]{code}
The WHERE clause is an uncorrelated boolean scalar subquery referencing no
columns.
Since SPARK-43402 such predicates are retained as data filters; at execution
time the finished subquery is substituted as {{Literal(true)}} and translated
to {{sources.AlwaysTrue}} , which is a filter with no references.
With {{COUNT(*)}} the required schema is empty. {{}}
{{OrderedFilters}} sizes its predicate-group arrays by
{{requiredSchema.fields.length}} (which is 0 in this case) but assigns
reference-free filters to index 0, so {{groupedFilters(0)}} throws. The
reordering step [immediately
below|https://github.com/apache/spark/blob/4dcdd4dba544210898180a410b8b30e0b85157cb/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/OrderedFilters.scala#L65]
already guards {{{}len > 0{}}}; the assignment loop does not. {{JsonFilters}}
handles the same input by mapping reference-free filters over the (empty)
field-name set, which is why JSON does not crash on a similar query.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
