[
https://issues.apache.org/jira/browse/SPARK-58484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Rosen updated SPARK-58484:
-------------------------------
Description:
The query
{code:java}
CREATE TABLE m(c0 INT) USING CSV;
INSERT INTO m VALUES (1);
SELECT COUNT FROM m WHERE (SELECT BOOL_OR(true) FROM m);{code}
crashes at execution time with
{code:java}
java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
at
org.apache.spark.sql.catalyst.OrderedFilters.$anonfun$predicates$3(OrderedFilters.scala:63)
at
org.apache.spark.sql.catalyst.OrderedFilters.$anonfun$predicates$3$adapted(OrderedFilters.scala:50)
at scala.collection.immutable.List.foreach(List.scala:323)
at
org.apache.spark.sql.catalyst.OrderedFilters.<init>(OrderedFilters.scala:50)
at
org.apache.spark.sql.catalyst.csv.UnivocityParser.<init>(UnivocityParser.scala:119)
[...]{code}
The WHERE clause is an uncorrelated boolean scalar subquery referencing no
columns.
Since SPARK-43402 such predicates are retained as data filters; at execution
time the finished subquery is substituted as {{Literal(true)}} and translated
to {{sources.AlwaysTrue}} , which is a filter with no references.
With {{COUNT(*)}} the required schema is empty. \{{}}
{{OrderedFilters}} sizes its predicate-group arrays by
{{requiredSchema.fields.length}} (which is 0 in this case) but assigns
reference-free filters to index 0, so {{groupedFilters(0)}} throws. The
reordering step [immediately
below|https://github.com/apache/spark/blob/4dcdd4dba544210898180a410b8b30e0b85157cb/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/OrderedFilters.scala#L65]
already guards {{{}len > 0{}}}; the assignment loop does not. {{JsonFilters}}
handles the same input by mapping reference-free filters over the (empty)
field-name set, which is why JSON does not crash on a similar query.
was:
The query
CREATE TABLE m(c0 INT) USING CSV;
INSERT INTO m VALUES (1);
SELECT COUNT(*) FROM m WHERE (SELECT BOOL_OR(true) FROM m);
crashes at execution time with
{code:java}
java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
at
org.apache.spark.sql.catalyst.OrderedFilters.$anonfun$predicates$3(OrderedFilters.scala:63)
at
org.apache.spark.sql.catalyst.OrderedFilters.$anonfun$predicates$3$adapted(OrderedFilters.scala:50)
at scala.collection.immutable.List.foreach(List.scala:323)
at
org.apache.spark.sql.catalyst.OrderedFilters.<init>(OrderedFilters.scala:50)
at
org.apache.spark.sql.catalyst.csv.UnivocityParser.<init>(UnivocityParser.scala:119)
[...]{code}
The WHERE clause is an uncorrelated boolean scalar subquery referencing no
columns.
Since SPARK-43402 such predicates are retained as data filters; at execution
time the finished subquery is substituted as {{Literal(true)}} and translated
to {{sources.AlwaysTrue}} , which is a filter with no references.
With {{COUNT(*)}} the required schema is empty. {{}}
{{OrderedFilters}} sizes its predicate-group arrays by
{{requiredSchema.fields.length}} (which is 0 in this case) but assigns
reference-free filters to index 0, so {{groupedFilters(0)}} throws. The
reordering step [immediately
below|https://github.com/apache/spark/blob/4dcdd4dba544210898180a410b8b30e0b85157cb/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/OrderedFilters.scala#L65]
already guards {{{}len > 0{}}}; the assignment loop does not. {{JsonFilters}}
handles the same input by mapping reference-free filters over the (empty)
field-name set, which is why JSON does not crash on a similar query.
> CSV scan fails with ArrayIndexOutOfBoundsException when a pushed filter has
> no column references and the required schema is empty
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-58484
> URL: https://issues.apache.org/jira/browse/SPARK-58484
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Josh Rosen
> Priority: Major
>
> The query
> {code:java}
> CREATE TABLE m(c0 INT) USING CSV;
> INSERT INTO m VALUES (1);
> SELECT COUNT FROM m WHERE (SELECT BOOL_OR(true) FROM m);{code}
> crashes at execution time with
> {code:java}
> java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
> at
> org.apache.spark.sql.catalyst.OrderedFilters.$anonfun$predicates$3(OrderedFilters.scala:63)
> at
> org.apache.spark.sql.catalyst.OrderedFilters.$anonfun$predicates$3$adapted(OrderedFilters.scala:50)
> at scala.collection.immutable.List.foreach(List.scala:323)
> at
> org.apache.spark.sql.catalyst.OrderedFilters.<init>(OrderedFilters.scala:50)
> at
> org.apache.spark.sql.catalyst.csv.UnivocityParser.<init>(UnivocityParser.scala:119)
> [...]{code}
> The WHERE clause is an uncorrelated boolean scalar subquery referencing no
> columns.
> Since SPARK-43402 such predicates are retained as data filters; at execution
> time the finished subquery is substituted as {{Literal(true)}} and translated
> to {{sources.AlwaysTrue}} , which is a filter with no references.
> With {{COUNT(*)}} the required schema is empty. \{{}}
> {{OrderedFilters}} sizes its predicate-group arrays by
> {{requiredSchema.fields.length}} (which is 0 in this case) but assigns
> reference-free filters to index 0, so {{groupedFilters(0)}} throws. The
> reordering step [immediately
> below|https://github.com/apache/spark/blob/4dcdd4dba544210898180a410b8b30e0b85157cb/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/OrderedFilters.scala#L65]
> already guards {{{}len > 0{}}}; the assignment loop does not.
> {{JsonFilters}} handles the same input by mapping reference-free filters over
> the (empty) field-name set, which is why JSON does not crash on a similar
> query.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
