Bruce Robbins created SPARK-39061:
-------------------------------------
Summary: Incorrect results or NPE when using Inline function
against an array of dynamically created structs
Key: SPARK-39061
URL: https://issues.apache.org/jira/browse/SPARK-39061
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.2.1, 3.1.3, 3.3.0, 3.4.0
Reporter: Bruce Robbins
The following query returns incorrect results:
{noformat}
spark-sql> select inline(array(named_struct('a', 1, 'b', 2), null));
1 2
-1 -1
Time taken: 4.053 seconds, Fetched 2 row(s)
spark-sql>
{noformat}
In Hive, the last row is {{NULL, NULL}}:
{noformat}
Beeline version 2.3.9 by Apache Hive
0: jdbc:hive2://localhost:10000> select inline(array(named_struct('a', 1, 'b',
2), null));
+-------+-------+
| a | b |
+-------+-------+
| 1 | 2 |
| NULL | NULL |
+-------+-------+
2 rows selected (1.355 seconds)
0: jdbc:hive2://localhost:10000>
{noformat}
If the struct has string fields, you get a {{NullPointerException}}:
{noformat}
spark-sql> select inline(array(named_struct('a', '1', 'b', '2'), null));
22/04/28 16:51:54 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 2)
java.lang.NullPointerException: null
at
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.write(UnsafeWriter.java:110)
~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.generate_doConsume_0$(Unknown
Source) ~[?:?]
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
Source) ~[?:?]
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
~[spark-sql_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
{noformat}
(Note: In Spark 3.1.3, both examples result in NPE).
You can work around the issue by casting the null entry of the array:
{noformat}
spark-sql> select inline(array(named_struct('a', 1, 'b', 2), cast(null as
struct<a:int, b:int>)));
1 2
NULL NULL
Time taken: 0.068 seconds, Fetched 2 row(s)
spark-sql>
{noformat}
(Note: In Spark 3.1.3, the above workaround does not work).
As far as I can tell, this issue only happens with arrays of structs where the
structs are created in an inline table or in a projection.
The fields of the struct are not getting set to {{nullable = true}} when there
is no example in the array where the field is set to {{null}}. As a result,
{{GenerateUnsafeProjection.createCode}} generates bad code: it has no code to
create a row of null columns, so it just creates a row from variables set with
default values.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
