[ https://issues.apache.org/jira/browse/SPARK-39061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17529702#comment-17529702 ]
Bruce Robbins commented on SPARK-39061: --------------------------------------- Btw, dataframe example: {noformat} scala> val df = Seq((1)).toDF.withColumn("b", array(struct(lit(1).alias("a"), lit(2).alias("a")), lit(null))) df: org.apache.spark.sql.DataFrame = [value: int, b: array<struct<a:int,a:int>>] scala> df.selectExpr("inline(b)").collect res2: Array[org.apache.spark.sql.Row] = Array([1,2], [-1,-1]) {noformat} > Incorrect results or NPE when using Inline function against an array of > dynamically created structs > --------------------------------------------------------------------------------------------------- > > Key: SPARK-39061 > URL: https://issues.apache.org/jira/browse/SPARK-39061 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.1.3, 3.2.1, 3.3.0, 3.4.0 > Reporter: Bruce Robbins > Priority: Major > Labels: correctness > > The following query returns incorrect results: > {noformat} > spark-sql> select inline(array(named_struct('a', 1, 'b', 2), null)); > 1 2 > -1 -1 > Time taken: 4.053 seconds, Fetched 2 row(s) > spark-sql> > {noformat} > In Hive, the last row is {{NULL, NULL}}: > {noformat} > Beeline version 2.3.9 by Apache Hive > 0: jdbc:hive2://localhost:10000> select inline(array(named_struct('a', 1, > 'b', 2), null)); > +-------+-------+ > | a | b | > +-------+-------+ > | 1 | 2 | > | NULL | NULL | > +-------+-------+ > 2 rows selected (1.355 seconds) > 0: jdbc:hive2://localhost:10000> > {noformat} > If the struct has string fields, you get a {{NullPointerException}}: > {noformat} > spark-sql> select inline(array(named_struct('a', '1', 'b', '2'), null)); > 22/04/28 16:51:54 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 2) > java.lang.NullPointerException: null > at > org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.write(UnsafeWriter.java:110) > ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT] > at > org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.generate_doConsume_0$(Unknown > Source) ~[?:?] > at > org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown > Source) ~[?:?] > at > org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) > ~[spark-sql_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT] > {noformat} > (Note: In Spark 3.1.3, both examples result in NPE). > You can work around the issue by casting the null entry of the array: > {noformat} > spark-sql> select inline(array(named_struct('a', 1, 'b', 2), cast(null as > struct<a:int, b:int>))); > 1 2 > NULL NULL > Time taken: 0.068 seconds, Fetched 2 row(s) > spark-sql> > {noformat} > (Note: In Spark 3.1.3, the above workaround does not work). > As far as I can tell, this issue only happens with arrays of structs where > the structs are created in an inline table or in a projection. > The fields of the struct are not getting set to {{nullable = true}} when > there is no example in the array where the field is set to {{null}}. As a > result, {{GenerateUnsafeProjection.createCode}} generates bad code: it has no > code to create a row of null columns, so it just creates a row from variables > set with default values. -- This message was sent by Atlassian Jira (v8.20.7#820007) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org