[ https://issues.apache.org/jira/browse/SPARK-20534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Li updated SPARK-20534: ---------------------------- Fix Version/s: (was: 2.2.1) (was: 2.3.0) 2.2.0 > Outer generators skip missing records if used alone > --------------------------------------------------- > > Key: SPARK-20534 > URL: https://issues.apache.org/jira/browse/SPARK-20534 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.2.0 > Environment: master 814a61a867ded965433c944c90961df529ac83ab > Reporter: Maciej Szymkiewicz > Assignee: Maciej Szymkiewicz > Fix For: 2.2.0 > > > Example data: > {code} > val df = Seq( > (1, Some("a" :: "b" :: "c" :: Nil)), > (2, None), > (3, Some("a" :: Nil) > )).toDF("k", "vs") > {code} > Correct behavior if there are other expressions: > {code} > df.select($"k", explode_outer($"vs")).show > // +---+----+ > // | k| col| > // +---+----+ > // | 1| a| > // | 1| b| > // | 1| c| > // | 2|null| > // | 3| a| > // +---+----+ > df.select($"k", posexplode_outer($"vs")).show > // +---+----+----+ > // | k| pos| col| > // +---+----+----+ > // | 1| 0| a| > // | 1| 1| b| > // | 1| 2| c| > // | 2|null|null| > // | 3| 0| a| > // +---+----+----+ > {code} > Incorrect behavior if used alone: > {code} > df.select(explode_outer($"vs")).show > // +---+ > // |col| > // +---+ > // | a| > // | b| > // | c| > // | a| > // +---+ > df.select(posexplode_outer($"vs")).show > // +---+---+ > // |pos|col| > // +---+---+ > // | 0| a| > // | 1| b| > // | 2| c| > // | 0| a| > // +---+---+ > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org