[ https://issues.apache.org/jira/browse/SPARK-27282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-27282: --------------------------------- Labels: (was: correctness) > Spark incorrect results when using UNION with GROUP BY clause > ------------------------------------------------------------- > > Key: SPARK-27282 > URL: https://issues.apache.org/jira/browse/SPARK-27282 > Project: Spark > Issue Type: Bug > Components: Spark Shell, Spark Submit, SQL > Affects Versions: 2.3.2 > Environment: I'm using : > IntelliJ IDEA ==> 2018.1.4 > spark-sql and spark-core ==> 2.3.2.3.1.0.0-78 (for HDP 3.1) > scala ==> 2.11.8 > Reporter: Sofia > Priority: Blocker > > When using UNION clause after a GROUP BY clause in spark, the results > obtained are wrong. > The following example explicit this issue: > {code:java} > CREATE TABLE test_un ( > col1 varchar(255), > col2 varchar(255), > col3 varchar(255), > col4 varchar(255) > ); > INSERT INTO test_un (col1, col2, col3, col4) > VALUES (1,1,2,4), > (1,1,2,4), > (1,1,3,5), > (2,2,2,null); > {code} > I used the following code : > {code:java} > val x = Toolkit.HiveToolkit.getDataFromHive("test","test_un") > val y = x > .filter(col("col4")isNotNull) > .groupBy("col1", "col2","col3") > .agg(count(col("col3")).alias("cnt")) > .withColumn("col_name", lit("col3")) > .select(col("col1"), col("col2"), > col("col_name"),col("col3").alias("col_value"), col("cnt")) > val z = x > .filter(col("col4")isNotNull) > .groupBy("col1", "col2","col4") > .agg(count(col("col4")).alias("cnt")) > .withColumn("col_name", lit("col4")) > .select(col("col1"), col("col2"), > col("col_name"),col("col4").alias("col_value"), col("cnt")) > y.union(z).show() > {code} > And i obtained the following results: > ||col1||col2||col_name||col_value||cnt|| > |1|1|col3|5|1| > |1|1|col3|4|2| > |1|1|col4|5|1| > |1|1|col4|4|2| > Expected results: > ||col1||col2||col_name||col_value||cnt|| > |1|1|col3|3|1| > |1|1|col3|2|2| > |1|1|col4|4|2| > |1|1|col4|5|1| > But when i remove the last row of the table, i obtain the correct results. > {code:java} > (2,2,2,null){code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org