[ https://issues.apache.org/jira/browse/SPARK-18163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ulrich zink closed SPARK-18163. ------------------------------- Resolution: Invalid > Union unexpected behaviour when generating data frames programatically > ---------------------------------------------------------------------- > > Key: SPARK-18163 > URL: https://issues.apache.org/jira/browse/SPARK-18163 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.0.1 > Reporter: Ulrich zink > > //expected behaviour > val df1 = Seq((1,2),(3,4)).toDF("a","b") > val df2 = Seq((5,6)).toDF("a","b") > df1.union(df2).show() > +---+---+ > | a| b| > +---+---+ > | 1| 2| > | 3| 4| > | 5| 6| > +---+---+ > // When generating the data frames programmatically > val nInst = 2 > val fltr = 1 > case class Instrument(id: Long, value: Double) > def dataset (nst:Int,fltrVal:Int) = sqlContext.range(0, nst).select(($"id"), > > round(abs(randn)).alias("value")).as[Instrument].filter('value > fltrVal) > val df3 = dataset(nInst,fltr) > val df4 = dataset(nInst,fltr) > df3.show() > df4.show() > df3.union(df4).show() > df3: org.apache.spark.sql.Dataset[Instrument] = [id: bigint, value: double] > +---+-----+ > | id|value| > +---+-----+ > | 0| 1.0| > | 1| 1.0| > +---+-----+ > df4: org.apache.spark.sql.Dataset[Instrument] = [id: bigint, value: double] > +---+-----+ > | id|value| > +---+-----+ > | 0| 1.0| > | 1| 0.0| > +---+-----+ > +---+-----+ > | id|value| > +---+-----+ > | 0| 1.0| > | 1| 1.0| > | 0| 1.0| > | 1| 2.0| > +---+-----+ -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org