On Wed, Oct 29, 2014 at 3:39 PM, spr <s...@yarcdata.com> wrote: > I need more precision to understand. If the elements of one DStream/RDD > are > (String) and the elements of the other are (Time, Int), what does "union" > mean? I'm hoping for (String, Time, Int) but that appears optimistic. :) > It won't compile.
> Do the elements have to be of homogeneous type? > Yes. From the scaladoc ( http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.streaming.dstream.DStream ) you can see DStreams are generic/templated on a type (T) and the union function works on a DStream of the same templated type. If you have hetrogeneous data you can first map each DStream it to a case class with options or try something like http://stackoverflow.com/questions/3508077/does-scala-have-type-disjunction-union-types > > > Holden Karau wrote > > The union function simply returns a DStream with the elements from both. > > This is the same behavior as when we call union on RDDs :) (You can think > > of union as similar to the union operator on sets except without the > > unique > > element restrictions). > > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/what-does-DStream-union-do-tp17673p17682.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > -- Cell : 425-233-8271
