Hi, join between streaming and batch/static Datasets is supported for sure --> http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#join-operations
I'm not sure about union, but that's just easy to check (and am leaving it as your home exercise). You cannot have datasets of different schema in a query. You'd have to use the most wide schema to cover all schemas. p.s. Have you tried anything...spark-shell's your friend, my friend :) Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski Spark Structured Streaming https://bit.ly/spark-structured-streaming Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Wed, Dec 13, 2017 at 11:16 PM, kant kodali <kanth...@gmail.com> wrote: > Hi All, > > I have messages in a queue that might be coming in with few different > schemas like > msg 1 schema 1, msg2 schema2, msg3 schema3, msg 4 schema1.... > > I want to put all of this in one data frame. is it possible with > structured streaming? > > I am using Spark 2.2.0 > > Thanks! > >