Re: Spark Structured Streaming is giving error “org.apache.spark.sql.AnalysisException: Inner join between two streaming DataFrames/Datasets is not supported;”

Hi, After you leave Spark Structured Streaming right after you generate RDDs (for your streaming queries) you can do any kind of "joins". You're again in the old good days of RDD programming (with all the whistles and bells). Please note that Spark Structured Streaming != Spark Streaming since

Re: Spark Structured Streaming is giving error “org.apache.spark.sql.AnalysisException: Inner join between two streaming DataFrames/Datasets is not supported;”

Hi Jacek, If we use RDD instead of Dataframe, can we accomplish the same? I mean, is joining between RDDS allowed in Spark streaming ? Best, Ravi On Sun, May 13, 2018 at 11:18 AM Jacek Laskowski wrote: > Hi, > > The exception message should be self-explanatory and says that

Re: Spark Structured Streaming is giving error “org.apache.spark.sql.AnalysisException: Inner join between two streaming DataFrames/Datasets is not supported;”

Hi, The exception message should be self-explanatory and says that you cannot join two streaming Datasets. This feature was added in 2.3 if I'm not mistaken. Just to be sure that you work with two streaming Datasets, can you show the query plan of the join query? Jacek On Sat, 12 May 2018,

Re: Spark Structured Streaming is giving error “org.apache.spark.sql.AnalysisException: Inner join between two streaming DataFrames/Datasets is not supported;”

Thanks for the quick response...I'm able to inner join the dataframes with regular spark session. The issue is only with the spark streaming session. BTW I'm using Spark 2.2.0 version... -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

Re: Spark Structured Streaming is giving error “org.apache.spark.sql.AnalysisException: Inner join between two streaming DataFrames/Datasets is not supported;”

Perhaps this link might help you. https://stackoverflow.com/questions/48699445/inner-join-not-working-in-dataframe-using-spark-2-1 Best, Passion On Sat, May 12, 2018, 10:57 AM ThomasThomas wrote: > Hi There, > > Our use case is like this. > > We have a nested(multiple)

Spark Structured Streaming is giving error “org.apache.spark.sql.AnalysisException: Inner join between two streaming DataFrames/Datasets is not supported;”

Hi There, Our use case is like this. We have a nested(multiple) JSON message flowing through Kafka Queue. Read the message from Kafka using Spark Structured Streaming(SSS) and explode the data and flatten all data into single record using DataFrame joins and land into a relational database