Re: Spark Structured Streaming is giving error “org.apache.spark.sql.AnalysisException: Inner join between two streaming DataFrames/Datasets is not supported;”

2018-05-12 Thread ThomasThomas
Thanks for the quick response...I'm able to inner join the dataframes with regular spark session. The issue is only with the spark streaming session. BTW I'm using Spark 2.2.0 version... -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ -

Spark Structured Streaming is giving error “org.apache.spark.sql.AnalysisException: Inner join between two streaming DataFrames/Datasets is not supported;”

2018-05-12 Thread ThomasThomas
Hi There, Our use case is like this. We have a nested(multiple) JSON message flowing through Kafka Queue. Read the message from Kafka using Spark Structured Streaming(SSS) and explode the data and flatten all data into single record using DataFrame joins and land into a relational database tabl