Hi Jacek, If we use RDD instead of Dataframe, can we accomplish the same? I mean, is joining between RDDS allowed in Spark streaming ?
Best, Ravi On Sun, May 13, 2018 at 11:18 AM Jacek Laskowski <ja...@japila.pl> wrote: > Hi, > > The exception message should be self-explanatory and says that you cannot > join two streaming Datasets. This feature was added in 2.3 if I'm not > mistaken. > > Just to be sure that you work with two streaming Datasets, can you show > the query plan of the join query? > > Jacek > > On Sat, 12 May 2018, 16:57 ThomasThomas, <thomaspt...@gmail.com> wrote: > >> Hi There, >> >> Our use case is like this. >> >> We have a nested(multiple) JSON message flowing through Kafka Queue. Read >> the message from Kafka using Spark Structured Streaming(SSS) and explode >> the data and flatten all data into single record using DataFrame joins and >> land into a relational database table(DB2). >> >> But we are getting the following error when we write into db using JDBC. >> >> “org.apache.spark.sql.AnalysisException: Inner join between two streaming >> DataFrames/Datasets is not supported;” >> >> Any help would be greatly appreciated. >> >> Thanks, >> Thomas Thomas >> Mastermind Solutions LLC. >> >> >> >> -- >> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >>
