Re: Verifying if DStream is empty

You have to create input1Pair inside foreachRDD function. As your original code, one solution will look like as following: val input2Pair = input2.map(x => (x._1, x)) input2Pair.cache() streamData.foreachRDD{rdd => if(!rdd.isEmpty()){ val input1Pair = rdd.map(x => (x._1, x)) val joinData

Re: Verifying if DStream is empty

Thanks a lot for the response. input1Pair is a DStream. I tried with the code snippet below, result.foreachRDD{externalRDD => if(!externalRDD.isEmpty()){ val ss = input1Pair.transform{ rdd => input2Pair.leftOuterJoin(rdd)} }else{ val ss = input1Pair.transform{ r

Re: Verifying if DStream is empty

Hi Praseetha, In order to check if DStream is empty or not, using isEmpty method is correct. I think the problem here is calling input1Pair.lefOuterJoin(input2Pair). I guess input1Pair rdd comes from above transformation. You should do it on DStream instead. In this case, do any transformation with

Verifying if DStream is empty

Hi Experts, I have 2 inputs, where first input is stream (say input1) and the second one is batch (say input2). I want to figure out if the keys in first input matches single row or more than one row in the second input. The further transformations/logic depends on the number of rows matching, whe