I have this function in the driver program which collects the result from rdds (in a stream) into an array and return. However, even though the RDDs (in the dstream) have data, the function is returning an empty array...What am I doing wrong?
I can print the RDD values inside the foreachRDD call but the array is always empty. def runTopFunction() : Array[(String, Int)] = { val topSearches = some function.... val summary = new ArrayBuffer[(String,Int)]() topSearches.foreachRDD(rdd => { summary = summary.++(rdd.collect()) }) return summary.toArray }