I didn't include the complete driver code but I do run the streaming
context from the main program which calls this function. Again, I can print
the red elements within the foreachrdd block but the array that is returned
is always empty. It appears that the function immediately returns even
before the foreachrdd stage is executed. Is that possible?
On Feb 25, 2015 5:41 PM, "Tathagata Das" <t...@databricks.com> wrote:
> You are just setting up the computation here using foreacRDD. You have not
> even run the streaming context to get any data.
>
>
> On Wed, Feb 25, 2015 at 2:21 PM, Thanigai Vellore <
> thanigai.vell...@gmail.com> wrote:
>
>> I have this function in the driver program which collects the result from
>> rdds (in a stream) into an array and return. However, even though the RDDs
>> (in the dstream) have data, the function is returning an empty array...What
>> am I doing wrong?
>>
>> I can print the RDD values inside the foreachRDD call but the array is
>> always empty.
>>
>> def runTopFunction() : Array[(String, Int)] = {
>> val topSearches = some function....
>> val summary = new ArrayBuffer[(String,Int)]()
>> topSearches.foreachRDD(rdd => {
>> summary = summary.++(rdd.collect())
>> })
>>
>> return summary.toArray
>> }
>>
>>
>
