How to restrict foreach on a streaming RDD only once upon receiver completion

I have created a Custom Receiver to fetch records pertaining to a specific query from Elastic Search and have implemented Streaming RDD transformations to process the data generated by the receiver. The final RDD is a sorted list of name value pairs and I want to read the top 20 results

Re: How to restrict foreach on a streaming RDD only once upon receiver completion

it in a Spark program. I think that's more natural as your application is more batch-like than streaming-like as you are using the results in real-time. TD On Mon, Apr 6, 2015 at 12:31 PM, Hari Polisetty hpoli...@icloud.com mailto:hpoli...@icloud.com wrote: I have created a Custom

Re: How to restrict foreach on a streaming RDD only once upon receiver completion

natural as your application is more batch-like than streaming-like as you are using the results in real-time. TD On Mon, Apr 6, 2015 at 12:31 PM, Hari Polisetty hpoli...@icloud.com mailto:hpoli...@icloud.com wrote: I have created a Custom Receiver to fetch records pertaining to a specific

Seeing message about receiver not being de-registered on invoking Streaming context stop

 My application is running Spark in local mode and  I have a Spark Streaming Listener as well as a Custom Receiver. When the receiver is done fetching all documents, it invokes “stop” on itself. I see the StreamingListener  getting a callback on “onReceiverStopped” where I stop the streaming