Re: Pause Spark Streaming reading or sampling streaming data

2015-08-05 Thread Heath Guo
Hi Dimitris, Thanks for your reply. Just wondering – are you asking about my streaming input source? I implemented a custom receiver and have been using that. Thanks. From: Dimitris Kouzis - Loukas mailto:look...@gmail.com>> Date: Wednesday, August 5, 2015 at 5:27 PM To: Heath Guo mailto

Pause Spark Streaming reading or sampling streaming data

2015-08-05 Thread Heath Guo
Hi, I have a question about sampling Spark Streaming data, or getting part of the data. For every minute, I only want the data read in during the first 10 seconds, and discard all data in the next 50 seconds. Is there any way to pause reading and discard data in that period? I'm doing this to sa

Re: Spark Streaming reads from stdin or output from command line utility

2015-06-11 Thread Heath Guo
Yes, it is lots of data, and the utility I'm working with prints out infinite real time data stream. Thanks. From: Tathagata Das mailto:t...@databricks.com>> Date: Thursday, June 11, 2015 at 11:43 PM To: Heath Guo mailto:heath...@fb.com>> Cc: user mailto:user@spark.apache.

Re: Spark Streaming reads from stdin or output from command line utility

2015-06-11 Thread Heath Guo
Thanks for your reply! In my use case, it would be stream from only one stdin. Also I'm working with Scala. It would be great if you could talk about multi stdin case as well! Thanks. From: Tathagata Das mailto:t...@databricks.com>> Date: Thursday, June 11, 2015 at 8:11 PM To