Hi Raj, What you need seems to be an event based initiation of a DStream. Have not seen one yet. There are many types of DStreams that Spark implements. You can also implement your own. InputDStream is a close match for your requirement.
See this for the available options with InputDStream: https://spark.apache.org/docs/1.2.0/api/java/org/apache/spark/streaming/dstream/InputDStream.html You could use a QueueInputDStream. It will wait for a message to arrive at a configured Queue before it can start processing. Another requirements could be to send an even from one SparkContext(your batch context) to another(your streaming context). I have not seen a build in API based solution for this. Broadcast variables(https://spark.apache.org/docs/latest/programming-guide.html#broadcast-variables) are tied to the sc which created it. There are a couple of adhoc approaches I have seen: http://mail-archives.apache.org/mod_mbox/spark-dev/201410.mbox/%3ccadtdqqk-wv9xdeeb-qh3biltjf0a46a5kuqmt2njzwvwuhg...@mail.gmail.com%3E Prajod From: diplomatic Guru [mailto:diplomaticg...@gmail.com] Sent: 11 June 2015 18:55 To: user@spark.apache.org Subject: Could Spark batch processing live within Spark Streaming? Hello all, I was wondering if it is possible to have a high latency batch processing job coexists within Spark Streaming job? If it's possible then could we share the state of the batch job with the Spark Streaming job? For example when the long-running batch computation is complete, could we inform that Spark streaming that batch job is complete? Kind regards, Raj The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com
