Re: Multiple Spark Streaming receiver model
I'm new to Spark. From my experience when I use asingle StreamingContext to create different input streams from different sources I get multiple errors and problems down stream. This seems like it is not the way to go. From what I read creating multiple StreamingContext is not advised. It appears to be heavyweight and OOM can occur. Creating a pool of StreamingContext seems not an elegant solution. I'm in a bind as well as to the best approach for multiple sources. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Multiple-Spark-Streaming-receiver-model-tp21002p21215.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Multiple Spark Streaming receiver model
Hi Manjul, Each StreamingContext will have its own batch size. If that doesn’t work for the different sources you have then you would have to create different streaming apps. You can only create a new StreamingContext in the same Spark app, once you’ve stopped the previous one. Spark certainly supports multiple streams within the same streaming app. In fact if the ingest rate of your stream is very high you can create multiple receivers and union the streams. Just remember that each receiver will require a core. The receivers will be distributed appropriately across your executors. Thanks, Silvio On 1/6/15, 7:11 PM, manjuldixit manjul.di...@kronos.com wrote: Hi, We have a requirement of receiving live input messages from RabbitMQ and process them into micro batches. For this we have selected SparkStreaming and we have written a connector for RabbitMQ receiver and Spark streaming, it is working fine. Now the main requirement is to receive different category of events from different channels/queues in a spark streaming context. Q1 : So how can I create different streaming context to receive message from different source (may or may not be in same frequency) within a Spark context? Q2: Is is advisable to use single StreamingContext to create different input streams from different sources? Q3: What all design consideration I need to take care in terms of specifying no of Cores for Spark master. Q4: Is there a was to distribute different Streaming receiving task to different workers ? You suggestions will be very valuable in my application design and if you can have some example for complex application which you can share may some URL reference or whatever ; it is very appreciated. Thanks Manjul -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Multiple-Spark-Streami ng-receiver-model-tp21002.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Multiple Spark Streaming receiver model
Hi, We have a requirement of receiving live input messages from RabbitMQ and process them into micro batches. For this we have selected SparkStreaming and we have written a connector for RabbitMQ receiver and Spark streaming, it is working fine. Now the main requirement is to receive different category of events from different channels/queues in a spark streaming context. Q1 : So how can I create different streaming context to receive message from different source (may or may not be in same frequency) within a Spark context? Q2: Is is advisable to use single StreamingContext to create different input streams from different sources? Q3: What all design consideration I need to take care in terms of specifying no of Cores for Spark master. Q4: Is there a was to distribute different Streaming receiving task to different workers ? You suggestions will be very valuable in my application design and if you can have some example for complex application which you can share may some URL reference or whatever ; it is very appreciated. Thanks Manjul -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Multiple-Spark-Streaming-receiver-model-tp21002.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
