1) yes, just use .repartition on the inbound stream, this will shuffle data across your whole cluster and process in parallel as specified. 2) yes, although I’m not sure how to do it for a totally custom receiver. Does this help as a starting point? http://spark.apache.org/docs/latest/streaming-programming-guide.html#level-of-parallelism-in-data-receiving
On 9/25/15, 4:08 PM, "nib...@free.fr" <nib...@free.fr> wrote: >Hello, >I used a custom receiver in order to receive JMS messages from MQ Servers. >I want to benefit of Yarn cluster, my questions are : > >- Is it possible to have only one node receiving JMS messages and parralelize >the RDD over all the cluster nodes ? >- Is it possible to parallelize also the message receiver over cluster nodes ? > >If you have any code example for the both items it would be fine, because the >parralelization mechanism in the code is not crystal clear for me ... > >Tks >Nicolas > >--------------------------------------------------------------------- >To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >For additional commands, e-mail: user-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
