or you can use this Receiver as well : http://spark-packages.org/package/dibbhatt/kafka-spark-consumer
Where you can specify how many Receivers you need for your topic and it will divides the partitions among the Receiver and return the joined stream for you . Say you specified 20 receivers , in that case each Receiver can handle 4 partitions and you get consumer parallelism of 20 receivers . Dibyendu On Wed, May 13, 2015 at 9:28 PM, 李森栋 <[email protected]> wrote: > thank you very much > > > 来自 魅族 MX4 Pro > > -------- 原始邮件 -------- > 发件人：Cody Koeninger <[email protected]> > 时间：周三 5月13日 23:52 > 收件人：hotdog <[email protected]> > 抄送：[email protected] > 主题：Re: force the kafka consumer process to different machines > > >I assume you're using the receiver based approach? Have you tried the > >createDirectStream api? > > > >https://spark.apache.org/docs/1.3.0/streaming-kafka-integration.html > > > >If you're sticking with the receiver based approach I think your only > >option would be to create more consumer streams and union them. That > >doesn't give you control over where they're run, but should increase the > >consumer parallelism. > > > >On Wed, May 13, 2015 at 10:33 AM, hotdog <[email protected]> wrote: > > > >> I 'm using streaming integrated with streaming-kafka. > >> > >> My kafka topic has 80 partitions, while my machines have 40 cores. I > found > >> that when the job is running, the kafka consumer processes are only > deploy > >> to 2 machines, the bandwidth of the 2 machines will be very very high. > >> > >> I wonder is there any way to control the kafka consumer's dispatch？ > >> > >> > >> > >> -- > >> View this message in context: > >> > http://apache-spark-user-list.1001560.n3.nabble.com/force-the-kafka-consumer-process-to-different-machines-tp22872.html > >> Sent from the Apache Spark User List mailing list archive at Nabble.com. > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > >> >
