Spark Streaming has a new Kafka direct stream, to be release as experimental feature with 1.3. That uses a low level consumer. Not sure if it satisfies your purpose. If you want more control, its best to create your own Receiver with the low level Kafka API.
TD On Tue, Feb 24, 2015 at 12:09 AM, bit1...@163.com <bit1...@163.com> wrote: > Thanks Akhil. > Not sure whether thelowlevel consumer. > <https://github.com/dibbhatt/kafka-spark-consumer>will be officially > supported by Spark Streaming. So far, I don't see it mentioned/documented > in the spark streaming programming guide. > > ------------------------------ > bit1...@163.com > > > *From:* Akhil Das <ak...@sigmoidanalytics.com> > *Date:* 2015-02-24 16:21 > *To:* bit1...@163.com > *CC:* user <user@spark.apache.org> > *Subject:* Re: Many Receiver vs. Many threads per Receiver > I believe when you go with 1, it will distribute the consumer across your > cluster (possibly on 6 machines), but still it i don't see a away to tell > from which partition it will consume etc. If you are looking to have a > consumer where you can specify the partition details and all, then you are > better off with the lowlevel consumer. > <https://github.com/dibbhatt/kafka-spark-consumer> > > > > Thanks > Best Regards > > On Tue, Feb 24, 2015 at 9:36 AM, bit1...@163.com <bit1...@163.com> wrote: > >> Hi, >> I am experimenting Spark Streaming and Kafka Integration, To read >> messages from Kafka in parallel, basically there are two ways >> 1. Create many Receivers like (1 to 6).map(_ => KakfaUtils.createStream). >> 2. Specifiy many threads when calling KakfaUtils.createStream like val >> topicMap("myTopic"=>6), this will create one receiver with 6 reading >> threads. >> >> My question is which option is better, sounds option 2 is better is to me >> because it saves a lot of cores(one Receiver one core), but I learned >> from somewhere else that choice 1 is better, so I would ask and see how you >> guys elaborate on this. Thank >> >> ------------------------------ >> bit1...@163.com >> > >
