As crazy as it sounds, there is an actual use-case there. Writing to HDFS can be very slow, so if you do a batch dump from a topic to HDFS, you may want more consumers reading from the topic than for "normal" streaming use-cases. We ended up modifying Camus to split a partition between multiple mappers (take start and end offsets and divide into ranges) to solve this problem. Not exactly a round-robin but same idea.
I think thats what J A was referring to in "decoupling consumers" - different consumers have slightly different requirements. Gwen On Thu, Jul 23, 2015 at 9:44 AM, Jiangjie Qin <j...@linkedin.com.invalid> wrote: > J A, > > It looks to me that in your case you actually want to scale the topic, > right? Otherwise wouldn't a single consumer be enough? > > Jiangjie (Becket) Qin > > On Wed, Jul 22, 2015 at 7:39 PM, J A <mbatth...@gmail.com> wrote: > >> Why have partition at all, if I don't need to scale topic. Coupling topic >> scalability with consumer scalability just goes against messaging systems >> core principle of decoupling consumer and producers >> >> On Wednesday, July 22, 2015, Aditya Auradkar >> <aaurad...@linkedin.com.invalid> >> wrote: >> >> > Hi, >> > >> > Why not simply have as many partitions as the set of consumers you want >> to >> > round robin across? >> > >> > Aditya >> > >> > On Wed, Jul 22, 2015 at 2:37 PM, Ashish Singh <asi...@cloudera.com >> > <javascript:;>> wrote: >> > >> > > Hey, don't you think that would be against the basic ordering >> guarantees >> > > Kafka provides? >> > > >> > > On Wed, Jul 22, 2015 at 2:14 PM, J A <mbatth...@gmail.com >> <javascript:;>> >> > wrote: >> > > >> > > > Hi, This is reference to stackoverflow question " >> > > > >> > > > >> > > >> > >> http://stackoverflow.com/questions/31547216/kafka-log-deletion-and-load-balancing-across-consumers >> > > > " >> > > > Since Kafka 0.8 already maintains a client offset, i would like to >> > > request >> > > > a feature, where a single partition consumption can be round robin >> > > across a >> > > > set of consumers. The message delivery strategy should be an option >> > > chosen >> > > > by the consumer. >> > > > >> > > >> > > >> > > >> > > -- >> > > >> > > Regards, >> > > Ashish >> > > >> > >>
