Agree.
On Thu, Jul 23, 2015 at 2:43 PM, Jiangjie Qin <j...@linkedin.com.invalid> wrote: > Ah, I see. Thanks for the use case, Gwen. I guess in that case it seems the > time to use low level consumer. > > Jiangjie (Becket) Qin > > On Thu, Jul 23, 2015 at 9:52 AM, Gwen Shapira <gshap...@cloudera.com> wrote: > >> As crazy as it sounds, there is an actual use-case there. >> >> Writing to HDFS can be very slow, so if you do a batch dump from a >> topic to HDFS, you may want more consumers reading from the topic than >> for "normal" streaming use-cases. We ended up modifying Camus to split >> a partition between multiple mappers (take start and end offsets and >> divide into ranges) to solve this problem. Not exactly a round-robin >> but same idea. >> >> I think thats what J A was referring to in "decoupling consumers" - >> different consumers have slightly different requirements. >> >> Gwen >> >> >> >> >> >> On Thu, Jul 23, 2015 at 9:44 AM, Jiangjie Qin <j...@linkedin.com.invalid> >> wrote: >> > J A, >> > >> > It looks to me that in your case you actually want to scale the topic, >> > right? Otherwise wouldn't a single consumer be enough? >> > >> > Jiangjie (Becket) Qin >> > >> > On Wed, Jul 22, 2015 at 7:39 PM, J A <mbatth...@gmail.com> wrote: >> > >> >> Why have partition at all, if I don't need to scale topic. Coupling >> topic >> >> scalability with consumer scalability just goes against messaging >> systems >> >> core principle of decoupling consumer and producers >> >> >> >> On Wednesday, July 22, 2015, Aditya Auradkar >> >> <aaurad...@linkedin.com.invalid> >> >> wrote: >> >> >> >> > Hi, >> >> > >> >> > Why not simply have as many partitions as the set of consumers you >> want >> >> to >> >> > round robin across? >> >> > >> >> > Aditya >> >> > >> >> > On Wed, Jul 22, 2015 at 2:37 PM, Ashish Singh <asi...@cloudera.com >> >> > <javascript:;>> wrote: >> >> > >> >> > > Hey, don't you think that would be against the basic ordering >> >> guarantees >> >> > > Kafka provides? >> >> > > >> >> > > On Wed, Jul 22, 2015 at 2:14 PM, J A <mbatth...@gmail.com >> >> <javascript:;>> >> >> > wrote: >> >> > > >> >> > > > Hi, This is reference to stackoverflow question " >> >> > > > >> >> > > > >> >> > > >> >> > >> >> >> http://stackoverflow.com/questions/31547216/kafka-log-deletion-and-load-balancing-across-consumers >> >> > > > " >> >> > > > Since Kafka 0.8 already maintains a client offset, i would like to >> >> > > request >> >> > > > a feature, where a single partition consumption can be round robin >> >> > > across a >> >> > > > set of consumers. The message delivery strategy should be an >> option >> >> > > chosen >> >> > > > by the consumer. >> >> > > > >> >> > > >> >> > > >> >> > > >> >> > > -- >> >> > > >> >> > > Regards, >> >> > > Ashish >> >> > > >> >> > >> >> >>
