Hi Kane, I just wanted to chime in as well. I currently maintain samsa ( https://github.com/getsamsa/samsa or PyPI), which does implement the rebalancing logic, but currently doesn't support 0.8. If you're on 0.7.x still, it might be helpful. If not, we're working on 0.8 support, but it probably won't be ready for a bit.
Cheers, Keith. On Tue, Oct 1, 2013 at 1:39 PM, Kane Kane <kane.ist...@gmail.com> wrote: > Thanks for reply, David, your library is great and indeed the rebalancing > is currently somewhat quirky and complicated. And I guess it doesn't make > sense to implement it considering 0.9 is planned relatively soon. > > > On Tue, Oct 1, 2013 at 10:09 AM, David Arthur <mum...@gmail.com> wrote: > > > Kane, > > > > I'm the creator of kafka-python, just thought I'd give some insight. > > > > Consumer rebalancing is actually pretty tricky to get right. It requires > > interaction with ZooKeeper which (though possible via kazoo) is something > > I've tried to avoid in kafka-python. It also seems a little strange to me > > to mix your consumers between Java/Scala and Python. If you really need > > rebalancing between Python consumers, you'd have to implement that on top > > of kafka-python. > > > > Once the coordinator API is finalized for 0.9, I (or someone) will work > on > > implementing it in kafka-python > > > > Cheers > > -David > > > > > > On 10/1/13 11:56 AM, Kane Kane wrote: > > > >> The reason i was asking is that this library seems to have support only > >> for > >> SimpleConsumer https://github.com/mumrah/**kafka-python/< > https://github.com/mumrah/kafka-python/>, > >> i was curious if > >> all should be implemented on client or kafka has some rebalancing logic > >> and > >> prevent consuming from the same queue on server side in case of > >> SimpleConsumer api, but I see now that everything should implemented on > >> the > >> client side. > >> > >> Thanks. > >> > >> > >> On Tue, Oct 1, 2013 at 8:52 AM, Guozhang Wang <wangg...@gmail.com> > wrote: > >> > >> I do not understand your question, what are you trying to implement? > >>> > >>> > >>> On Tue, Oct 1, 2013 at 8:42 AM, Kane Kane <kane.ist...@gmail.com> > wrote: > >>> > >>> So essentially you can't do "queue" pattern, unless you somehow > >>>> implement > >>>> locking on the client? > >>>> > >>>> > >>>> On Tue, Oct 1, 2013 at 8:35 AM, Guozhang Wang <wangg...@gmail.com> > >>>> > >>> wrote: > >>> > >>>> SimpleConsumer do not have any concept of group management, only the > >>>>> high-level consumers have. So multiple simple consumers can > >>>>> > >>>> independently > >>> > >>>> consume from the same partition(s). > >>>>> > >>>>> Guozhang > >>>>> > >>>>> > >>>>> On Tue, Oct 1, 2013 at 8:11 AM, Kane Kane <kane.ist...@gmail.com> > >>>>> > >>>> wrote: > >>> > >>>> Yeah, I noticed that, i'm curious how balancing happens if > >>>>>> > >>>>> SimpleConsumer > >>>> > >>>>> is used. I.e. i can provide a partition to read from if i use > >>>>>> SimpleConsumer, but what if someone else already attached to that > >>>>>> partition, what would happen? Also what would happen if one > >>>>>> > >>>>> SimpleConsumer > >>>>> > >>>>>> attached to all partitions? Noone would be able to join? > >>>>>> > >>>>>> > >>>>>> On Tue, Oct 1, 2013 at 6:33 AM, Neha Narkhede < > >>>>>> > >>>>> neha.narkh...@gmail.com > >>> > >>>> wrote: > >>>>>>> There are 2 types of consumer clients in Kafka - > >>>>>>> > >>>>>> ZookeeperConsumerConnector > >>>>>> > >>>>>>> and SimpleConsumer. Only the former has the re balancing logic. > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Neha > >>>>>>> On Oct 1, 2013 6:30 AM, "Kane Kane" <kane.ist...@gmail.com> wrote: > >>>>>>> > >>>>>>> But it looks like some clients don't implement it? > >>>>>>>> > >>>>>>>> > >>>>> > >>>>> -- > >>>>> -- Guozhang > >>>>> > >>>>> > >>> > >>> -- > >>> -- Guozhang > >>> > >>> > > >
