high-level consumer design

2014-09-26 Thread Birla, Lokesh
Hi, Is this implemented? I am looking for high level consumer implementation in golang. [Kafka-users] high-level consumer design [http://grokbase.com/ico/sbj/rss/24]http://grokbase.com/g/kafka/users/feed [http://grokbase.com/ico/sbj/rss/24] http://grokbase.com/t/kafka/users/141vamny9j/high

high-level consumer design

2014-01-27 Thread David Birdsong
Hey All, I've been cobbling together a high-level consumer for golang building on top of Shopify's Sarama package and wanted to run the basic design by the list and get some feedback or pointers on things I've missed or will eventually encounter on my own. I'm using zookeeper to coordinate

Re: high-level consumer design

2014-01-27 Thread Guozhang Wang
Hello David, One thing about using ZK locks to own a partition is load balancing. If you are unlucky some consumer may get all the locks and some may get none, hence have no partitions to consume. Also you may need some synchronization between the consumer thread with the offset thread. For

Re: high-level consumer design

2014-01-27 Thread David Birdsong
On Mon, Jan 27, 2014 at 4:19 PM, Guozhang Wang wangg...@gmail.com wrote: Hello David, One thing about using ZK locks to own a partition is load balancing. If you are unlucky some consumer may get all the locks and some may get none, hence have no partitions to consume. I've considered this