To avoid some consumers not consuming anything, one small trick might be
that if a consumer found itself not getting any partition, it can force a
rebalancing by deleting its own registration path and re-register in ZK.
On Mon, Jan 27, 2014 at 4:32 PM, David Birdsong <david.birds...@gmail.com>wrote:
> On Mon, Jan 27, 2014 at 4:19 PM, Guozhang Wang <wangg...@gmail.com> wrote:
>
> > Hello David,
> >
> > One thing about using ZK locks to "own" a partition is load balancing. If
> > you are unlucky some consumer may get all the locks and some may get
> none,
> > hence have no partitions to consume.
> >
>
> I've considered this and even encountered it in testing. For our current
> load levels, we won't hurt us, but if there's a good solution, I'd rather
> codify smooth consumer balance.
>
> Got any suggestions?
>
> My thinking thus far is to establish some sort of identity on the consumer
> and derive an evenness or oddness or some modulo value that induces a small
> delay when encountering particular partition numbers. It's a hacky idea,
> but is pretty simple and might be good enough for smoothing consumers.
>
>
> > Also you may need some synchronization between the consumer thread with
> the
> > offset thread. For example, when an event is fired and the consumers need
> > to re-try grabbing the locks, it needs to first stop current fetchers,
> > commit offsets, and then start owning new partitions.
> >
>
> This is current design and what I have implemented so far. The last thread
> to exit is the offset thread and it has a direct communication channel to
> the consumer threads so it waits for those channels to be closed before
> it's last flush and exit.
>
>
> > Guozhang
> >
> >
> Thanks for the input!
>
>
> >
> > On Mon, Jan 27, 2014 at 3:03 PM, David Birdsong <
> david.birds...@gmail.com
> > >wrote:
> >
> > > Hey All, I've been cobbling together a high-level consumer for golang
> > > building on top of Shopify's Sarama package and wanted to run the basic
> > > design by the list and get some feedback or pointers on things I've
> > missed
> > > or will eventually encounter on my own.
> > >
> > > I'm using zookeeper to coordinate topic-partition owners for consumer
> > > members in each consumer group. I followed the znode layout that's
> > apparent
> > > from watching the console consumer.
> > >
> > > <consumer_root>/<consumer_group_name>/{offsets,owners,ids}.
> > >
> > > The consumer uses an outer loop to discover the partition list for a
> > given
> > > topic, attempts to grab a zookeeper lock on each (topic,partition)
> tuple,
> > > and then for each (topic, partition) it successfully locks, launches a
> > > thread (goroutine) for each partition to read the partition stream.
> > >
> > > The outer loop continues to watch for children events either of:
> > >
> > >
> >
> <consumer_root>/<consumer_group>/owners/<topic_name><kafka_root>/brokers/topics/<topic_name>/partitions
> > >
> > > ...any watch event that fires causes all offset data and consumer
> handles
> > > to be flushed and closed, goroutines watching topic-partitions exit.
> The
> > > loop is restarted.
> > >
> > > Another thread reads topic-partition-offset data and flushes the offset
> > >
> >
> to:<consumer_root>/<consumer_group>/offsets/<topic_name/<partition_number>
> > >
> > > Have I oversimplified or missed any critical steps?
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
