Hey All, I've been cobbling together a high-level consumer for golang
building on top of Shopify's Sarama package and wanted to run the basic
design by the list and get some feedback or pointers on things I've missed
or will eventually encounter on my own.
I'm using zookeeper to coordinate topic-partition owners for consumer
members in each consumer group. I followed the znode layout that's apparent
from watching the console consumer.
<consumer_root>/<consumer_group_name>/{offsets,owners,ids}.
The consumer uses an outer loop to discover the partition list for a given
topic, attempts to grab a zookeeper lock on each (topic,partition) tuple,
and then for each (topic, partition) it successfully locks, launches a
thread (goroutine) for each partition to read the partition stream.
The outer loop continues to watch for children events either of:
<consumer_root>/<consumer_group>/owners/<topic_name><kafka_root>/brokers/topics/<topic_name>/partitions
...any watch event that fires causes all offset data and consumer handles
to be flushed and closed, goroutines watching topic-partitions exit. The
loop is restarted.
Another thread reads topic-partition-offset data and flushes the offset
to:<consumer_root>/<consumer_group>/offsets/<topic_name/<partition_number>
Have I oversimplified or missed any critical steps?
