Hey All, I've been cobbling together a high-level consumer for golang building on top of Shopify's Sarama package and wanted to run the basic design by the list and get some feedback or pointers on things I've missed or will eventually encounter on my own.
I'm using zookeeper to coordinate topic-partition owners for consumer members in each consumer group. I followed the znode layout that's apparent from watching the console consumer. <consumer_root>/<consumer_group_name>/{offsets,owners,ids}. The consumer uses an outer loop to discover the partition list for a given topic, attempts to grab a zookeeper lock on each (topic,partition) tuple, and then for each (topic, partition) it successfully locks, launches a thread (goroutine) for each partition to read the partition stream. The outer loop continues to watch for children events either of: <consumer_root>/<consumer_group>/owners/<topic_name><kafka_root>/brokers/topics/<topic_name>/partitions ...any watch event that fires causes all offset data and consumer handles to be flushed and closed, goroutines watching topic-partitions exit. The loop is restarted. Another thread reads topic-partition-offset data and flushes the offset to:<consumer_root>/<consumer_group>/offsets/<topic_name/<partition_number> Have I oversimplified or missed any critical steps?