When using the 0.8.2 high level consumer, what is the impact of creating many one-time use groupIds and checkpointing offsets using those?
I have a use case where upon every boot, I want to consume an entire topic from the very beginning, all partitions. We are using the high level consumer for convenience in handling leader discovery and rebalancing, but we do not need need consumer groups functionality. We do not need checkpointing of offsets to allow continuing after a restart of our application, since we want to re-consume the stream upon restarts. However, it appears that if you do *not* checkpoint, then when there is an intermittent disconnect, the consumer will restart at the beginning of the topic. I haven't yet traced down why this happens. We were thinking of simply creating a new consumer group id upon every reboot, but this seems messy, leaving around a lot of unused consumer group ids. A couple questions: 1) What resources does a groupId use, when it is active (a consumer using it) and when it is inactive (no consumers using it)? The only resources I can identify are: * kafka/zookeeper using it for group membership (only when the group is active) * disk storage for most recent offset in zookeeper (only the most recent is stored per partition) * disk storage for all offsets in kafka (all checkpoints stored, but there is log compaction) * in-memory storage for most recent offset in kafka, for lookups. 2) Are old non-active groupId's ever deleted? Thanks, -James ________________________________ This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.
