Hi, thanks for your reply.
Also the groups were deleted month ago, there are still valid values there. So I guess deleting the group did not produce the tombstone record correctly. Your explanation made it clearer for me. I know I should keep cleanup policy compact in general. I decided to switch to policy for just long enough, so that these old records get deleted. I guess I could have produced the tombstone recods to be on the safe side. Thanks, Claudia -----Ursprüngliche Nachricht----- Von: Vincent Maurin <vincent.maurin...@gmail.com> Gesendet: Freitag, 29. März 2019 15:24 An: users@kafka.apache.org Betreff: Re: Offsets of deleted consumer groups do not get deleted correctly Hi, You should keep the policy compact for the topic __consumer_offsets This topic stores for each group/topic/partition the offset consumed. As only the latest message for a group/topic/partition is relevant, the policy compact will keep only this message. When you delete a group, actually it will produce a tombstone to this topic (i.e body NULL). Then when the log compaction is running, it will definitively remove the tombstone. But to have an effective delete of the tombstones, keep in mind : * compaction runs only on rolled out segments * deletion of tombstone only occurs if the delete.retention.ms delay is expired Best regards On Fri, Mar 29, 2019 at 2:16 PM Claudia Wegmann <c.wegm...@kasasi.de> wrote: > Hey there, > > I've got the problem that the "__consumer_offsets" topic grows pretty > big over time. After some digging, I found offsets for consumer groups > that were deleted a long time ago still being present in the topic. > Many of them are offsets for console consumers, that have been deleted > with "kafka-consumer-groups.sh --delete --group ...". > > As far as I understand log cleaning, those offsets should have been > deleted a long time ago, because these consumers are no longer active. > When I query "kafka-consumer-groups.sh --bootstrap-server ... --list" > I don't see those consumers either. > > Is there a bug in "kafka-consumer-groups.sh --delete --group ..." that > let's kafka hang on to those consumer groups? > > How can I get the log cleaner to delete these old offsets? Is there > another way than setting "cleanup.policy" to "delete"? > > Thanks for our help! > > Best, > Claudia >
