Hi all, we several time faced situation where consumer-group started to re-consume old events from beginning. Here is scenario:
1. x3 broker kafka cluster on top of x3 node zookeeper 2. RF=3 for all topics 3. log.retention.hours=168 and offsets.retention.minutes=20160 4. running sustainable load (pushing events) 5. doing disaster testing by randomly shutting down 1 of 3 broker nodes (then provision new broker back) Several times after bouncing broker we faced situation where consumer group started to re-consume old events. consumer group: 1. enable.auto.commit = false 2. tried graceful group shutdown, kill -9 and terminating AWS nodes 3. never experienced re-consumption for given cases. What can cause that old events re-consumption? Is it related to bouncing one of brokers? What to search in a logs? Any broker settings to try? Thanks in advance.
