chia7712 commented on PR #21291: URL: https://github.com/apache/kafka/pull/21291#issuecomment-3746424293
The primary concern here is the impact on high traffic scenarios. I agree that the current limit of 500 might be too conservative, leading to frequent snapshots and write amplification. However, completely removing the limit poses a significant risk: without a new snapshot, the coordinator cannot prune stale update records. This leads to unbounded growth in log segments (disk bloat) and unpredictable replay times during restart. Ideally, if we want to optimize for specific high-volume scenarios, we should introduce a group-level configuration for fine-grained control. In this context, the server-level config should serve as a safety guardrail (an upper bound) for all groups to prevent a single misconfigured group from blocking the cleanup of the entire partition. Maybe KIP-1240 could include this improvement if @AndrewJSchofield agrees WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
