hudeqi opened a new pull request, #13852: URL: https://github.com/apache/kafka/pull/13852
### Activation As the config 'segment.bytes' for topics related MM2(such as offset.storage.topic, config.storage.topic,status.storage.topic), if following the default configuration of the broker or set it larger, then when the MM cluster runs many and complicated tasks, especially the log volume of the topic 'offset.storage.topic' is very large, it will affect the restart speed of the MM workers. After investigation, the reason is that a consumer needs to be started to read the data of ‘offset.storage.topic’ at startup. Although this topic is set to compact, if the 'segment size' is set to a large value, such as the default value of 1G, then this topic may have tens of gigabytes of data that cannot be compacted and has to be read from the earliest (because the active segment cannot be cleaned), which will consume a lot of time (in our online environment, we found that this topic stores 13G of data, it took nearly half an hour for all the data to be consumed), which caused the worker to be unable to start and execute tasks for a long time. ### Solution The number of consumer threads can also be adjusted, but I think it may be easier to set a upper limit for the 'segment size', for example, refer to the default value of __consumer_offsets: 100MB -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
