Hello, We’re seeing a strange behavior in Kafka 0.9.0.1 which occurs about every other week. I’m curious if others have seen it and know of a solution.
Setup and Scenario: - Brokers initially setup with log compaction turned off - After 30 days, log compaction was turned on - At this time, the number of Open FDs was ~ 30K per broker. - After 2 days, the __consumer_offsets topic was compacted fully. Open FDs reduced to ~5K per broker. - Cluster has been under normal load for roughly 7 days. - At the 7 day mark, __consumer_offsets topic seems to have stopped compacting on two of the brokers, and on those brokers, the FD count is up to ~25K. We have tried rebalancing the partitions before. The first time, the destination broker had compacted the data fine and open FDs were low. The second time, the destination broker kept the FDs open. In all the broker logs, we’re seeing this messages: INFO [Group Metadata Manager on Broker 8]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager) There are only 4 consumers at the moment on the cluster; one topic with 92 partitions. Is there a reason why log compaction may stop working or why the __consumer_offsets topic would start holding thousands of FDs? Thank you all for your help! Lawrence Weikum
