Anna Povzner created KAFKA-7151: ----------------------------------- Summary: Broker running out of disk space may result in state where unclean leader election is required Key: KAFKA-7151 URL: https://issues.apache.org/jira/browse/KAFKA-7151 Project: Kafka Issue Type: Bug Reporter: Anna Povzner
We have seen situations like the following: 1) Broker A is a leader for topic partition, and brokers B and C are the followers 2) Broker A is running out of disk space, shrinks ISR only to itself, and then sometime later gets disk errors, etc. 3) Broker A is stopped, disk space is reclaimed, and broker A is restarted Result: Broker A becomes a leader, but followers cannot fetch because their log is ahead. The only way to continue is to enable unclean leader election. There are several issues here: -- if the machine is running out of disk space, we do not reliably get an error from a file system as soon as that happens. The broker could be in a state where some writes succeed (possibly if the write is not flushed to disk) and some writes fails, or maybe fail later. This may cause fetchers fetch records that are still in the leader's file system cache, and then the flush to disk failing on the leader, causes followers to be ahead of the leader. -- I am not sure exactly why, but it seems like the leader broker (that is running out of disk space) may also stop servicing fetch requests making followers fall behind and kicked out of ISR. Ideally, the broker should stop being a leader for any topic partition before accepting any records that may fail to be flushed to disk. One option is to automatically detect disk space usage and make a broker read-only for topic partitions if disk space gets to 80% or something. Maybe there is a better option. -- This message was sent by Atlassian JIRA (v7.6.3#76005)