Hi All, Brokers in one of our Apache Kafka clusters are continuously crashing as they have run out of disk space. As per my understanding, reducing the value of retention.ms and retention.bytes properties will not work because the broker is crashing before the log-retention thread can be scheduled ( link <https://github.com/apache/kafka/blob/3eaf44ba8ea26a7a820894390e8877d404ddd5a2/core/src/main/scala/kafka/log/LogManager.scala#L394-L398> ). One option we are exploring is if we can manually delete some of the old segment files to make some space in our data disk for the broker to startup while reducing the retention.ms config at the same time. There is an old email thread (link <https://mail-archives.apache.org/mod_mbox/kafka-users/201403.mbox/%3CCAOG_4Qbwx44T-=vrpkvqgrum8lpmdzl2bxxrgz5c9h1_noh...@mail.gmail.com%3E>) which suggests it is safe to do so, but we want to understand if there have been recent changes to topic-partition metadata which we might end up corrupting if we try this? If so, are there any tips to get around this issue?
Thanks, Sankalp
