Nicholas Feinberg created KAFKA-16779: -----------------------------------------
Summary: Kafka retains logs past specified retention Key: KAFKA-16779 URL: https://issues.apache.org/jira/browse/KAFKA-16779 Project: Kafka Issue Type: Bug Affects Versions: 3.7.0 Reporter: Nicholas Feinberg Attachments: OOM.txt, kafka-20240512.log.gz, kafka-20240514.log.gz, kafka-ooms.png, server.log.2024-05-12.gz, server.log.2024-05-14.gz, state-change.log.2024-05-12.gz, state-change.log.2024-05-14.gz In a Kafka cluster with all topics set to four days of retention or longer (345600000ms), most brokers seem to be retaining six days of data. This is true even for topics which have high throughput (500MB/s, 50k msgs/s) and thus are regularly rolling new log segments. We observe this unexpectedly high retention both via disk usage statistics and by requesting the oldest available messages from Kafka. Some of these brokers crashed with an 'mmap failed' error (attached). When those brokers started up again, they returned to the expected four days of retention. Manually restarting brokers also seems to cause them to return to four days of retention. Demoting and promoting brokers only has this effect on a small part of the data hosted on a broker. These hosts had ~170GiB of free memory available. We saw no signs of pressure on either system or JVM heap memory before or after they reported this error. Committed memory seems to be around 10%, so this doesn't seem to be an overcommit issue. This Kafka cluster was upgraded to Kafka 3.7 two weeks ago (April 29th). Prior to the upgrade, it was running on Kafka 2.4. We last reduced retention for ops on May 7th, after which we restored retention to our default of four days. This was the second time we've temporarily reduced and restored retention since the upgrade. This problem did not manifest the previous time we did so, nor did it manifest on our other Kafka 3.7 clusters. We are running on AWS [d3en.12xlarge|https://instances.vantage.sh/aws/ec2/d3en.12xlarge] hosts. We have 23 brokers, each with 24 disks. We're running in a JBOD configuration (i.e. unraided). Since this cluster was upgraded from Kafka 2.4 and since we're using JBOD, we're still using Zookeeper. Sample broker logs are attached. The 05-12 and 05-14 logs are from separate hosts. Please let me know if I can provide any further information. -- This message was sent by Atlassian Jira (v8.20.10#820010)