[
https://issues.apache.org/jira/browse/KAFKA-16779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17847997#comment-17847997
]
Nicholas Feinberg edited comment on KAFKA-16779 at 5/21/24 9:41 PM:
--------------------------------------------------------------------
When we explicitly set topics' retention to 4d (345600000ms), our brokers
immediately expired the surprisingly old logs. When we removed the setting,
they began accumulating old logs again.
I've confirmed that the same setting is present in the brokers'
`server.properties` file - that is, they have `log.retention.hours=96`. I've
also checked and confirmed that topics do not have an explicitly set retention
that would override this.
was (Author: nfeinberg):
When we explicitly set topics' retention to 4d (345600000ms), our brokers
immediately expired the surprisingly old logs.
I've confirmed that the same setting is present in the brokers'
`server.properties` file - that is, they have `log.retention.hours=96`. I've
also checked and confirmed that topics do not have an explicitly set retention
that would override this.
> Kafka retains logs past specified retention
> -------------------------------------------
>
> Key: KAFKA-16779
> URL: https://issues.apache.org/jira/browse/KAFKA-16779
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 3.7.0
> Reporter: Nicholas Feinberg
> Priority: Major
> Labels: expiration, retention
> Attachments: OOM.txt, kafka-20240512.log.gz, kafka-20240514.log.gz,
> kafka-ooms.png, server.log.2024-05-12.gz, server.log.2024-05-14.gz,
> state-change.log.2024-05-12.gz, state-change.log.2024-05-14.gz
>
>
> In a Kafka cluster with all topics set to four days of retention or longer
> (345600000ms), most brokers seem to be retaining six days of data.
> This is true even for topics which have high throughput (500MB/s, 50k msgs/s)
> and thus are regularly rolling new log segments. We observe this unexpectedly
> high retention both via disk usage statistics and by requesting the oldest
> available messages from Kafka.
> Some of these brokers crashed with an 'mmap failed' error (attached). When
> those brokers started up again, they returned to the expected four days of
> retention.
> Manually restarting brokers also seems to cause them to return to four days
> of retention. Demoting and promoting brokers only has this effect on a small
> part of the data hosted on a broker.
> These hosts had ~170GiB of free memory available. We saw no signs of pressure
> on either system or JVM heap memory before or after they reported this error.
> Committed memory seems to be around 10%, so this doesn't seem to be an
> overcommit issue.
> This Kafka cluster was upgraded to Kafka 3.7 two weeks ago (April 29th).
> Prior to the upgrade, it was running on Kafka 2.4.
> We last reduced retention for ops on May 7th, after which we restored
> retention to our default of four days. This was the second time we've
> temporarily reduced and restored retention since the upgrade. This problem
> did not manifest the previous time we did so, nor did it manifest on our
> other Kafka 3.7 clusters.
> We are running on AWS
> [d3en.12xlarge|https://instances.vantage.sh/aws/ec2/d3en.12xlarge] hosts. We
> have 23 brokers, each with 24 disks. We're running in a JBOD configuration
> (i.e. unraided).
> Since this cluster was upgraded from Kafka 2.4 and since we're using JBOD,
> we're still using Zookeeper.
> Sample broker logs are attached. The 05-12 and 05-14 logs are from separate
> hosts. Please let me know if I can provide any further information.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
