This is my topic setup: Topic:mytopic PartitionCount:6 ReplicationFactor:3 Configs:retention.ms =604800000,cleanup.policy=delete,retention.bytes=58000000000 Topic: mytopic Partition: 0 Leader: 1 Replicas: 1,3,2 Isr: 3,2,1 Topic: mytopic Partition: 1 Leader: 2 Replicas: 2,1,3 Isr: 3,2,1 Topic: mytopic Partition: 2 Leader: 3 Replicas: 3,2,1 Isr: 3,2,1 Topic: mytopic Partition: 3 Leader: 1 Replicas: 1,2,3 Isr: 3,2,1 Topic: mytopic Partition: 4 Leader: 2 Replicas: 2,3,1 Isr: 3,2,1 Topic: mytopic Partition: 5 Leader: 3 Replicas: 3,1,2 Isr: 3,2,1
And this is the only config I use on the brokers/cluster: log.retention.ms=604800000 log.retention.bytes=96636764160 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 Presumably log.segment.ms will be the default of 604800000 (7 days) Yet I still have logs going back 15 days, 37 files to be precise. These are the oldest: 1073740291 Jun 4 17:57 00000000000079419837.log 1073739923 Jun 4 22:11 00000000000080022206.log 1073740578 Jun 5 04:25 00000000000080597678.log If my understanding is correct, this should be happening: - A new log segment file will be created when log.segment.bytes (1 GB) is hit. This is happening as I have 37 x 1 GB files. - Old log segments will be deleted when retention.bytes (58 GB) of logs is hit for that partition. I don't have this much data yet so this isn't being triggered. - Old log segments will be deleted when retention.ms (7 days) is hit. This isn't happening as my oldest file is now 15 days old. - Old log segments should be checked if eligible for deletion every log.retention.check.interval.ms (5 mins). This isn't happening as my oldest file is now 15 days old. On Wed, Jun 19, 2019 at 7:04 AM Babu Turlapati <b...@confluent.io> wrote: > retention policy applied only for closed segments. Please check when > current segment of topic getting closed.segment gets rotated when either > the log.segment.bytes or the log.segment.ms limit is reached. > > Get Outlook for iOS<https://aka.ms/o0ukef> > ________________________________ > From: Tom Vernon <t...@machineadvertising.com> > Sent: Tuesday, June 18, 2019 12:06 PM > To: users@kafka.apache.org > Subject: retention.ms not working for a single topic > > Hi, I have several 2.2.0 Kafka clusters, each with several topics. One of > the topics has retention.ms=604800000 (7days) set at the topic level and > also log.retention.ms=604800000 (7 days) set at the cluster level. There > are no other hour/day retention settings in place. The log files for this > topic are now up to 14 days old and they don't seem to be getting deleted > by the standard cleanup policy. Any ideas why this would be? Cleanup looks > to be working as expected everywhere else. The only thing different about > this particular topic is that we have been experimenting with different > compression codecs over the last few weeks, therefore it contains a mixture > of lz4, snappy etc. I cant see why this would make a difference. > > Thanks > -- *Tom Vernon* Infrastructure Architect +44 (0)7903 399 160 <07903399160> machineadvertising.com
