Your understanding about segment.bytes and retention.ms is correct. But Tood Palino said just after having reached the segment size, that is when the segment is "closed" PLUS all messages within the segment that was closed are older than the retention policy defined ( in this case retention.ms) THEN delete the segment.
At least according to my testing, it is not necessary to wait until the segment is closed to delete it. Simply if all messages in a segment ( no matter if the segment reached the size defined by segment.bytes) are older than the policy defined by retention.ms , THEN delete the segment. I have been playing a lot today with kafka, and at least that is what I figured out. On Dec 9, 2016 02:13, "Sachin Mittal" <sjmit...@gmail.com> wrote: > I think segment.bytes defines the size of single log file before creating a > new one. > retention.ms defines number of ms to wait on a log file before deleting > it. > > So it is working as defined in docs. > > > On Fri, Dec 9, 2016 at 2:42 AM, Rodrigo Sandoval < > rodrigo.madfe...@gmail.com > > wrote: > > > How is that about that when the segment size is reached, plus every > single > > message inside the segment is older than the retention time, then the > > segment will be deleted? > > > > > > I have playing with Kafka and I have the following: > > > > bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic topic1 > > config retention.ms=60000 > > > > bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic topic1 > > —config file.delete.delay.ms=40000 > > > > bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic topic1 > > --config segment.bytes=400000 > > > > My understanding according to your thoughts is a segment will be deleted > > when the segment reaches out the segment size above defined > > (segment.bytes=400000) PLUS every single message within the segment is > > older than the retention time above defined (retention.ms=60000). > > > > What I noticed is a segment of just 35 bytes, which conteined just one > > message, was deleted after the minute (maybe a little more). Therefore, > the > > segment size was not met in order to delete it. > > >
