Hi All,

I am seeing some strange behavior on how the rolled over segments are
deleted. Here is the log for one of the partition.

[2018-10-18 07:49:31,557] Rolled new log segment at offset 3600 in 24 ms.
(kafka.log.Log)
[2018-10-18 07:51:32,443] Deleting segment 0 (kafka.log.Log)
^ Log was rolled over and deleted within 2 minutes. Which is good.

[2018-10-18 08:22:32,131] Rolled new log segment at offset 7216 in 261 ms.
(kafka.log.Log)
[2018-10-18 08:31:44,928] Deleting segment 0 (kafka.log.Log)
[2018-10-18 08:31:44,928] Deleting segment 3600 (kafka.log.Log)
^ Log was rolled over and deleted within 2 minutes. But, why kafka trying
to delete segment 0 again?


[2018-10-18 08:55:10,442] Rolled new log segment at offset 10816 in 1 ms.
(kafka.log.Log)
[2018-10-18 09:27:45,567] Rolled new log segment at offset 14420 in 1 ms.
(kafka.log.Log)
[2018-10-18 09:37:07,728] Deleting segment 0 (kafka.log.Log)
[2018-10-18 09:37:11,549] Deleting segment 7216 (kafka.log.Log)
[2018-10-18 09:37:14,356] Deleting segment 10816 (kafka.log.Log)
^ Again same behavior, but the time between rollover and deletion has
increased to 30 minutes.


[2018-10-18 10:00:37,208] Rolled new log segment at offset 18016 in 1 ms.
(kafka.log.Log)
[2018-10-18 10:32:43,677] Rolled new log segment at offset 21583 in 6 ms.
(kafka.log.Log)
[2018-10-18 10:35:28,808] Deleting segment 0 (kafka.log.Log)
[2018-10-18 10:35:28,808] Deleting segment 7216 (kafka.log.Log)
[2018-10-18 10:35:28,809] Deleting segment 10816 (kafka.log.Log)
[2018-10-18 10:35:32,035] Deleting segment 14420 (kafka.log.Log)
[2018-10-18 10:35:33,953] Deleting segment 18016 (kafka.log.Log)
^ Again deletion is happening after about 35 mins.


[2018-10-18 11:05:18,246] Rolled new log segment at offset 25178 in 12 ms.
(kafka.log.Log)
[2018-10-18 11:37:55,385] Rolled new log segment at offset 28754 in 8 ms.
(kafka.log.Log)
[2018-10-18 12:10:20,695] Rolled new log segment at offset 32347 in 25 ms.
(kafka.log.Log)
[2018-10-18 12:43:07,535] Rolled new log segment at offset 35913 in 1 ms.
(kafka.log.Log)
[2018-10-18 12:57:10,200] Deleting segment 0 (kafka.log.Log)
[2018-10-18 12:57:10,200] Deleting segment 14420 (kafka.log.Log)
[2018-10-18 12:57:10,201] Deleting segment 18016 (kafka.log.Log)
[2018-10-18 12:57:12,584] Deleting segment 21583 (kafka.log.Log)
[2018-10-18 12:57:16,420] Deleting segment 25178 (kafka.log.Log)
[2018-10-18 12:57:19,187] Deleting segment 28754 (kafka.log.Log)
[2018-10-18 12:57:22,864] Deleting segment 32347 (kafka.log.Log)
^ Now the time difference has increased to ~1 hour 50 minutes (segment with
offset 25178)


[2018-10-18 13:15:24,665] Rolled new log segment at offset 39512 in 3 ms.
(kafka.log.Log)
[2018-10-18 13:47:39,469] Rolled new log segment at offset 43067 in 1 ms.
(kafka.log.Log)
[2018-10-18 14:20:38,673] Rolled new log segment at offset 46654 in 1 ms.
(kafka.log.Log)
[2018-10-18 14:53:07,311] Rolled new log segment at offset 50244 in 1 ms.
(kafka.log.Log)
[2018-10-18 14:56:35,313] Deleting segment 0 (kafka.log.Log)
[2018-10-18 14:56:35,313] Deleting segment 21583 (kafka.log.Log)
[2018-10-18 14:56:35,313] Deleting segment 25178 (kafka.log.Log)
[2018-10-18 14:56:35,313] Deleting segment 28754 (kafka.log.Log)
[2018-10-18 14:56:35,313] Deleting segment 32347 (kafka.log.Log)
[2018-10-18 14:56:40,024] Deleting segment 35913 (kafka.log.Log)
[2018-10-18 14:56:43,125] Deleting segment 39512 (kafka.log.Log)
[2018-10-18 14:56:44,951] Deleting segment 43067 (kafka.log.Log)
[2018-10-18 14:56:47,398] Deleting segment 46654 (kafka.log.Log)
^ Time difference between roll over and deletion ~1 hour 40 minutes
(segment with offset 25178)


[2018-10-18 15:25:51,075] Rolled new log segment at offset 53832 in 15 ms.
(kafka.log.Log)
[2018-10-18 15:58:02,546] Rolled new log segment at offset 57386 in 46 ms.
(kafka.log.Log)
[2018-10-18 16:30:16,005] Rolled new log segment at offset 60955 in 1 ms.
(kafka.log.Log)
[2018-10-18 17:02:48,638] Rolled new log segment at offset 64507 in 1 ms.
(kafka.log.Log)
[2018-10-18 17:35:47,851] Rolled new log segment at offset 68106 in 1 ms.
(kafka.log.Log)
[2018-10-18 18:08:25,337] Rolled new log segment at offset 71693 in 2 ms.
(kafka.log.Log)
^ No deletion even after 2 hour 30 minutes.

Any idea what is going on.?
Also note that these are for internal topics, and the properties configured
are

props.put(StreamsConfig.topicPrefix(TopicConfig.SEGMENT_BYTES_CONFIG),
1073741824);
props.put(StreamsConfig.topicPrefix(TopicConfig.RETENTION_BYTES_CONFIG),
1073741824);
props.put(StreamsConfig.topicPrefix(TopicConfig.RETENTION_MS_CONFIG), 20 *
60 * 1000);
props.put(StreamsConfig.topicPrefix(TopicConfig.DELETE_RETENTION_MS_CONFIG),
1000);

I plan to add
props.put(StreamsConfig.topicPrefix(TopicConfig.MESSAGE_TIMESTAMP_TYPE_CONFIG),
"LogAppendTime");
since the message doesn't have CreationTime in it.
But I still wonder why the delay between roll over and deletion keeps
increasing and why the earlier deleted segments are being deleted again (at
least in the logs)

Thanks,
Vishnu

Reply via email to