Sorry, Please ignore the question of deleting older segments again. Those were for other partitions. The only question is about the delay.
Thanks, Vishnu On Thu, Oct 18, 2018 at 1:26 PM Vishnu Viswanath < [email protected]> wrote: > Hi All, > > I am seeing some strange behavior on how the rolled over segments are > deleted. Here is the log for one of the partition. > > [2018-10-18 07:49:31,557] Rolled new log segment at offset 3600 in 24 ms. > (kafka.log.Log) > [2018-10-18 07:51:32,443] Deleting segment 0 (kafka.log.Log) > ^ Log was rolled over and deleted within 2 minutes. Which is good. > > [2018-10-18 08:22:32,131] Rolled new log segment at offset 7216 in 261 ms. > (kafka.log.Log) > [2018-10-18 08:31:44,928] Deleting segment 0 (kafka.log.Log) > [2018-10-18 08:31:44,928] Deleting segment 3600 (kafka.log.Log) > ^ Log was rolled over and deleted within 2 minutes. But, why kafka trying > to delete segment 0 again? > > > [2018-10-18 08:55:10,442] Rolled new log segment at offset 10816 in 1 ms. > (kafka.log.Log) > [2018-10-18 09:27:45,567] Rolled new log segment at offset 14420 in 1 ms. > (kafka.log.Log) > [2018-10-18 09:37:07,728] Deleting segment 0 (kafka.log.Log) > [2018-10-18 09:37:11,549] Deleting segment 7216 (kafka.log.Log) > [2018-10-18 09:37:14,356] Deleting segment 10816 (kafka.log.Log) > ^ Again same behavior, but the time between rollover and deletion has > increased to 30 minutes. > > > [2018-10-18 10:00:37,208] Rolled new log segment at offset 18016 in 1 ms. > (kafka.log.Log) > [2018-10-18 10:32:43,677] Rolled new log segment at offset 21583 in 6 ms. > (kafka.log.Log) > [2018-10-18 10:35:28,808] Deleting segment 0 (kafka.log.Log) > [2018-10-18 10:35:28,808] Deleting segment 7216 (kafka.log.Log) > [2018-10-18 10:35:28,809] Deleting segment 10816 (kafka.log.Log) > [2018-10-18 10:35:32,035] Deleting segment 14420 (kafka.log.Log) > [2018-10-18 10:35:33,953] Deleting segment 18016 (kafka.log.Log) > ^ Again deletion is happening after about 35 mins. > > > [2018-10-18 11:05:18,246] Rolled new log segment at offset 25178 in 12 ms. > (kafka.log.Log) > [2018-10-18 11:37:55,385] Rolled new log segment at offset 28754 in 8 ms. > (kafka.log.Log) > [2018-10-18 12:10:20,695] Rolled new log segment at offset 32347 in 25 ms. > (kafka.log.Log) > [2018-10-18 12:43:07,535] Rolled new log segment at offset 35913 in 1 ms. > (kafka.log.Log) > [2018-10-18 12:57:10,200] Deleting segment 0 (kafka.log.Log) > [2018-10-18 12:57:10,200] Deleting segment 14420 (kafka.log.Log) > [2018-10-18 12:57:10,201] Deleting segment 18016 (kafka.log.Log) > [2018-10-18 12:57:12,584] Deleting segment 21583 (kafka.log.Log) > [2018-10-18 12:57:16,420] Deleting segment 25178 (kafka.log.Log) > [2018-10-18 12:57:19,187] Deleting segment 28754 (kafka.log.Log) > [2018-10-18 12:57:22,864] Deleting segment 32347 (kafka.log.Log) > ^ Now the time difference has increased to ~1 hour 50 minutes (segment > with offset 25178) > > > [2018-10-18 13:15:24,665] Rolled new log segment at offset 39512 in 3 ms. > (kafka.log.Log) > [2018-10-18 13:47:39,469] Rolled new log segment at offset 43067 in 1 ms. > (kafka.log.Log) > [2018-10-18 14:20:38,673] Rolled new log segment at offset 46654 in 1 ms. > (kafka.log.Log) > [2018-10-18 14:53:07,311] Rolled new log segment at offset 50244 in 1 ms. > (kafka.log.Log) > [2018-10-18 14:56:35,313] Deleting segment 0 (kafka.log.Log) > [2018-10-18 14:56:35,313] Deleting segment 21583 (kafka.log.Log) > [2018-10-18 14:56:35,313] Deleting segment 25178 (kafka.log.Log) > [2018-10-18 14:56:35,313] Deleting segment 28754 (kafka.log.Log) > [2018-10-18 14:56:35,313] Deleting segment 32347 (kafka.log.Log) > [2018-10-18 14:56:40,024] Deleting segment 35913 (kafka.log.Log) > [2018-10-18 14:56:43,125] Deleting segment 39512 (kafka.log.Log) > [2018-10-18 14:56:44,951] Deleting segment 43067 (kafka.log.Log) > [2018-10-18 14:56:47,398] Deleting segment 46654 (kafka.log.Log) > ^ Time difference between roll over and deletion ~1 hour 40 minutes > (segment with offset 25178) > > > [2018-10-18 15:25:51,075] Rolled new log segment at offset 53832 in 15 ms. > (kafka.log.Log) > [2018-10-18 15:58:02,546] Rolled new log segment at offset 57386 in 46 ms. > (kafka.log.Log) > [2018-10-18 16:30:16,005] Rolled new log segment at offset 60955 in 1 ms. > (kafka.log.Log) > [2018-10-18 17:02:48,638] Rolled new log segment at offset 64507 in 1 ms. > (kafka.log.Log) > [2018-10-18 17:35:47,851] Rolled new log segment at offset 68106 in 1 ms. > (kafka.log.Log) > [2018-10-18 18:08:25,337] Rolled new log segment at offset 71693 in 2 ms. > (kafka.log.Log) > ^ No deletion even after 2 hour 30 minutes. > > Any idea what is going on.? > Also note that these are for internal topics, and the properties > configured are > > props.put(StreamsConfig.topicPrefix(TopicConfig.SEGMENT_BYTES_CONFIG), > 1073741824); > props.put(StreamsConfig.topicPrefix(TopicConfig.RETENTION_BYTES_CONFIG), > 1073741824); > props.put(StreamsConfig.topicPrefix(TopicConfig.RETENTION_MS_CONFIG), 20 * > 60 * 1000); > props.put(StreamsConfig.topicPrefix(TopicConfig.DELETE_RETENTION_MS_CONFIG), > 1000); > > I plan to add > props.put(StreamsConfig.topicPrefix(TopicConfig.MESSAGE_TIMESTAMP_TYPE_CONFIG), > "LogAppendTime"); > since the message doesn't have CreationTime in it. > But I still wonder why the delay between roll over and deletion keeps > increasing and why the earlier deleted segments are being deleted again (at > least in the logs) > > Thanks, > Vishnu >
