[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
--------------------------------
Description:
Kafka log cannot be deleted after the configured retention hours (12 hours for
log retention).
Is it reproducible?
I am not sure since our Kafka cluster is working well over 1.5 years without
retention issue until 4/13/2019 ~ 4/20/2019.
is it related to the active segment?
as I know Kafka will not delete an active segment, my case those old logs are
not activated, they should be inactivated.
What's current status?
Those old logs have been deleted after I manually rolling Kafka server with
retention hours adjustment.
A sample old log
```
// it has been there from 4/12
-rw-r--r-- 1 root root 136866627{color:#d04437} Apr 12{color} 04:33
00000000002581377820.log
// It was still being opened by Kafka when I check it with the tool lsof on
{color:#d04437}4/19/2019 {color:#333333}before server rolling restart with
retention hours adjustment{color}{color}.
java 20281 0 1678u REG 202,32 136866627 1074562295
/kafka/data/.../00000000002581377820.log
```
was:Currently, when a consumer falls out of a consumer group, it will restart
processing from the last checkpointed offset. However, this design could result
in a lag which some users could not afford to let happen. For example, lets say
a consumer crashed at offset 100, with the last checkpointed offset being at
70. When it recovers at a later offset (say, 120), it will be behind by an
offset range of 50 (120 - 70). This is because the consumer restarted at 70,
forcing it to reprocess old data. To avoid this from happening, one option
would be to allow the current consumer to start processing not from the last
checkpointed offset (which is 70 in the example), but from 120 where it
recovers. Meanwhile, a new KafkaConsumer will be instantiated and start reading
from offset 70 in concurrency with the old process, and will be terminated once
it reaches 120. In this manner, a considerable amount of lag can be avoided,
particularly since the old consumer could proceed as if nothing had happened.
> Kafka retention hour is not working
> -----------------------------------
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
> Issue Type: Bug
> Components: log
> Reporter: Jiangtao Liu
> Assignee: Richard Yu
> Priority: Major
> Labels: storage
>
> Kafka log cannot be deleted after the configured retention hours (12 hours
> for log retention).
> Is it reproducible?
> I am not sure since our Kafka cluster is working well over 1.5 years without
> retention issue until 4/13/2019 ~ 4/20/2019.
>
> is it related to the active segment?
> as I know Kafka will not delete an active segment, my case those old logs are
> not activated, they should be inactivated.
>
> What's current status?
> Those old logs have been deleted after I manually rolling Kafka server with
> retention hours adjustment.
> A sample old log
> ```
> // it has been there from 4/12
> -rw-r--r-- 1 root root 136866627{color:#d04437} Apr 12{color} 04:33
> 00000000002581377820.log
> // It was still being opened by Kafka when I check it with the tool lsof on
> {color:#d04437}4/19/2019 {color:#333333}before server rolling restart with
> retention hours adjustment{color}{color}.
> java 20281 0 1678u REG 202,32 136866627 1074562295
> /kafka/data/.../00000000002581377820.log
> ```
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
