[ https://issues.apache.org/jira/browse/KAFKA-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Manikumar resolved KAFKA-2623. ------------------------------ Resolution: Fixed Time-based log retention is enforced in KIP-33. Pl reopen if you think the issue still exists > Kakfa broker not deleting logs after configured retention time properly > ----------------------------------------------------------------------- > > Key: KAFKA-2623 > URL: https://issues.apache.org/jira/browse/KAFKA-2623 > Project: Kafka > Issue Type: Bug > Components: log > Affects Versions: 0.8.2.0 > Environment: DISTRIB_ID=Ubuntu > DISTRIB_RELEASE=12.04 > DISTRIB_CODENAME=precise > DISTRIB_DESCRIPTION="Ubuntu 12.04.5 LTS" > NAME="Ubuntu" > VERSION="12.04.5 LTS, Precise Pangolin" > ID=ubuntu > ID_LIKE=debian > PRETTY_NAME="Ubuntu precise (12.04.5 LTS)" > VERSION_ID="12.04" > Reporter: Hao Zhang > Assignee: Jay Kreps > > Context: > To get an accurate estimate on how much retention we have for each > topic/partition, we have a cron job iterating each topic/partition folder on > each node of a cluster, measuring the timestamp difference between the newest > and oldest log files. > Problem: > We notice that it's very common that between leaders and followers, the time > differences are vastly different. On the leader the timestamp differences are > normally about a week (our retention policy), but on the follower the > timestamp differences can sometimes range between just a few hours to 2-3 > days. > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001536840178.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001537497855.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001538155208.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001538811692.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001539468154.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001540122891.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001540775681.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001541430669.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001542088333.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001542746722.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001543405006.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001544062197.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001544718413.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001545374173.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001546029145.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001546686144.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001547344190.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001548001698.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001548657672.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001549312958.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001549969014.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001550623380.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001551279821.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001551937920.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001552597354.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001553256336.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001553914505.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001554571426.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001555228277.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001555882081.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001556538902.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001557196332.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001557852974.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001558510709.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001559166839.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001559823667.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001560478631.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001561136505.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001561792222.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001562450149.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001563107321.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001563763826.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001564420526.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001565076456.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001565735877.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001566394151.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001567051743.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001567709678.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001568366114.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001569022963.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 08:09 00000000001569681970.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 10:37 00000000001570340180.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 13:05 00000000001570997358.log > -rw-rw-r-- 1 kloak kloak 256M Oct 4 16:11 00000000001571654289.log > -rw-rw-r-- 1 kloak kloak 256M Oct 5 01:56 00000000001572310787.log > -rw-rw-r-- 1 kloak kloak 256M Oct 5 05:20 00000000001572967484.log > -rw-rw-r-- 1 kloak kloak 256M Oct 5 08:19 00000000001573626503.log > -rw-rw-r-- 1 kloak kloak 256M Oct 5 11:02 00000000001574284909.log > -rw-rw-r-- 1 kloak kloak 256M Oct 5 13:49 00000000001574944379.log > -rw-rw-r-- 1 kloak kloak 256M Oct 5 18:26 00000000001575603057.log > -rw-rw-r-- 1 kloak kloak 28M Oct 5 21:02 00000000001576258146.log > Observation: > We noticed that there are a lot of index files with size 200. The modified > timestamp of these index files are pretty much identical. > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001536840178.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001537497855.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001538155208.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001538811692.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001539468154.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001540122891.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001540775681.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001541430669.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001542088333.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001542746722.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001543405006.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001544062197.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001544718413.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001545374173.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001546029145.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001546686144.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001547344190.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001548001698.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001548657672.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001549312958.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001549969014.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001550623380.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001551279821.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001551937920.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001552597354.index > -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001553256336.index > Theory: > Our guess is that the broker is trying to delete these log and index files, > thus the timestamps are all modified at pretty much the same time. However, > for some reason the deletion doesn't succeed and thus leave all the log/index > files with the same timestamp. But there is no log deletion error in the log > whatsoever. -- This message was sent by Atlassian JIRA (v6.4.14#64029)