[
https://issues.apache.org/jira/browse/KAFKA-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hao Zhang updated KAFKA-2623:
-----------------------------
Environment:
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.5 LTS"
NAME="Ubuntu"
VERSION="12.04.5 LTS, Precise Pangolin"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu precise (12.04.5 LTS)"
VERSION_ID="12.04"
was:
hzhang@kloak65-sjc1:/var/kafka-spool/data/dispatch.events.client_eyeballed-7$
cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.5 LTS"
NAME="Ubuntu"
VERSION="12.04.5 LTS, Precise Pangolin"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu precise (12.04.5 LTS)"
VERSION_ID="12.04"
Description:
Context:
To get an accurate estimate on how much retention we have for each
topic/partition, we have a cron job iterating each topic/partition folder on
each node of a cluster, measuring the timestamp difference between the newest
and oldest log files.
Problem:
We notice that it's very common that between leaders and followers, the time
differences are vastly different. On the leader the timestamp differences are
normally about a week (our retention policy), but on the follower the timestamp
differences can sometimes range between just a few hours to 2-3 days.
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001536840178.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001537497855.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001538155208.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001538811692.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001539468154.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001540122891.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001540775681.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001541430669.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001542088333.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001542746722.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001543405006.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001544062197.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001544718413.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001545374173.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001546029145.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001546686144.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001547344190.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001548001698.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001548657672.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001549312958.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001549969014.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001550623380.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001551279821.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001551937920.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001552597354.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001553256336.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001553914505.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001554571426.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001555228277.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001555882081.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001556538902.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001557196332.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001557852974.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001558510709.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001559166839.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001559823667.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001560478631.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001561136505.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001561792222.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001562450149.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001563107321.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001563763826.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001564420526.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001565076456.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001565735877.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001566394151.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001567051743.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001567709678.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001568366114.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001569022963.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 08:09 00000000001569681970.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 10:37 00000000001570340180.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 13:05 00000000001570997358.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 16:11 00000000001571654289.log
-rw-rw-r-- 1 kloak kloak 256M Oct 5 01:56 00000000001572310787.log
-rw-rw-r-- 1 kloak kloak 256M Oct 5 05:20 00000000001572967484.log
-rw-rw-r-- 1 kloak kloak 256M Oct 5 08:19 00000000001573626503.log
-rw-rw-r-- 1 kloak kloak 256M Oct 5 11:02 00000000001574284909.log
-rw-rw-r-- 1 kloak kloak 256M Oct 5 13:49 00000000001574944379.log
-rw-rw-r-- 1 kloak kloak 256M Oct 5 18:26 00000000001575603057.log
-rw-rw-r-- 1 kloak kloak 28M Oct 5 21:02 00000000001576258146.log
Observation:
We noticed that there are a lot of index files with size 200. The modified
timestamp of these index files are pretty much identical.
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001536840178.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001537497855.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001538155208.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001538811692.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001539468154.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001540122891.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001540775681.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001541430669.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001542088333.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001542746722.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001543405006.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001544062197.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001544718413.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001545374173.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001546029145.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001546686144.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001547344190.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001548001698.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001548657672.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001549312958.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001549969014.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001550623380.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001551279821.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001551937920.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001552597354.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001553256336.index
Theory:
Our guess is that the broker is trying to delete these log and index files,
thus the timestamps are all modified at pretty much the same time. However, for
some reason the deletion doesn't succeed and thus leave all the log/index files
with the same timestamp. But there is no log deletion error in the log
whatsoever.
was:
Context:
To get an accurate estimate on how much retention we have for each
topic/partition, we have a cron job iterating each topic/partition folder on
each node of a cluster, measuring the timestamp difference between the newest
and oldest log files.
Problem:
We notice that it's very common that between leaders and followers, the time
differences are vastly different. On the leader the timestamp differences are
normally about a week (our retention policy), but on the follower the timestamp
differences can sometimes range between just a few hours to 2-3 days.
hzhang@kloak65-sjc1:/var/kafka-spool/data/dispatch.events.client_eyeballed-7$
ls -lrht *.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001536840178.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001537497855.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001538155208.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001538811692.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001539468154.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001540122891.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001540775681.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001541430669.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001542088333.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001542746722.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001543405006.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001544062197.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001544718413.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001545374173.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001546029145.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001546686144.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001547344190.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001548001698.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001548657672.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001549312958.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001549969014.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001550623380.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001551279821.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001551937920.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001552597354.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001553256336.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001553914505.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001554571426.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001555228277.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001555882081.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001556538902.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001557196332.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001557852974.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001558510709.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001559166839.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001559823667.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001560478631.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001561136505.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001561792222.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001562450149.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001563107321.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001563763826.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001564420526.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001565076456.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001565735877.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001566394151.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001567051743.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001567709678.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001568366114.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001569022963.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 08:09 00000000001569681970.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 10:37 00000000001570340180.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 13:05 00000000001570997358.log
-rw-rw-r-- 1 kloak kloak 256M Oct 4 16:11 00000000001571654289.log
-rw-rw-r-- 1 kloak kloak 256M Oct 5 01:56 00000000001572310787.log
-rw-rw-r-- 1 kloak kloak 256M Oct 5 05:20 00000000001572967484.log
-rw-rw-r-- 1 kloak kloak 256M Oct 5 08:19 00000000001573626503.log
-rw-rw-r-- 1 kloak kloak 256M Oct 5 11:02 00000000001574284909.log
-rw-rw-r-- 1 kloak kloak 256M Oct 5 13:49 00000000001574944379.log
-rw-rw-r-- 1 kloak kloak 256M Oct 5 18:26 00000000001575603057.log
-rw-rw-r-- 1 kloak kloak 28M Oct 5 21:02 00000000001576258146.log
Observation:
We noticed that there are a lot of index files with size 200. The modified
timestamp of these index files are pretty much identical.
hzhang@kloak65-sjc1:/var/kafka-spool/data/dispatch.events.client_eyeballed-7$
ls -lhrt *.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001536840178.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001537497855.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001538155208.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001538811692.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001539468154.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001540122891.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001540775681.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001541430669.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001542088333.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001542746722.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001543405006.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001544062197.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001544718413.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001545374173.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001546029145.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001546686144.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001547344190.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001548001698.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001548657672.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001549312958.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001549969014.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001550623380.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001551279821.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001551937920.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001552597354.index
-rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001553256336.index
Theory:
Our guess is that the broker is trying to delete these log and index files,
thus the timestamps are all modified at pretty much the same time. However, for
some reason the deletion doesn't succeed and thus leave all the log/index files
with the same timestamp. But there is no log deletion error in the log
whatsoever.
> Kakfa broker not deleting logs after configured retention time properly
> -----------------------------------------------------------------------
>
> Key: KAFKA-2623
> URL: https://issues.apache.org/jira/browse/KAFKA-2623
> Project: Kafka
> Issue Type: Bug
> Components: log
> Affects Versions: 0.8.2.0
> Environment: DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=12.04
> DISTRIB_CODENAME=precise
> DISTRIB_DESCRIPTION="Ubuntu 12.04.5 LTS"
> NAME="Ubuntu"
> VERSION="12.04.5 LTS, Precise Pangolin"
> ID=ubuntu
> ID_LIKE=debian
> PRETTY_NAME="Ubuntu precise (12.04.5 LTS)"
> VERSION_ID="12.04"
> Reporter: Hao Zhang
> Assignee: Jay Kreps
>
> Context:
> To get an accurate estimate on how much retention we have for each
> topic/partition, we have a cron job iterating each topic/partition folder on
> each node of a cluster, measuring the timestamp difference between the newest
> and oldest log files.
> Problem:
> We notice that it's very common that between leaders and followers, the time
> differences are vastly different. On the leader the timestamp differences are
> normally about a week (our retention policy), but on the follower the
> timestamp differences can sometimes range between just a few hours to 2-3
> days.
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001536840178.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001537497855.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001538155208.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001538811692.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001539468154.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001540122891.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001540775681.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001541430669.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001542088333.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001542746722.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001543405006.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001544062197.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001544718413.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001545374173.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001546029145.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001546686144.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001547344190.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001548001698.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001548657672.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001549312958.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001549969014.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001550623380.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001551279821.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001551937920.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001552597354.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001553256336.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001553914505.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001554571426.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001555228277.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001555882081.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001556538902.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001557196332.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001557852974.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001558510709.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001559166839.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001559823667.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001560478631.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001561136505.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001561792222.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001562450149.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001563107321.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001563763826.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001564420526.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001565076456.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001565735877.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001566394151.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001567051743.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001567709678.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001568366114.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001569022963.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 08:09 00000000001569681970.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 10:37 00000000001570340180.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 13:05 00000000001570997358.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 16:11 00000000001571654289.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 5 01:56 00000000001572310787.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 5 05:20 00000000001572967484.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 5 08:19 00000000001573626503.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 5 11:02 00000000001574284909.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 5 13:49 00000000001574944379.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 5 18:26 00000000001575603057.log
> -rw-rw-r-- 1 kloak kloak 28M Oct 5 21:02 00000000001576258146.log
> Observation:
> We noticed that there are a lot of index files with size 200. The modified
> timestamp of these index files are pretty much identical.
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001536840178.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001537497855.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001538155208.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001538811692.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001539468154.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001540122891.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001540775681.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001541430669.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001542088333.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001542746722.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001543405006.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001544062197.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001544718413.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001545374173.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001546029145.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001546686144.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001547344190.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001548001698.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001548657672.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001549312958.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001549969014.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001550623380.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001551279821.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001551937920.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001552597354.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001553256336.index
> Theory:
> Our guess is that the broker is trying to delete these log and index files,
> thus the timestamps are all modified at pretty much the same time. However,
> for some reason the deletion doesn't succeed and thus leave all the log/index
> files with the same timestamp. But there is no log deletion error in the log
> whatsoever.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
