[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Description:
What's the issue?
{quote} # There were log segments, which can not be deleted over configured
retention hours.{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a smaller size. e.g log segment
may be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.
# It will be a way used to attack Kafka server.{quote}
What's workaround adopted to resolve this issue?
{quote} # If it's already happened on your Kafka system, you will need to run a
very tricky steps to resolve it.
# If it has not happened on your Kafka system yet, you may need to evaluate
whether you can switch to LogAppendTime for log.message.timestamp.type. {quote}
What are the reproduce steps?
{quote} # Make sure Kafka client and server are not hosted in the same machine.
# Configure log.message.timestamp.type with *CreateTime*, not LogAppendTime.
# Hack Kafka client's system clock time with a *future time*, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
# Send message from Kafka client to server.{quote}
What kinds of things you need to have a look after message handled by Kafka
server?
{quote} # Check the value of timestamp in log segment **.timeindex**.* The
timestamp will be *a future time after `03/04/*2025, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`*. _(Let's
say 035957300794.log is the log segment which first receive the test
client's message. It will be referenced in #3)_
# After testing for couples of hours, there will be lots of log segment rolled
with a smaller size (e.g 50mb) than the configured segment size (e.g 1gb).
# All of log segments including 035957300794.* and new ones, will not
be deleted over retention hours.{quote}
What's the particular logic to cause this issue?
{quote} # No deletable log segments will be returned from the following method.
private def deletableSegments(predicate: (LogSegment, Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]].{quote}
was:
What's the issue?
{quote} # There were log segments, which can not be deleted over configured
retention hours.{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a smaller size. e.g log segment
may be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.
# It will be a way used to attack Kafka server.{quote}
What's workaround adopted to resolve this issue?
{quote} # If it's already happened on your Kafka system, you will need to run a
very tricky steps to resolve it.
# If it has not happened on your Kafka system yet, you may need to evaluate
whether you can switch to LogAppendTime for log.message.timestamp.type. {quote}
What are the reproduce steps?
{quote} # Make sure Kafka client and server are not hosted in the same machine.
# Configure log.message.timestamp.type with *CreateTime*, not LogAppendTime.
# Hack Kafka client's system clock time with a *future time*, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
# Send message from Kafka client to server.{quote}
What kinds of things you need to have a look after message handled by Kafka
server?
{quote} # Check the timestamp in segment *.timeindex and record in segment
*.log. You will see all the timestamp values in **.timeindex are messed up with
a future time after `03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`. (Let's
say 035957300794.log is the log segment which first receive the test
client's message. It will be referenced in #3)
# You will also see the log segment will be rolled with a smaller size (e.g
50mb) than configured segment max size (e.g 1gb).
# All of log segments including 035957300794.* and new rolled, will
not be deleted over retention hours.{quote}
What's the particular logic to cause this issue?
{quote} # No deletable log segments will be returned from the following method.
private def deletableSegments(predicate: (LogSegment, Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]].{quote}
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
> ---
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
> Issue Type: Bug
>
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Description:
What's the issue?
{quote} # There were log segments, which can not be deleted over configured
retention hours.{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a smaller size. e.g log segment
may be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.
# It will be a way used to attack Kafka server.{quote}
What's workaround adopted to resolve this issue?
{quote} # If it's already happened on your Kafka system, you will need to run a
very tricky steps to resolve it.
# If it has not happened on your Kafka system yet, you may need to evaluate
whether you can switch to LogAppendTime for log.message.timestamp.type. {quote}
What are the reproduce steps?
{quote} # Make sure Kafka client and server are not hosted in the same machine.
# Configure log.message.timestamp.type with *CreateTime*, not LogAppendTime.
# Hack Kafka client's system clock time with a *future time*, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
# Send message from Kafka client to server.{quote}
What kinds of things you need to have a look after message handled by Kafka
server?
{quote} # Check the timestamp in segment *.timeindex and record in segment
*.log. You will see all the timestamp values in **.timeindex are messed up with
a future time after `03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`. (Let's
say 035957300794.log is the log segment which first receive the test
client's message. It will be referenced in #3)
# You will also see the log segment will be rolled with a smaller size (e.g
50mb) than configured segment max size (e.g 1gb).
# All of log segments including 035957300794.* and new rolled, will
not be deleted over retention hours.{quote}
What's the particular logic to cause this issue?
{quote} # No deletable log segments will be returned from the following method.
private def deletableSegments(predicate: (LogSegment, Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]].{quote}
was:
What's the issue?
{quote} # There were log segments, which can not be deleted over configured
retention hours.{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a smaller size. e.g log segment
may be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.
# It will be a way used to attack Kafka server.{quote}
What's workaround adopted to resolve this issue?
{quote} # If it's already happened on your Kafka system, you will need to run a
very tricky steps to resolve it.
# If it has not happened on your Kafka system yet, you may need to evaluate
whether you can switch to LogAppendTime for log.message.timestamp.type. {quote}
What are the reproduce steps?
{quote} # Make sure Kafka client and server are not hosted in the same machine.
# Configure log.message.timestamp.type with *CreateTime*, not LogAppendTime.
# Hack Kafka client's system clock time with a *future time*, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
# Send message from Kafka client to server.{quote}
What kinds of things you need to have a look after message handled by Kafka
server?
{quote} # Check the timestamp in segment *.timeindex and record in segment
*.log. You will see all the timestamp values in **.timeindex are messed up with
a future time after `03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`. (Let's
say 035957300794.log is the log segment which first receive the test
client's message. It will be referenced in #3)
# You will also see the log segment will be rolled with a smaller size (e.g
50mb) than configured segment max size (e.g 1gb).
# All of log segments including 035957300794.* and new rolled, will
not be deleted over retention hours.{quote}
What's the particular logic to cause this issue?
{quote} # private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.{color:#172b4d} {color}{quote}
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
> ---
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Attachment: space issue.png
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
> ---
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
> Issue Type: Bug
> Components: log, log cleaner, logging
>Affects Versions: 1.1.1
>Reporter: Jiangtao Liu
>Priority: Major
> Labels: storage
> Attachments: space issue.png
>
>
> What's the issue?
> {quote} # There were log segments, which can not be deleted over configured
> retention hours.{quote}
> What are impacts?
> {quote} # Log space keep in increasing and finally cause space shortage.
> # There are lots of log segment rolled with a smaller size. e.g log segment
> may be only 50mb, not the expected 1gb.
> # Kafka stream or client may experience missing data.
> # It will be a way used to attack Kafka server.{quote}
> What's workaround adopted to resolve this issue?
> {quote} # If it's already happened on your Kafka system, you will need to run
> a very tricky steps to resolve it.
> # If it has not happened on your Kafka system yet, you may need to evaluate
> whether you can switch to LogAppendTime for log.message.timestamp.type.
> {quote}
> What are the reproduce steps?
> {quote} # Make sure Kafka client and server are not hosted in the same
> machine.
> # Configure log.message.timestamp.type with *CreateTime*, not LogAppendTime.
> # Hack Kafka client's system clock time with a *future time*, e.g
> 03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
> # Send message from Kafka client to server.{quote}
> What kinds of things you need to have a look after message handled by Kafka
> server?
> {quote} # Check the timestamp in segment *.timeindex and record in segment
> *.log. You will see all the timestamp values in **.timeindex are messed up
> with a future time after `03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`. (Let's
> say 035957300794.log is the log segment which first receive the test
> client's message. It will be referenced in #3)
> # You will also see the log segment will be rolled with a smaller size (e.g
> 50mb) than configured segment max size (e.g 1gb).
> # All of log segments including 035957300794.* and new rolled, will
> not be deleted over retention hours.{quote}
> What's the particular logic to cause this issue?
> {quote} # private def deletableSegments(predicate: (LogSegment,
> Option[LogSegment]) =>
> Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
> will always return empty deletable log segments.{color:#172b4d}
> {color}{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Attachment: Screen Shot 2020-04-15 at 11.02.55 AM.png
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
> ---
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
> Issue Type: Bug
> Components: log, log cleaner, logging
>Affects Versions: 1.1.1
>Reporter: Jiangtao Liu
>Priority: Major
> Labels: storage
>
> What's the issue?
> {quote} # There were log segments, which can not be deleted over configured
> retention hours.{quote}
> What are impacts?
> {quote} # Log space keep in increasing and finally cause space shortage.
> # There are lots of log segment rolled with a smaller size. e.g log segment
> may be only 50mb, not the expected 1gb.
> # Kafka stream or client may experience missing data.
> # It will be a way used to attack Kafka server.{quote}
> What's workaround adopted to resolve this issue?
> {quote} # If it's already happened on your Kafka system, you will need to run
> a very tricky steps to resolve it.
> # If it has not happened on your Kafka system yet, you may need to evaluate
> whether you can switch to LogAppendTime for log.message.timestamp.type.
> {quote}
> What are the reproduce steps?
> {quote} # Make sure Kafka client and server are not hosted in the same
> machine.
> # Configure log.message.timestamp.type with *CreateTime*, not LogAppendTime.
> # Hack Kafka client's system clock time with a *future time*, e.g
> 03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
> # Send message from Kafka client to server.{quote}
> What kinds of things you need to have a look after message handled by Kafka
> server?
> {quote} # Check the timestamp in segment *.timeindex and record in segment
> *.log. You will see all the timestamp values in **.timeindex are messed up
> with a future time after `03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`. (Let's
> say 035957300794.log is the log segment which first receive the test
> client's message. It will be referenced in #3)
> # You will also see the log segment will be rolled with a smaller size (e.g
> 50mb) than configured segment max size (e.g 1gb).
> # All of log segments including 035957300794.* and new rolled, will
> not be deleted over retention hours.{quote}
> What's the particular logic to cause this issue?
> {quote} # private def deletableSegments(predicate: (LogSegment,
> Option[LogSegment]) =>
> Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
> will always return empty deletable log segments.{color:#172b4d}
> {color}{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Attachment: (was: Screen Shot 2020-04-15 at 11.02.55 AM.png)
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
> ---
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
> Issue Type: Bug
> Components: log, log cleaner, logging
>Affects Versions: 1.1.1
>Reporter: Jiangtao Liu
>Priority: Major
> Labels: storage
>
> What's the issue?
> {quote} # There were log segments, which can not be deleted over configured
> retention hours.{quote}
> What are impacts?
> {quote} # Log space keep in increasing and finally cause space shortage.
> # There are lots of log segment rolled with a smaller size. e.g log segment
> may be only 50mb, not the expected 1gb.
> # Kafka stream or client may experience missing data.
> # It will be a way used to attack Kafka server.{quote}
> What's workaround adopted to resolve this issue?
> {quote} # If it's already happened on your Kafka system, you will need to run
> a very tricky steps to resolve it.
> # If it has not happened on your Kafka system yet, you may need to evaluate
> whether you can switch to LogAppendTime for log.message.timestamp.type.
> {quote}
> What are the reproduce steps?
> {quote} # Make sure Kafka client and server are not hosted in the same
> machine.
> # Configure log.message.timestamp.type with *CreateTime*, not LogAppendTime.
> # Hack Kafka client's system clock time with a *future time*, e.g
> 03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
> # Send message from Kafka client to server.{quote}
> What kinds of things you need to have a look after message handled by Kafka
> server?
> {quote} # Check the timestamp in segment *.timeindex and record in segment
> *.log. You will see all the timestamp values in **.timeindex are messed up
> with a future time after `03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`. (Let's
> say 035957300794.log is the log segment which first receive the test
> client's message. It will be referenced in #3)
> # You will also see the log segment will be rolled with a smaller size (e.g
> 50mb) than configured segment max size (e.g 1gb).
> # All of log segments including 035957300794.* and new rolled, will
> not be deleted over retention hours.{quote}
> What's the particular logic to cause this issue?
> {quote} # private def deletableSegments(predicate: (LogSegment,
> Option[LogSegment]) =>
> Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
> will always return empty deletable log segments.{color:#172b4d}
> {color}{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Attachment: (was: Screen Shot 2019-04-20 at 10.57.59 PM.png)
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
> ---
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
> Issue Type: Bug
> Components: log, log cleaner, logging
>Affects Versions: 1.1.1
>Reporter: Jiangtao Liu
>Priority: Major
> Labels: storage
>
> What's the issue?
> {quote} # There were log segments, which can not be deleted over configured
> retention hours.{quote}
> What are impacts?
> {quote} # Log space keep in increasing and finally cause space shortage.
> # There are lots of log segment rolled with a smaller size. e.g log segment
> may be only 50mb, not the expected 1gb.
> # Kafka stream or client may experience missing data.
> # It will be a way used to attack Kafka server.{quote}
> What's workaround adopted to resolve this issue?
> {quote} # If it's already happened on your Kafka system, you will need to run
> a very tricky steps to resolve it.
> # If it has not happened on your Kafka system yet, you may need to evaluate
> whether you can switch to LogAppendTime for log.message.timestamp.type.
> {quote}
> What are the reproduce steps?
> {quote} # Make sure Kafka client and server are not hosted in the same
> machine.
> # Configure log.message.timestamp.type with *CreateTime*, not LogAppendTime.
> # Hack Kafka client's system clock time with a *future time*, e.g
> 03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
> # Send message from Kafka client to server.{quote}
> What kinds of things you need to have a look after message handled by Kafka
> server?
> {quote} # Check the timestamp in segment *.timeindex and record in segment
> *.log. You will see all the timestamp values in **.timeindex are messed up
> with a future time after `03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`. (Let's
> say 035957300794.log is the log segment which first receive the test
> client's message. It will be referenced in #3)
> # You will also see the log segment will be rolled with a smaller size (e.g
> 50mb) than configured segment max size (e.g 1gb).
> # All of log segments including 035957300794.* and new rolled, will
> not be deleted over retention hours.{quote}
> What's the particular logic to cause this issue?
> {quote} # private def deletableSegments(predicate: (LogSegment,
> Option[LogSegment]) =>
> Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
> will always return empty deletable log segments.{color:#172b4d}
> {color}{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Description:
What's the issue?
{quote} # There were log segments, which can not be deleted over configured
retention hours.{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a smaller size. e.g log segment
may be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.
# It will be a way used to attack Kafka server.{quote}
What's workaround adopted to resolve this issue?
{quote} # If it's already happened on your Kafka system, you will need to run a
very tricky steps to resolve it.
# If it has not happened on your Kafka system yet, you may need to evaluate
whether you can switch to LogAppendTime for log.message.timestamp.type. {quote}
What are the reproduce steps?
{quote} # Make sure Kafka client and server are not hosted in the same machine.
# Configure log.message.timestamp.type with *CreateTime*, not LogAppendTime.
# Hack Kafka client's system clock time with a *future time*, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
# Send message from Kafka client to server.{quote}
What kinds of things you need to have a look after message handled by Kafka
server?
{quote} # Check the timestamp in segment *.timeindex and record in segment
*.log. You will see all the timestamp values in **.timeindex are messed up with
a future time after `03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`. (Let's
say 035957300794.log is the log segment which first receive the test
client's message. It will be referenced in #3)
# You will also see the log segment will be rolled with a smaller size (e.g
50mb) than configured segment max size (e.g 1gb).
# All of log segments including 035957300794.* and new rolled, will
not be deleted over retention hours.{quote}
What's the particular logic to cause this issue?
{quote} # private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.{color:#172b4d} {color}{quote}
was:
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.
# It will be a way which may be used to attack Kafka server.{quote}
What's workaround adopted to resolve this issue?
{quote} # if it's already happened on your Kafka system, you will follow a very
tricky steps to resolve it. (I have documented it at
[here|[https://medium.com/@jiangtaoliu/a-kafka-pitfall-when-to-set-log-message-timestamp-type-to-createtime-c17846813ca3]])
# if it has not happened on your Kafka system yet, you may need to evaluate
whether you can switch to LogAppendTime for log.message.timestamp.type. {quote}
What are the reproduce steps?
{quote} # Make sure Kafka client and server are not hosted in the same machine.
# Configure log.message.timestamp.type with *CreateTime*, not LogAppendTime.
# Hack Kafka client's system clock time with a *future time*, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
# Send message from Kafka client to server.{quote}
What kinds of things you need to have a look after message handled by Kafka
server?
{quote} # Check the timestamp in segment *.timeindex (e.g
035957300794.timeindex) and record in segment *.log. You will see all
the timestamp values in **.timeindex are messed up with a future time after
`03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`. (let's
say 035957300794.log is the log segment which receive the record
firstly. Later we will use it in #3)
# You will also see the log segment will be rolled with a smaller size (e.g
50mb) than configured segment max size (e.g 1gb).
# All of log segments including 035957300794.* and new rolled, will
not be deleted over retention hours.{quote}
What's the particular logic to cause this issue?
{quote} # private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.{color:#172b4d} {color}{quote}
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
>
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Description:
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.
# It will be a way which may be used to attack Kafka server.{quote}
What's workaround adopted to resolve this issue?
{quote} # if it's already happened on your Kafka system, you will follow a very
tricky steps to resolve it. (I have documented it at
[here|[https://medium.com/@jiangtaoliu/a-kafka-pitfall-when-to-set-log-message-timestamp-type-to-createtime-c17846813ca3]])
# if it has not happened on your Kafka system yet, you may need to evaluate
whether you can switch to LogAppendTime for log.message.timestamp.type. {quote}
What are the reproduce steps?
{quote} # Make sure Kafka client and server are not hosted in the same machine.
# Configure log.message.timestamp.type with *CreateTime*, not LogAppendTime.
# Hack Kafka client's system clock time with a *future time*, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
# Send message from Kafka client to server.{quote}
What kinds of things you need to have a look after message handled by Kafka
server?
{quote} # Check the timestamp in segment *.timeindex (e.g
035957300794.timeindex) and record in segment *.log. You will see all
the timestamp values in **.timeindex are messed up with a future time after
`03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`. (let's
say 035957300794.log is the log segment which receive the record
firstly. Later we will use it in #3)
# You will also see the log segment will be rolled with a smaller size (e.g
50mb) than configured segment max size (e.g 1gb).
# All of log segments including 035957300794.* and new rolled, will
not be deleted over retention hours.{quote}
What's the particular logic to cause this issue?
{quote} # private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.{color:#172b4d} {color}{quote}
was:
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.
# It will be a way which may be used to attack Kafka server.{quote}
What's workaround adopted to resolve this issue?
{quote} # if it's already happened on your Kafka system, you will follow a very
tricky steps to resolve it.
# if it has not happened on your Kafka system yet, you may need to evaluate
whether you can switch to LogAppendTime for log.message.timestamp.type. {quote}
What are the reproduce steps?
{quote} # Make sure Kafka client and server are not hosted in the same machine.
# Configure log.message.timestamp.type with *CreateTime*, not LogAppendTime.
# Hack Kafka client's system clock time with a *future time*, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
# Send message from Kafka client to server.{quote}
What kinds of things you need to have a look after message handled by Kafka
server?
{quote} # Check the timestamp in segment *.timeindex (e.g
035957300794.timeindex) and record in segment *.log. You will see all
the timestamp values in **.timeindex are messed up with a future time after
`03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`. (let's
say 035957300794.log is the log segment which receive the record
firstly. Later we will use it in #3)
# You will also see the log segment will be rolled with a smaller size (e.g
50mb) than configured segment max size (e.g 1gb).
# All of log segments including 035957300794.* and new rolled, will
not be deleted over retention hours.{quote}
What's the particular logic to cause this issue?
{quote} # private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.{color:#172b4d} {color}{quote}
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
>
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Description:
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.
# It will be a way which may be used to attack Kafka server.{quote}
What's workaround adopted to resolve this issue?
{quote} # if it's already happened on your Kafka system, you will follow a very
tricky steps to resolve it.
# if it has not happened on your Kafka system yet, you may need to evaluate
whether you can switch to LogAppendTime for log.message.timestamp.type. {quote}
What are the reproduce steps?
{quote} # Make sure Kafka client and server are not hosted in the same machine.
# Configure log.message.timestamp.type with *CreateTime*, not LogAppendTime.
# Hack Kafka client's system clock time with a *future time*, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
# Send message from Kafka client to server.{quote}
What kinds of things you need to have a look after message handled by Kafka
server?
{quote} # Check the timestamp in segment *.timeindex (e.g
035957300794.timeindex) and record in segment *.log. You will see all
the timestamp values in **.timeindex are messed up with a future time after
`03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`. (let's
say 035957300794.log is the log segment which receive the record
firstly. Later we will use it in #3)
# You will also see the log segment will be rolled with a smaller size (e.g
50mb) than configured segment max size (e.g 1gb).
# All of log segments including 035957300794.* and new rolled, will
not be deleted over retention hours.{quote}
What's the particular logic to cause this issue?
{quote} # private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.{color:#172b4d} {color}{quote}
was:
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.
# It will be a way which may be used to attack Kafka server.{quote}
What's workaround adopted to resolve this issue?
What are the reproduce steps?
# Make sure Kafka client and server are not hosted in the same machine.
# Configure log.message.timestamp.type with *CreateTime*, not LogAppendTime.
# Hack Kafka client's system clock time with a *future time*, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
# Send message from Kafka client to server.
What kinds of things you need to have a look after message handled by Kafka
server?
#
{quote}Check the timestamp in segment *.timeindex (e.g
035957300794.timeindex) and record in segment *.log. You will see all
the timestamp values in **.timeindex are messed up with a future time after
`03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`. (let's
say 035957300794.log is the log segment which receive the record
firstly. Later we will use it in #3)
{quote}
#
{quote}You will also see the log segment will be rolled with a smaller size
(e.g 50mb) than configured segment max size (e.g 1gb).
{quote}
#
{quote}All of log segments including 035957300794.* and new rolled,
will not be deleted over retention hours.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.{color:#172b4d} {color}
{quote}
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
> ---
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
> Issue Type: Bug
> Components: log, log cleaner, logging
>Affects Versions: 1.1.1
>Reporter: Jiangtao Liu
>Priority: Major
>
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Description:
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.
# It will be a way which may be used to attack Kafka server.{quote}
What's workaround adopted to resolve this issue?
What are the reproduce steps?
# Make sure Kafka client and server are not hosted in the same machine.
# Configure log.message.timestamp.type with *CreateTime*, not LogAppendTime.
# Hack Kafka client's system clock time with a *future time*, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
# Send message from Kafka client to server.
What kinds of things you need to have a look after message handled by Kafka
server?
#
{quote}Check the timestamp in segment *.timeindex (e.g
035957300794.timeindex) and record in segment *.log. You will see all
the timestamp values in **.timeindex are messed up with a future time after
`03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`. (let's
say 035957300794.log is the log segment which receive the record
firstly. Later we will use it in #3)
{quote}
#
{quote}You will also see the log segment will be rolled with a smaller size
(e.g 50mb) than configured segment max size (e.g 1gb).
{quote}
#
{quote}All of log segments including 035957300794.* and new rolled,
will not be deleted over retention hours.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.{color:#172b4d} {color}
{quote}
was:
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.
# It will be a way which may be used to attack Kafka server.{quote}
How to reproduce it?
#
{quote}Make sure Kafka client and server are not hosted in the same machine.
{quote}
#
{quote}Configure log.message.timestamp.type with *CreateTime*, not
LogAppendTime.
{quote}
#
{quote}Hack Kafka client's system clock time with a *future time*, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
{quote}
#
{quote}Send message from Kafka client to server.
{quote}
What's Next?
#
{quote}Check the timestamp in segment *.timeindex (e.g
035957300794.timeindex) and record in segment *.log. You will see all
the timestamp values in **.timeindex are messed up with a future time after
`03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`. (let's
say 035957300794.log is the log segment which receive the record
firstly. Later we will use it in #3)
{quote}
#
{quote}You will also see the log segment will be rolled with a smaller size
(e.g 50mb) than configured segment max size (e.g 1gb).
{quote}
#
{quote}All of log segments including 035957300794.* and new rolled,
will not be deleted over retention hours.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.{color:#172b4d} {color}
{quote}
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
> ---
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
> Issue Type: Bug
> Components: log, log cleaner, logging
>Affects Versions: 1.1.1
>Reporter: Jiangtao Liu
>Priority: Major
> Labels: storage
> Attachments: Screen Shot 2019-04-20 at 10.57.59 PM.png
>
>
> What's the issue?
> {quote}There were log segments, which can not be deleted over configured
> retention hours.
> {quote}
> What are impacts?
> {quote} # Log space keep in increasing and finally cause space
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Description:
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.
# It will be a way which may be used to attack Kafka server.{quote}
How to reproduce it?
#
{quote}Make sure Kafka client and server are not hosted in the same machine.
{quote}
#
{quote}Configure log.message.timestamp.type with *CreateTime*, not
LogAppendTime.
{quote}
#
{quote}Hack Kafka client's system clock time with a *future time*, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
{quote}
#
{quote}Send message from Kafka client to server.
{quote}
What's Next?
#
{quote}Check the timestamp in segment *.timeindex (e.g
035957300794.timeindex) and record in segment *.log. You will see all
the timestamp values in **.timeindex are messed up with a future time after
`03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`. (let's
say 035957300794.log is the log segment which receive the record
firstly. Later we will use it in #3)
{quote}
#
{quote}You will also see the log segment will be rolled with a smaller size
(e.g 50mb) than configured segment max size (e.g 1gb).
{quote}
#
{quote}All of log segments including 035957300794.* and new rolled,
will not be deleted over retention hours.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.{color:#172b4d} {color}
{quote}
was:
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.{quote}
How to reproduce it?
#
{quote}Make sure Kafka client and server are not hosted in the same machine.
{quote}
#
{quote}Configure log.message.timestamp.type with *CreateTime*, not
LogAppendTime.
{quote}
#
{quote}Hack Kafka client's system clock time with a *future time*, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
{quote}
#
{quote}Send message from Kafka client to server.
{quote}
What's Next?
#
{quote}Check the timestamp in segment *.timeindex (e.g
035957300794.timeindex) and record in segment *.log. You will see all
the timestamp values in **.timeindex are messed up with a future time after
`03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`. (let's
say 035957300794.log is the log segment which receive the record
firstly. Later we will use it in #3)
{quote}
#
{quote}You will also see the log segment will be rolled with a smaller size
(e.g 50mb) than configured segment max size (e.g 1gb).
{quote}
#
{quote}All of log segments including 035957300794.* and new rolled,
will not be deleted over retention hours.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.{color:#172b4d} {color}
{quote}
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
> ---
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
> Issue Type: Bug
> Components: log, log cleaner, logging
>Affects Versions: 1.1.1
>Reporter: Jiangtao Liu
>Priority: Major
> Labels: storage
> Attachments: Screen Shot 2019-04-20 at 10.57.59 PM.png
>
>
> What's the issue?
> {quote}There were log segments, which can not be deleted over configured
> retention hours.
> {quote}
> What are impacts?
> {quote} # Log space keep in increasing and finally cause space shortage.
> # There are lots of log segment rolled with a small size. e.g log segment
> may be only 50mb, not the expected 1gb.
> #
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Description:
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.{quote}
How to reproduce it?
#
{quote}Make sure Kafka client and server are not hosted in the same machine.
{quote}
#
{quote}Configure log.message.timestamp.type with *CreateTime*, not
LogAppendTime.
{quote}
#
{quote}Hack Kafka client's system clock time with a *future time*, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
{quote}
#
{quote}Send message from Kafka client to server.
{quote}
What's Next?
#
{quote}Check the timestamp in segment *.timeindex (e.g
035957300794.timeindex) and record in segment *.log. You will see all
the timestamp values in **.timeindex are messed up with a future time after
`03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`. (let's
say 035957300794.log is the log segment which receive the record
firstly. Later we will use it in #3)
{quote}
#
{quote}You will also see the log segment will be rolled with a smaller size
(e.g 50mb) than configured segment max size (e.g 1gb).
{quote}
#
{quote}All of log segments including 035957300794.* and new rolled,
will not be deleted over retention hours.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.{color:#172b4d} {color}
{quote}
was:
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.{quote}
How to reproduce it?
#
{quote}Make sure Kafka client and server are not on the same machine.
{quote}
#
{quote}Configure log.message.timestamp.type as *CreateTime*, not LogAppendTime.
{quote}
#
{quote}Hack Kafka client's system clock time as a *future time*, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
{quote}
#
{quote}Send message from Kafka client to server.
{quote}
What's Next?
#
{quote}Check the timestamp in log time index and record in log segment(e.g
035957300794.timeindex). You will see all the timestamp values in
*.timeindex are messed up with a future time after `03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
{quote}
#
{quote}You will also see the new rolled log segment's size smaller than the
configured log segment size after waiting for a while.
{quote}
#
{quote}Log segment (035957300794.log|timeindex) and the rest of new
rolled log segments will not be deleted over retention hours.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.{color:#172b4d} {color}
{quote}
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
> ---
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
> Issue Type: Bug
> Components: log, log cleaner, logging
>Affects Versions: 1.1.1
>Reporter: Jiangtao Liu
>Priority: Major
> Labels: storage
> Attachments: Screen Shot 2019-04-20 at 10.57.59 PM.png
>
>
> What's the issue?
> {quote}There were log segments, which can not be deleted over configured
> retention hours.
> {quote}
> What are impacts?
> {quote} # Log space keep in increasing and finally cause space shortage.
> # There are lots of log segment rolled with a small size. e.g log segment
> may be only 50mb, not the expected 1gb.
> # Kafka stream or client may experience missing data.{quote}
> How to reproduce it?
> #
> {quote}Make sure Kafka client and server are not hosted in the same machine.
> {quote}
> #
>
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Description:
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.{quote}
How to reproduce it?
#
{quote}Make sure Kafka client and server are not on the same machine.
{quote}
#
{quote}Configure log.message.timestamp.type as *CreateTime*, not LogAppendTime.
{quote}
#
{quote}Hack Kafka client's system clock time as a *future time*, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
{quote}
#
{quote}Send message from Kafka client to server.
{quote}
What's Next?
#
{quote}Check the timestamp in log time index and record in log segment(e.g
035957300794.timeindex). You will see all the timestamp values in
*.timeindex are messed up with a future time after `03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
{quote}
#
{quote}You will also see the new rolled log segment's size smaller than the
configured log segment size after waiting for a while.
{quote}
#
{quote}Log segment (035957300794.log|timeindex) and the rest of new
rolled log segments will not be deleted over retention hours.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.{color:#172b4d} {color}
{quote}
was:
I have a particular blog to explain the whole context at here
[https://medium.com/@jiangtaoliu/a-kafka-pitfall-when-to-set-log-message-timestamp-type-to-createtime-c17846813ca3]
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.{quote}
How to reproduce it?
#
{quote}Make sure Kafka client and server are not on the same machine.{quote}
#
{quote}Configure log.message.timestamp.type as *CreateTime*, not
LogAppendTime.{quote}
#
{quote}Hack Kafka client's system clock time as a *future time*, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]{quote}
#
{quote}Send message from Kafka client to server.{quote}
What's Next?
#
{quote}Check the timestamp in log time index and record in log segment(e.g
035957300794.timeindex). You will see all the timestamp values in
*.timeindex are messed up with a future time after `03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
{quote}
#
{quote}You will also see the new rolled log segment's size smaller than the
configured log segment size after waiting for a while.
{quote}
#
{quote}Log segment (035957300794.log|timeindex) and the rest of new
rolled log segments will not be deleted over retention hours.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.{color:#172b4d} {color}
{quote}
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
> ---
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
> Issue Type: Bug
> Components: log, log cleaner, logging
>Affects Versions: 1.1.1
>Reporter: Jiangtao Liu
>Priority: Major
> Labels: storage
> Attachments: Screen Shot 2019-04-20 at 10.57.59 PM.png
>
>
> What's the issue?
> {quote}There were log segments, which can not be deleted over configured
> retention hours.
> {quote}
> What are impacts?
> {quote} # Log space keep in increasing and finally cause space shortage.
> # There are lots of log segment rolled with a small size. e.g log segment
> may be only 50mb, not the expected 1gb.
> # Kafka stream or client may experience missing data.{quote}
> How to reproduce it?
> #
> {quote}Make sure Kafka client and server
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Description:
I have a particular blog to explain the whole context at here
[https://medium.com/@jiangtaoliu/a-kafka-pitfall-when-to-set-log-message-timestamp-type-to-createtime-c17846813ca3]
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.{quote}
How to reproduce it?
#
{quote}Make sure Kafka client and server are not on the same machine.{quote}
#
{quote}Configure log.message.timestamp.type as *CreateTime*, not
LogAppendTime.{quote}
#
{quote}Hack Kafka client's system clock time as a *future time*, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]{quote}
#
{quote}Send message from Kafka client to server.{quote}
What's Next?
#
{quote}Check the timestamp in log time index and record in log segment(e.g
035957300794.timeindex). You will see all the timestamp values in
*.timeindex are messed up with a future time after `03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
{quote}
#
{quote}You will also see the new rolled log segment's size smaller than the
configured log segment size after waiting for a while.
{quote}
#
{quote}Log segment (035957300794.log|timeindex) and the rest of new
rolled log segments will not be deleted over retention hours.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.{color:#172b4d} {color}
{quote}
was:
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.{quote}
How to reproduce it?
#
{quote}Configure message.timestamp.type as CreateTime.
{quote}
#
{quote}Reset Kafka client's system clock time as a future time, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
Send message to Kafka.
{quote}
What's Next?
#
{quote}Go to check the timestamp in log time index and record (e.g
035957300794.log|timeindex). You should see all the value of the
timestamp is messed up with future time after `03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
{quote}
#
{quote}You may also see the new rolled log segment's size smaller than the
configured log segment size).
{quote}
#
{quote}Log segment (035957300794.log|timeindex) and any of rolled log
segments will not be deleted.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.
{quote}
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
> ---
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
> Issue Type: Bug
> Components: log, log cleaner, logging
>Affects Versions: 1.1.1
>Reporter: Jiangtao Liu
>Priority: Major
> Labels: storage
> Attachments: Screen Shot 2019-04-20 at 10.57.59 PM.png
>
>
> I have a particular blog to explain the whole context at here
> [https://medium.com/@jiangtaoliu/a-kafka-pitfall-when-to-set-log-message-timestamp-type-to-createtime-c17846813ca3]
> What's the issue?
> {quote}There were log segments, which can not be deleted over configured
> retention hours.
> {quote}
> What are impacts?
> {quote} # Log space keep in increasing and finally cause space shortage.
> # There are lots of log segment rolled with a small size. e.g log segment
> may be only 50mb, not the expected 1gb.
> # Kafka stream or client may experience missing data.{quote}
> How to reproduce it?
> #
> {quote}Make sure Kafka client and server are not on the same machine.{quote}
> #
>
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Description:
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.{quote}
How to reproduce it?
#
{quote}Configure message.timestamp.type as CreateTime.
{quote}
#
{quote}Reset Kafka client's system clock time as a future time, e.g
03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
Send message to Kafka.
{quote}
What's Next?
#
{quote}Go to check the timestamp in log time index and record (e.g
035957300794.log|timeindex). You should see all the value of the
timestamp is messed up with future time after `03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
{quote}
#
{quote}You may also see the new rolled log segment's size smaller than the
configured log segment size).
{quote}
#
{quote}Log segment (035957300794.log|timeindex) and any of rolled log
segments will not be deleted.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.
{quote}
was:
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.{quote}
How to reproduce it?
#
{quote}Configure message.timestamp.type as CreateTime.
{quote}
#
{quote}Reset Kafka client's system clock time a future time, e.g 03/04/*2025*,
3:25:52 PM [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
{quote}
What's Next?
#
{quote}Go to check the timestamp in log time index and record (e.g
035957300794.log|timeindex). You should see all the value of the
timestamp is messed up with future time after `03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
{quote}
#
{quote}You may also see the new rolled log segment's size smaller than the
configured log segment size).
{quote}
#
{quote}Log segment (035957300794.log|timeindex) and any of rolled log
segments will not be deleted.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.
{quote}
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
> ---
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
> Issue Type: Bug
> Components: log, log cleaner, logging
>Affects Versions: 1.1.1
>Reporter: Jiangtao Liu
>Priority: Major
> Labels: storage
> Attachments: Screen Shot 2019-04-20 at 10.57.59 PM.png
>
>
> What's the issue?
> {quote}There were log segments, which can not be deleted over configured
> retention hours.
> {quote}
> What are impacts?
> {quote} # Log space keep in increasing and finally cause space shortage.
> # There are lots of log segment rolled with a small size. e.g log segment
> may be only 50mb, not the expected 1gb.
> # Kafka stream or client may experience missing data.{quote}
> How to reproduce it?
> #
> {quote}Configure message.timestamp.type as CreateTime.
> {quote}
> #
> {quote}Reset Kafka client's system clock time as a future time, e.g
> 03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
> Send message to Kafka.
> {quote}
> What's Next?
> #
> {quote}Go to check the timestamp in log time index and record (e.g
> 035957300794.log|timeindex). You should see all the value of the
> timestamp is messed up with future time after `03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
> {quote}
> #
> {quote}You may also see the new rolled log segment's size smaller than the
> configured log segment size).
>
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Description:
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.{quote}
How to reproduce it?
#
{quote}Configure message.timestamp.type as CreateTime.
{quote}
#
{quote}Reset Kafka client's system clock time a future time, e.g 03/04/*2025*,
3:25:52 PM [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
{quote}
What's Next?
#
{quote}Go to check the timestamp in log time index and record (e.g
035957300794.log|timeindex). You should see all the value of the
timestamp is messed up with future time after `03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
{quote}
#
{quote}You may also see the new rolled log segment's size smaller than the
configured log segment size).
{quote}
#
{quote}Log segment (035957300794.log|timeindex) and any of rolled log
segments will not be deleted.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.
{quote}
was:
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.{quote}
How to reproduce it?
#
{quote}Configure message.timestamp.type as CreateTime.
{quote}
#
{quote}Reset Kafka client's system clock time a future time, e.g 03/04/*2025*,
3:25:52 PM [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
{quote}
What's Next?
#
{quote}Go to check the timestamp in log time index and record (e.g
035957300794.log|timeindex). You should see all the value of the
timestamp is messed up with future time after `03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
{quote}
#
{quote}You may also see the new rolled log segment's size smaller than the
configured log segment size).
{quote}
#
{quote}Log segment (035957300794.log|timeindex) and any of rolled log
segments will not be deleted.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.
{quote}
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
> ---
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
> Issue Type: Bug
> Components: log, log cleaner, logging
>Affects Versions: 1.1.1
>Reporter: Jiangtao Liu
>Priority: Major
> Labels: storage
> Attachments: Screen Shot 2019-04-20 at 10.57.59 PM.png
>
>
> What's the issue?
> {quote}There were log segments, which can not be deleted over configured
> retention hours.
> {quote}
> What are impacts?
> {quote} # Log space keep in increasing and finally cause space shortage.
> # There are lots of log segment rolled with a small size. e.g log segment
> may be only 50mb, not the expected 1gb.
> # Kafka stream or client may experience missing data.{quote}
> How to reproduce it?
> #
> {quote}Configure message.timestamp.type as CreateTime.
> {quote}
> #
> {quote}Reset Kafka client's system clock time a future time, e.g
> 03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
> {quote}
> What's Next?
> #
> {quote}Go to check the timestamp in log time index and record (e.g
> 035957300794.log|timeindex). You should see all the value of the
> timestamp is messed up with future time after `03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
> {quote}
> #
> {quote}You may also see the new rolled log segment's size smaller than the
> configured log segment size).
> {quote}
> #
> {quote}Log segment
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Description:
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.{quote}
How to reproduce it?
#
{quote}Configure message.timestamp.type as CreateTime.
{quote}
#
{quote}Reset Kafka client's system clock time a future time, e.g 03/04/*2025*,
3:25:52 PM [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
{quote}
What's Next?
#
{quote}Go to check the timestamp in log time index and record (e.g
035957300794.log|timeindex). You should see all the value of the
timestamp is messed up with future time after `03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
{quote}
#
{quote}You may also see the new rolled log segment's size smaller than the
configured log segment size).
{quote}
#
{quote}Log segment (035957300794.log|timeindex) and any of rolled log
segments will not be deleted.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.
{quote}
was:
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.{quote}
How to reproduce it?
#
{quote}Configure message.timestamp.type as CreateTime.
{quote}
#
{quote}Reset Kafka client's system clock time a future time, e.g 03/04/*2025*,
3:25:52 PM [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
{quote}
What's Next?
#
{quote}Go to check the timestamp in log time index and record (e.g
035957300794.log|timeindex). You should see all the value of the
timestamp is messed up with future time after `03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
{quote}
#
{quote}You may also see the new rolled log segment's size smaller than the
configured log segment size).
{quote}
#
{quote}Log segment (035957300794.log|timeindex) and any of rolled log
segments will not be deleted.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.
{quote}
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
> ---
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
> Issue Type: Bug
> Components: log, log cleaner, logging
>Affects Versions: 1.1.1
>Reporter: Jiangtao Liu
>Priority: Major
> Labels: storage
> Attachments: Screen Shot 2019-04-20 at 10.57.59 PM.png
>
>
> What's the issue?
> {quote}There were log segments, which can not be deleted over configured
> retention hours.
> {quote}
> What are impacts?
> {quote} # Log space keep in increasing and finally cause space shortage.
> # There are lots of log segment rolled with a small size. e.g log segment
> may be only 50mb, not the expected 1gb.
> # Kafka stream or client may experience missing data.{quote}
> How to reproduce it?
> #
> {quote}Configure message.timestamp.type as CreateTime.
> {quote}
> #
> {quote}Reset Kafka client's system clock time a future time, e.g
> 03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
> {quote}
> What's Next?
> #
> {quote}Go to check the timestamp in log time index and record (e.g
> 035957300794.log|timeindex). You should see all the value of the
> timestamp is messed up with future time after `03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
> {quote}
> #
> {quote}You may also see the new rolled log segment's size smaller than the
> configured log segment size).
> {quote}
> #
> {quote}Log segment
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Description:
What's the issue?
{quote}There were log segments, which can not be deleted over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.{quote}
How to reproduce it?
#
{quote}Configure message.timestamp.type as CreateTime.
{quote}
#
{quote}Reset Kafka client's system clock time a future time, e.g 03/04/*2025*,
3:25:52 PM [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
{quote}
What's Next?
#
{quote}Go to check the timestamp in log time index and record (e.g
035957300794.log|timeindex). You should see all the value of the
timestamp is messed up with future time after `03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
{quote}
#
{quote}You may also see the new rolled log segment's size smaller than the
configured log segment size).
{quote}
#
{quote}Log segment (035957300794.log|timeindex) and any of rolled log
segments will not be deleted.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.
{quote}
was:
What's the issue?
{quote}There were log segments, which can not deleted after over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.{quote}
How to reproduce it?
#
{quote}Configure message.timestamp.type as CreateTime.
{quote}
#
{quote}Reset Kafka client's system clock time a future time, e.g 03/04/*2025*,
3:25:52 PM [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
{quote}
What's Next?
#
{quote}Go to check the timestamp in log time index and record (e.g
035957300794.log|timeindex). You should see all the value of the
timestamp is messed up with future time after `03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
{quote}
#
{quote}You may also see the new rolled log segment's size smaller than the
configured log segment size).
{quote}
#
{quote}Log segment (035957300794.log|timeindex) and any of rolled log
segments will not be deleted.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.
{quote}
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
> ---
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
> Issue Type: Bug
> Components: log, log cleaner, logging
>Affects Versions: 1.1.1
>Reporter: Jiangtao Liu
>Priority: Major
> Labels: storage
> Attachments: Screen Shot 2019-04-20 at 10.57.59 PM.png
>
>
> What's the issue?
> {quote}There were log segments, which can not be deleted over configured
> retention hours.
> {quote}
> What are impacts?
> {quote} # Log space keep in increasing and finally cause space shortage.
> # There are lots of log segment rolled with a small size. e.g log segment
> may be only 50mb, not the expected 1gb.
> # Kafka stream or client may experience missing data.{quote}
> How to reproduce it?
> #
> {quote}Configure message.timestamp.type as CreateTime.
> {quote}
> #
> {quote}Reset Kafka client's system clock time a future time, e.g
> 03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
> {quote}
> What's Next?
> #
> {quote}Go to check the timestamp in log time index and record (e.g
> 035957300794.log|timeindex). You should see all the value of the
> timestamp is messed up with future time after `03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
> {quote}
> #
> {quote}You may also see the new rolled log segment's size smaller than the
> configured log segment size).
> {quote}
> #
> {quote}Log segment
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Description:
What's the issue?
{quote}There were log segments, which can not deleted after over configured
retention hours.
{quote}
What are impacts?
{quote} # Log space keep in increasing and finally cause space shortage.
# There are lots of log segment rolled with a small size. e.g log segment may
be only 50mb, not the expected 1gb.
# Kafka stream or client may experience missing data.{quote}
How to reproduce it?
#
{quote}Configure message.timestamp.type as CreateTime.
{quote}
#
{quote}Reset Kafka client's system clock time a future time, e.g 03/04/*2025*,
3:25:52 PM [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
{quote}
What's Next?
#
{quote}Go to check the timestamp in log time index and record (e.g
035957300794.log|timeindex). You should see all the value of the
timestamp is messed up with future time after `03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
{quote}
#
{quote}You may also see the new rolled log segment's size smaller than the
configured log segment size).
{quote}
#
{quote}Log segment (035957300794.log|timeindex) and any of rolled log
segments will not be deleted.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.
{quote}
was:
What's the issue?
{quote}We have log retention with 12 hours. there are log segments not deleted
after 12 hours.
{quote}
How to reproduce it?
#
{quote}Configure message.timestamp.type as CreateTime.
{quote}
#
{quote}Reset Kafka client's system clock time a future time, e.g 03/04/*2025*,
3:25:52 PM [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
{quote}
What's Next?
#
{quote}Go to check the timestamp in log time index and record (e.g
035957300794.log|timeindex). You should see all the value of the
timestamp is messed up with future time after `03/04/*2025*, 3:25:52 PM
[GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
{quote}
#
{quote}You may also see the new rolled log segment's size smaller than the
configured log segment size).
{quote}
#
{quote}Log segment (035957300794.log|timeindex) and any of rolled log
segments will not be deleted.
{quote}
What's the particular logic to cause this issue?
#
{quote}private def deletableSegments(predicate: (LogSegment,
Option[LogSegment]) =>
Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
will always return empty deletable log segments.
{quote}
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
> ---
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
> Issue Type: Bug
> Components: log, log cleaner, logging
>Affects Versions: 1.1.1
>Reporter: Jiangtao Liu
>Priority: Major
> Labels: storage
> Attachments: Screen Shot 2019-04-20 at 10.57.59 PM.png
>
>
> What's the issue?
> {quote}There were log segments, which can not deleted after over configured
> retention hours.
> {quote}
> What are impacts?
> {quote} # Log space keep in increasing and finally cause space shortage.
> # There are lots of log segment rolled with a small size. e.g log segment
> may be only 50mb, not the expected 1gb.
> # Kafka stream or client may experience missing data.{quote}
> How to reproduce it?
> #
> {quote}Configure message.timestamp.type as CreateTime.
> {quote}
> #
> {quote}Reset Kafka client's system clock time a future time, e.g
> 03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
> {quote}
> What's Next?
> #
> {quote}Go to check the timestamp in log time index and record (e.g
> 035957300794.log|timeindex). You should see all the value of the
> timestamp is messed up with future time after `03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
> {quote}
> #
> {quote}You may also see the new rolled log segment's size smaller than the
> configured log segment size).
> {quote}
> #
> {quote}Log segment (035957300794.log|timeindex) and any of rolled log
> segments will not be deleted.
> {quote}
>
> What's the particular logic to cause this issue?
> #
> {quote}private def deletableSegments(predicate: (LogSegment,
> Option[LogSegment]) =>
>
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Component/s: logging
log cleaner
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
> ---
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
> Issue Type: Bug
> Components: log, log cleaner, logging
>Reporter: Jiangtao Liu
>Priority: Major
> Labels: storage
> Attachments: Screen Shot 2019-04-20 at 10.57.59 PM.png
>
>
> What's the issue?
> {quote}We have log retention with 12 hours. there are log segments not
> deleted after 12 hours.
> {quote}
> How to reproduce it?
> #
> {quote}Configure message.timestamp.type as CreateTime.
> {quote}
> #
> {quote}Reset Kafka client's system clock time a future time, e.g
> 03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
> {quote}
> What's Next?
> #
> {quote}Go to check the timestamp in log time index and record (e.g
> 035957300794.log|timeindex). You should see all the value of the
> timestamp is messed up with future time after `03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
> {quote}
> #
> {quote}You may also see the new rolled log segment's size smaller than the
> configured log segment size).
> {quote}
> #
> {quote}Log segment (035957300794.log|timeindex) and any of rolled log
> segments will not be deleted.
> {quote}
>
> What's the particular logic to cause this issue?
> #
> {quote}private def deletableSegments(predicate: (LogSegment,
> Option[LogSegment]) =>
> Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
> will always return empty deletable log segments.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Affects Version/s: 1.1.1
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
> ---
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
> Issue Type: Bug
> Components: log, log cleaner, logging
>Affects Versions: 1.1.1
>Reporter: Jiangtao Liu
>Priority: Major
> Labels: storage
> Attachments: Screen Shot 2019-04-20 at 10.57.59 PM.png
>
>
> What's the issue?
> {quote}We have log retention with 12 hours. there are log segments not
> deleted after 12 hours.
> {quote}
> How to reproduce it?
> #
> {quote}Configure message.timestamp.type as CreateTime.
> {quote}
> #
> {quote}Reset Kafka client's system clock time a future time, e.g
> 03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
> {quote}
> What's Next?
> #
> {quote}Go to check the timestamp in log time index and record (e.g
> 035957300794.log|timeindex). You should see all the value of the
> timestamp is messed up with future time after `03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
> {quote}
> #
> {quote}You may also see the new rolled log segment's size smaller than the
> configured log segment size).
> {quote}
> #
> {quote}Log segment (035957300794.log|timeindex) and any of rolled log
> segments will not be deleted.
> {quote}
>
> What's the particular logic to cause this issue?
> #
> {quote}private def deletableSegments(predicate: (LogSegment,
> Option[LogSegment]) =>
> Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
> will always return empty deletable log segments.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-8270) Kafka timestamp-based retention policy is not working when Kafka client's time is not reliable.
[
https://issues.apache.org/jira/browse/KAFKA-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangtao Liu updated KAFKA-8270:
Summary: Kafka timestamp-based retention policy is not working when Kafka
client's time is not reliable. (was: Kafka timestamp-based retention policy is
not working when Kafka client's timestamp is not reliable.)
> Kafka timestamp-based retention policy is not working when Kafka client's
> time is not reliable.
> ---
>
> Key: KAFKA-8270
> URL: https://issues.apache.org/jira/browse/KAFKA-8270
> Project: Kafka
> Issue Type: Bug
> Components: log
>Reporter: Jiangtao Liu
>Priority: Major
> Labels: storage
> Attachments: Screen Shot 2019-04-20 at 10.57.59 PM.png
>
>
> What's the issue?
> {quote}We have log retention with 12 hours. there are log segments not
> deleted after 12 hours.
> {quote}
> How to reproduce it?
> #
> {quote}Configure message.timestamp.type as CreateTime.
> {quote}
> #
> {quote}Reset Kafka client's system clock time a future time, e.g
> 03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]
> {quote}
> What's Next?
> #
> {quote}Go to check the timestamp in log time index and record (e.g
> 035957300794.log|timeindex). You should see all the value of the
> timestamp is messed up with future time after `03/04/*2025*, 3:25:52 PM
> [GMT-08:00|https://www.epochconverter.com/timezones?q=1741130752]`.
> {quote}
> #
> {quote}You may also see the new rolled log segment's size smaller than the
> configured log segment size).
> {quote}
> #
> {quote}Log segment (035957300794.log|timeindex) and any of rolled log
> segments will not be deleted.
> {quote}
>
> What's the particular logic to cause this issue?
> #
> {quote}private def deletableSegments(predicate: (LogSegment,
> Option[LogSegment]) =>
> Boolean)|[https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/log/Log.scala#L1227]]
> will always return empty deletable log segments.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
