[
https://issues.apache.org/jira/browse/KAFKA-14991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mehari Beyene reassigned KAFKA-14991:
-------------------------------------
Assignee: Mehari Beyene
> Improving Producer's record timestamp validation
> ------------------------------------------------
>
> Key: KAFKA-14991
> URL: https://issues.apache.org/jira/browse/KAFKA-14991
> Project: Kafka
> Issue Type: Improvement
> Components: core, log
> Reporter: Mehari Beyene
> Assignee: Mehari Beyene
> Priority: Minor
>
> When time-based retention is configured, the timestamp provided by the
> producer is used by default to determine the retention period of the log.
> Customers have the option of changing the configuration to use the broker's
> timestamp by overriding the configuration for "log.message.timestamp.type",
> but by default, the producer's timestamp is used. The producer's record
> timestamp can be in the past or future. Kafka determines the retention time
> of the log by comparing the broker's time with the record's time.
> Arguably, there can be use cases for a producer to send records with
> timestamps that are in the past (for example, for replaying old data), but it
> is inaccurate for records to have a timestamp that is far in the future
> compared to the broker's current time.
> There is a configurable property called "message.timestamp.difference.max.ms"
> that customers can use to control the allowed time difference between the
> broker's current time and the record timestamp. However, the validation from
> the Kafka code side can be improved by rejecting records with future
> timestamps from being written in the first place.
> Customers have run into this issue in the past where a producer is configured
> erroneously to set the record timestamp in nanoseconds instead of
> milliseconds, resulting in a record timestamp that is in the future, and the
> time-based retention policy did not kick in as expected.
> The improvement I am proposing is to add basic validation in
> org.apache.kafka.storage.internals.log.LogValidator to reject record
> timestamps that are in the future compared to the broker current timestamp
> after accounting for a sensible tolerance for potential clock skew.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
