[
https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023862#comment-16023862
]
Jun Rao commented on KAFKA-4340:
--------------------------------
[~edenhill], as Jiangjie said, currently, the broker takes the whole batch as
the unit. So, it's a bit weird for the broker to take only some of the messages
in a batch. This is especially true with the EOS work in which the producer
wants to commit multiple messages all or nothing. If the broker silently
rejects some messages without telling the producer, the EOS guarantees will be
broken.
Your point is valid though in that if the broker rejects the whole batch,
currently the client doesn't know which messages fail the timestamp check. One
way to address this is to include the array index of messages failing the
timestamp check in the partition level produce response. Then, the client still
fails all messages in the batch but with different error code in the callback,
depending on whether the message fails the timestamp check or not. That way,
the application can choose to resend failed messages with good timestamp, if
desired.
> Change the default value of log.message.timestamp.difference.max.ms to the
> same as log.retention.ms
> ---------------------------------------------------------------------------------------------------
>
> Key: KAFKA-4340
> URL: https://issues.apache.org/jira/browse/KAFKA-4340
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.10.1.0
> Reporter: Jiangjie Qin
> Assignee: Jiangjie Qin
> Fix For: 0.11.0.0
>
>
> [~junrao] brought up the following scenario:
> If users are pumping data with timestamp already passed log.retention.ms into
> Kafka, the messages will be appended to the log but will be immediately
> rolled out by log retention thread when it kicks in and the messages will be
> deleted.
> To avoid this produce-and-deleted scenario, we can set the default value of
> log.message.timestamp.difference.max.ms to be the same as log.retention.ms.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
