[jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16027616#comment-16027616
 ] 

Jun Rao commented on KAFKA-4340:


[~edenhill], I wasn't suggesting that this commit is a breaking change. In 
fact, it gives the user the same current behavior if 
log.message.timestamp.difference.max.ms is explicitly set to log.retention.ms. 
I am just saying that the current behavior of rejecting the whole batch w/o 
telling the clients the violating messages can be improved, which may need a 
separate KIP. So, I am not sure if we need to revert this commit since it 
avoids unnecessary log rolling.

> Change the default value of log.message.timestamp.difference.max.ms to the 
> same as log.retention.ms
> ---
>
> Key: KAFKA-4340
> URL: https://issues.apache.org/jira/browse/KAFKA-4340
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.11.0.0
>
>
> [~junrao] brought up the following scenario: 
> If users are pumping data with timestamp already passed log.retention.ms into 
> Kafka, the messages will be appended to the log but will be immediately 
> rolled out by log retention thread when it kicks in and the messages will be 
> deleted. 
> To avoid this produce-and-deleted scenario, we can set the default value of 
> log.message.timestamp.difference.max.ms to be the same as log.retention.ms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

[Magnus Edenhill (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16027366#comment-16027366
 ] 

Magnus Edenhill commented on KAFKA-4340:


Since this is a change to the protocol API (change of behaviour), I suspect 
that a proper KIP is required and would recommend to revert this commit for the 
upcoming release.


> Change the default value of log.message.timestamp.difference.max.ms to the 
> same as log.retention.ms
> ---
>
> Key: KAFKA-4340
> URL: https://issues.apache.org/jira/browse/KAFKA-4340
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.11.0.0
>
>
> [~junrao] brought up the following scenario: 
> If users are pumping data with timestamp already passed log.retention.ms into 
> Kafka, the messages will be appended to the log but will be immediately 
> rolled out by log retention thread when it kicks in and the messages will be 
> deleted. 
> To avoid this produce-and-deleted scenario, we can set the default value of 
> log.message.timestamp.difference.max.ms to be the same as log.retention.ms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023862#comment-16023862
 ] 

Jun Rao commented on KAFKA-4340:


[~edenhill], as Jiangjie said, currently, the broker takes the whole batch as 
the unit. So, it's a bit weird for the broker to take only some of the messages 
in a batch. This is especially true with the EOS work in which the producer 
wants to commit multiple messages all or nothing. If the broker silently 
rejects some messages without telling the producer, the EOS guarantees will be 
broken.

Your point is valid though in that if the broker rejects the whole batch, 
currently the client doesn't know which messages fail the timestamp check. One 
way to address this is to include the array index of messages failing the 
timestamp check in the partition level produce response. Then, the client still 
fails all messages in the batch but with different error code in the callback, 
depending on whether the message fails the timestamp check or not. That way, 
the application can choose to resend failed messages with good timestamp, if 
desired. 

> Change the default value of log.message.timestamp.difference.max.ms to the 
> same as log.retention.ms
> ---
>
> Key: KAFKA-4340
> URL: https://issues.apache.org/jira/browse/KAFKA-4340
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.11.0.0
>
>
> [~junrao] brought up the following scenario: 
> If users are pumping data with timestamp already passed log.retention.ms into 
> Kafka, the messages will be appended to the log but will be immediately 
> rolled out by log retention thread when it kicks in and the messages will be 
> deleted. 
> To avoid this produce-and-deleted scenario, we can set the default value of 
> log.message.timestamp.difference.max.ms to be the same as log.retention.ms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

[Magnus Edenhill (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023668#comment-16023668
 ] 

Magnus Edenhill commented on KAFKA-4340:


Generally I would agree, but in this case I don't think there is a practical 
difference to discarding outdated messages, or removing them with the retention 
cleaner, it is just a matter of time frame - discarding and not appending 
outdated messages to the logs is immediate, while the retention cleaner might 
kick in immediately or in whatever max interval it is configured to.
So from the producer and consumer perspectives the end result is pretty much 
the same: there is no guarantee that an outdated message will be seen by a 
consumer.

However, rejecting the entire batch means there will be guaranteed data loss in 
the message stream: the producer will not try to re-send those failed messages, 
even if all but one message in the batch were actually okay. I strongly feel 
this is undesired behaviour from the application's point of view.


> Change the default value of log.message.timestamp.difference.max.ms to the 
> same as log.retention.ms
> ---
>
> Key: KAFKA-4340
> URL: https://issues.apache.org/jira/browse/KAFKA-4340
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.11.0.0
>
>
> [~junrao] brought up the following scenario: 
> If users are pumping data with timestamp already passed log.retention.ms into 
> Kafka, the messages will be appended to the log but will be immediately 
> rolled out by log retention thread when it kicks in and the messages will be 
> deleted. 
> To avoid this produce-and-deleted scenario, we can set the default value of 
> log.message.timestamp.difference.max.ms to be the same as log.retention.ms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023619#comment-16023619
 ] 

Jiangjie Qin commented on KAFKA-4340:
-

[~edenhill] I think one of the well established guarantee in producing is that 
either the entire batch succeeds or the entire batch fails. A lot of the code 
are actually built on top of this guarantee, for example the callback and 
offsets in the callback.

So if one of the messages in the batch had problem, it is by current design the 
entire batch will fail. I do agree that in this case the producer does not 
really have much to do, but that seems not a problem introduced by this patch. 

Silently discard the message on the broker side will introduce other problems 
because the producer acks are on a batch. The producer will assume the message 
has been successfully sent while it has been removed. And this also breaks the 
log retention enforcing order which always remove an old segment before 
deleting messages in newer segments. 

> Change the default value of log.message.timestamp.difference.max.ms to the 
> same as log.retention.ms
> ---
>
> Key: KAFKA-4340
> URL: https://issues.apache.org/jira/browse/KAFKA-4340
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.11.0.0
>
>
> [~junrao] brought up the following scenario: 
> If users are pumping data with timestamp already passed log.retention.ms into 
> Kafka, the messages will be appended to the log but will be immediately 
> rolled out by log retention thread when it kicks in and the messages will be 
> deleted. 
> To avoid this produce-and-deleted scenario, we can set the default value of 
> log.message.timestamp.difference.max.ms to be the same as log.retention.ms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

[Magnus Edenhill (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022945#comment-16022945
 ] 

Magnus Edenhill commented on KAFKA-4340:


While the idea behind this JIRA is good (as a means of optimization) I think it 
might be troublesome in practice.

If a producer sends a batch of N messages, with one message being too old, the 
entire batch will fail (errors are propagated per partition, not message) and 
the producer can't really recover and retry gracefully to produce the 
non-timedout messages.

This problem is not related to a specific client, but rather the nature of the 
data being produced:
it will manifest itself with old timestamps, such as app-sourced timestamps, or 
things like MM.

A better alternative would perhaps be to silently discard the message on the 
broker instead (which is effectively the same as writing the message to log and 
then immediately cleaning it before a consumer picks up the message).

> Change the default value of log.message.timestamp.difference.max.ms to the 
> same as log.retention.ms
> ---
>
> Key: KAFKA-4340
> URL: https://issues.apache.org/jira/browse/KAFKA-4340
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.11.0.0
>
>
> [~junrao] brought up the following scenario: 
> If users are pumping data with timestamp already passed log.retention.ms into 
> Kafka, the messages will be appended to the log but will be immediately 
> rolled out by log retention thread when it kicks in and the messages will be 
> deleted. 
> To avoid this produce-and-deleted scenario, we can set the default value of 
> log.message.timestamp.difference.max.ms to be the same as log.retention.ms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

[Colin P. McCabe (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876499#comment-15876499
 ] 

Colin P. McCabe commented on KAFKA-4340:


Thanks for the fix!

> Change the default value of log.message.timestamp.difference.max.ms to the 
> same as log.retention.ms
> ---
>
> Key: KAFKA-4340
> URL: https://issues.apache.org/jira/browse/KAFKA-4340
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.10.3.0
>
>
> [~junrao] brought up the following scenario: 
> If users are pumping data with timestamp already passed log.retention.ms into 
> Kafka, the messages will be appended to the log but will be immediately 
> rolled out by log retention thread when it kicks in and the messages will be 
> deleted. 
> To avoid this produce-and-deleted scenario, we can set the default value of 
> log.message.timestamp.difference.max.ms to be the same as log.retention.ms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868777#comment-15868777
 ] 

Jiangjie Qin commented on KAFKA-4340:
-

[~junrao] [~ijuma] [~cmccabe] I created 
https://github.com/apache/kafka/pull/2071 for this. I am run the branch builder 
now.

> Change the default value of log.message.timestamp.difference.max.ms to the 
> same as log.retention.ms
> ---
>
> Key: KAFKA-4340
> URL: https://issues.apache.org/jira/browse/KAFKA-4340
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.10.3.0
>
>
> [~junrao] brought up the following scenario: 
> If users are pumping data with timestamp already passed log.retention.ms into 
> Kafka, the messages will be appended to the log but will be immediately 
> rolled out by log retention thread when it kicks in and the messages will be 
> deleted. 
> To avoid this produce-and-deleted scenario, we can set the default value of 
> log.message.timestamp.difference.max.ms to be the same as log.retention.ms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868449#comment-15868449
 ] 

Ismael Juma commented on KAFKA-4340:


[~becket_qin], do you think you'll be able to look into this today? If not, we 
may need to temporarily revert the merged commit in order to get the system 
tests passing again.

> Change the default value of log.message.timestamp.difference.max.ms to the 
> same as log.retention.ms
> ---
>
> Key: KAFKA-4340
> URL: https://issues.apache.org/jira/browse/KAFKA-4340
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.10.3.0
>
>
> [~junrao] brought up the following scenario: 
> If users are pumping data with timestamp already passed log.retention.ms into 
> Kafka, the messages will be appended to the log but will be immediately 
> rolled out by log retention thread when it kicks in and the messages will be 
> deleted. 
> To avoid this produce-and-deleted scenario, we can set the default value of 
> log.message.timestamp.difference.max.ms to be the same as log.retention.ms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

[Colin P. McCabe (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868437#comment-15868437
 ] 

Colin P. McCabe commented on KAFKA-4340:


ClientCompatibilityTestNewBroker, TestUpgrade, TestSecurityRollingUpgrade, and 
MessageFormatChangeTest also fail after this change.

> Change the default value of log.message.timestamp.difference.max.ms to the 
> same as log.retention.ms
> ---
>
> Key: KAFKA-4340
> URL: https://issues.apache.org/jira/browse/KAFKA-4340
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.10.3.0
>
>
> [~junrao] brought up the following scenario: 
> If users are pumping data with timestamp already passed log.retention.ms into 
> Kafka, the messages will be appended to the log but will be immediately 
> rolled out by log retention thread when it kicks in and the messages will be 
> deleted. 
> To avoid this produce-and-deleted scenario, we can set the default value of 
> log.message.timestamp.difference.max.ms to be the same as log.retention.ms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868433#comment-15868433
 ] 

Jun Rao commented on KAFKA-4340:


[~becket_qin], after committing the patch in this jira. One of our system tests 
start to fail consistently: 
https://testbreak.confluent.io/kiosk/test_result?id=15685

In the failed tests, we use 0.9.0.1 producer to publish to trunk brokers and 
the producer keeps getting the following error.

{code}
[2017-02-15 10:10:17,275] INFO Kafka version : 0.9.0.1 
(org.apache.kafka.common.utils.AppInfoParser)
[2017-02-15 10:10:17,276] INFO Kafka commitId : 23c69d62a0cabf06 
(org.apache.kafka.common.utils.AppInfoParser)
[2017-02-15 10:10:17,467] WARN Unexpected error code: 32. 
(org.apache.kafka.common.protocol.Errors)
[2017-02-15 10:10:17,697] WARN Unexpected error code: 32. 
(org.apache.kafka.common.protocol.Errors)
[2017-02-15 10:10:17,730] WARN Unexpected error code: 32. 
(org.apache.kafka.common.protocol.Errors)
[2017-02-15 10:10:17,735] WARN Unexpected error code: 32. 
(org.apache.kafka.common.protocol.Errors)
[2017-02-15 10:10:17,739] WARN Unexpected error code: 32. 
(org.apache.kafka.common.protocol.Errors)
[2017-02-15 10:10:17,744] WARN Unexpected error code: 32. 
(org.apache.kafka.common.protocol.Errors)
[2017-02-15 10:10:18,108] WARN Unexpected error code: 32. 
(org.apache.kafka.common.protocol.Errors)
[2017-02-15 10:10:18,123] WARN Unexpected error code: 32. 
(org.apache.kafka.common.protocol.Errors)
[2017-02-15 10:10:18,126] WARN Unexpected error code: 32. 
(org.apache.kafka.common.protocol.Errors)
[2017-02-15 10:10:18,128] WARN Unexpected error code: 32. 
(org.apache.kafka.common.protocol.Errors)
[2017-02-15 10:10:18,130] WARN Unexpected error code: 32. 
(org.apache.kafka.common.protocol.Errors)
[2017-02-15 10:10:18,295] INFO Closing the Kafka producer with timeoutMillis = 
9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)
[2017-02-15 10:10:18,319] WARN Unexpected error code: 32. 
(org.apache.kafka.common.protocol.Errors)
[2017-02-15 10:10:18,325] WARN Unexpected error code: 32. 
(org.apache.kafka.common.protocol.Errors)
[2017-02-15 10:10:18,327] WARN Unexpected error code: 32. 
(org.apache.kafka.common.protocol.Errors)
[2017-02-15 10:10:18,328] WARN Unexpected error code: 32. 
(org.apache.kafka.common.protocol.Errors)
[2017-02-15 10:10:18,329] WARN Unexpected error code: 32. 
(org.apache.kafka.common.protocol.Errors)
{code}

It seems that the issue is that messages from the 0.9 producer will get the 
default -1 timestamp and will be rejected by the new default 
message.timestamp.difference.max.ms. For backward compatibility, perhaps we 
should not reject messages with -1 timestamp?

> Change the default value of log.message.timestamp.difference.max.ms to the 
> same as log.retention.ms
> ---
>
> Key: KAFKA-4340
> URL: https://issues.apache.org/jira/browse/KAFKA-4340
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.10.3.0
>
>
> [~junrao] brought up the following scenario: 
> If users are pumping data with timestamp already passed log.retention.ms into 
> Kafka, the messages will be appended to the log but will be immediately 
> rolled out by log retention thread when it kicks in and the messages will be 
> deleted. 
> To avoid this produce-and-deleted scenario, we can set the default value of 
> log.message.timestamp.difference.max.ms to be the same as log.retention.ms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862654#comment-15862654
 ] 

ASF GitHub Bot commented on KAFKA-4340:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2071


> Change the default value of log.message.timestamp.difference.max.ms to the 
> same as log.retention.ms
> ---
>
> Key: KAFKA-4340
> URL: https://issues.apache.org/jira/browse/KAFKA-4340
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.10.3.0
>
>
> [~junrao] brought up the following scenario: 
> If users are pumping data with timestamp already passed log.retention.ms into 
> Kafka, the messages will be appended to the log but will be immediately 
> rolled out by log retention thread when it kicks in and the messages will be 
> deleted. 
> To avoid this produce-and-deleted scenario, we can set the default value of 
> log.message.timestamp.difference.max.ms to be the same as log.retention.ms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610518#comment-15610518
 ] 

ASF GitHub Bot commented on KAFKA-4340:
---

GitHub user becketqin opened a pull request:

https://github.com/apache/kafka/pull/2071

KAFKA-4340: Change default message.timestamp.difference.max.ms to the same 
as log.retention.ms



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/becketqin/kafka KAFKA-4340

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2071.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2071


commit 5693cf603afdffe4d49659de2a1d9b0faa9f
Author: Jiangjie Qin 
Date:   2016-10-27T03:05:52Z

KAFKA-4340: Change default message.timestamp.difference.max.ms to the same 
as log.retention.ms




> Change the default value of log.message.timestamp.difference.max.ms to the 
> same as log.retention.ms
> ---
>
> Key: KAFKA-4340
> URL: https://issues.apache.org/jira/browse/KAFKA-4340
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.10.1.1
>
>
> [~junrao] brought up the following scenario: 
> If users are pumping data with timestamp already passed log.retention.ms into 
> Kafka, the messages will be appended to the log but will be immediately 
> rolled out by log retention thread when it kicks in and the messages will be 
> deleted. 
> To avoid this produce-and-deleted scenario, we can set the default value of 
> log.message.timestamp.difference.max.ms to be the same as log.retention.ms.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15607162#comment-15607162
 ] 

Jiangjie Qin commented on KAFKA-4340:
-

[~Kamal C] I have actually already started working on this. I am also making 
some other changes in this patch to make the future configuration validation 
easier. 

> Change the default value of log.message.timestamp.difference.max.ms to the 
> same as log.retention.ms
> ---
>
> Key: KAFKA-4340
> URL: https://issues.apache.org/jira/browse/KAFKA-4340
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.10.1.1
>
>
> [~junrao] brought up the following scenario: 
> If users are pumping data with timestamp already passed log.retention.ms into 
> Kafka, the messages will be appended to the log but will be immediately 
> rolled out by log retention thread when it kicks in and the messages will be 
> deleted. 
> To avoid this produce-and-deleted scenario, we can set the default value of 
> log.message.timestamp.difference.max.ms to be the same as log.retention.ms.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

[Kamal Chandraprakash (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15607070#comment-15607070
 ] 

Kamal Chandraprakash commented on KAFKA-4340:
-

Jiangjie Qin,

I would like to work on this issue as it's look like a newbie-one. Could you 
assign it to me?

> Change the default value of log.message.timestamp.difference.max.ms to the 
> same as log.retention.ms
> ---
>
> Key: KAFKA-4340
> URL: https://issues.apache.org/jira/browse/KAFKA-4340
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.10.1.1
>
>
> [~junrao] brought up the following scenario: 
> If users are pumping data with timestamp already passed log.retention.ms into 
> Kafka, the messages will be appended to the log but will be immediately 
> rolled out by log retention thread when it kicks in and the messages will be 
> deleted. 
> To avoid this produce-and-deleted scenario, we can set the default value of 
> log.message.timestamp.difference.max.ms to be the same as log.retention.ms.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)