[jira] [Commented] (KAFKA-3025) KIP-31 (part 1): Add timestamp field to message, configs, and Producer/ConsumerRecord

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167792#comment-15167792
 ] 

Jiangjie Qin commented on KAFKA-3025:
-

Sure, will do it soon.

> KIP-31 (part 1): Add timestamp field to message, configs, and 
> Producer/ConsumerRecord
> 
>
> Key: KAFKA-3025
> URL: https://issues.apache.org/jira/browse/KAFKA-3025
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Anna Povzner
>Assignee: Jiangjie Qin
> Fix For: 0.10.0.0
>
>
> This JIRA is for changes for KIP-32 excluding broker checking and acting on 
> timestamp field in a message.
> This JIRA includes:
> 1. Bump up MessageAndOffset version to version 1 to:
> a. Change absolute offset to relative offset.
> b. Add time field to the message
> Timestamp => int64
> Timestamp is the number of milliseconds since Unix Epoch
> 2. Add time field to both ProducerRecord and Consumer Record
> If a user specifies the timestamp in a ProducerRecord, the ProducerRecord is 
> sent with this timestamp.
> If a user does not specify the timestamp in a ProducerRecord, the producer 
> stamps the ProducerRecord with current time.
> ConsumerRecord will have the timestamp of the message that were stored on 
> broker.
> 3. Add two new configurations to the broker. Configuration is per topic.
> * message.timestamp.type: type of a timestamp. Possible values: CreateTime, 
> LogAppendTime. Default: CreateTime
> * max.message.time.difference.ms: threshold for the acceptable time 
> difference between Timestamp in the message and local time on the broker. 
> Default: Long.MaxValue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3025) KIP-31 (part 1): Add timestamp field to message, configs, and Producer/ConsumerRecord

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167485#comment-15167485
 ] 

Jun Rao commented on KAFKA-3025:


[~becket_qin], another thing. Could you update 
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol 
with the new wire protocol changes?

> KIP-31 (part 1): Add timestamp field to message, configs, and 
> Producer/ConsumerRecord
> 
>
> Key: KAFKA-3025
> URL: https://issues.apache.org/jira/browse/KAFKA-3025
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Anna Povzner
>Assignee: Jiangjie Qin
> Fix For: 0.10.0.0
>
>
> This JIRA is for changes for KIP-32 excluding broker checking and acting on 
> timestamp field in a message.
> This JIRA includes:
> 1. Bump up MessageAndOffset version to version 1 to:
> a. Change absolute offset to relative offset.
> b. Add time field to the message
> Timestamp => int64
> Timestamp is the number of milliseconds since Unix Epoch
> 2. Add time field to both ProducerRecord and Consumer Record
> If a user specifies the timestamp in a ProducerRecord, the ProducerRecord is 
> sent with this timestamp.
> If a user does not specify the timestamp in a ProducerRecord, the producer 
> stamps the ProducerRecord with current time.
> ConsumerRecord will have the timestamp of the message that were stored on 
> broker.
> 3. Add two new configurations to the broker. Configuration is per topic.
> * message.timestamp.type: type of a timestamp. Possible values: CreateTime, 
> LogAppendTime. Default: CreateTime
> * max.message.time.difference.ms: threshold for the acceptable time 
> difference between Timestamp in the message and local time on the broker. 
> Default: Long.MaxValue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3025) KIP-31 (part 1): Add timestamp field to message, configs, and Producer/ConsumerRecord

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154695#comment-15154695
 ] 

Jiangjie Qin commented on KAFKA-3025:
-

[~junrao] Thanks a lot for the review. Sure, I will submit a follow up ticket 
for that.

> KIP-31 (part 1): Add timestamp field to message, configs, and 
> Producer/ConsumerRecord
> 
>
> Key: KAFKA-3025
> URL: https://issues.apache.org/jira/browse/KAFKA-3025
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Anna Povzner
>Assignee: Jiangjie Qin
> Fix For: 0.9.1.0
>
>
> This JIRA is for changes for KIP-32 excluding broker checking and acting on 
> timestamp field in a message.
> This JIRA includes:
> 1. Bump up MessageAndOffset version to version 1 to:
> a. Change absolute offset to relative offset.
> b. Add time field to the message
> Timestamp => int64
> Timestamp is the number of milliseconds since Unix Epoch
> 2. Add time field to both ProducerRecord and Consumer Record
> If a user specifies the timestamp in a ProducerRecord, the ProducerRecord is 
> sent with this timestamp.
> If a user does not specify the timestamp in a ProducerRecord, the producer 
> stamps the ProducerRecord with current time.
> ConsumerRecord will have the timestamp of the message that were stored on 
> broker.
> 3. Add two new configurations to the broker. Configuration is per topic.
> * message.timestamp.type: type of a timestamp. Possible values: CreateTime, 
> LogAppendTime. Default: CreateTime
> * max.message.time.difference.ms: threshold for the acceptable time 
> difference between Timestamp in the message and local time on the broker. 
> Default: Long.MaxValue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3025) KIP-31 (part 1): Add timestamp field to message, configs, and Producer/ConsumerRecord

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154642#comment-15154642
 ] 

Jun Rao commented on KAFKA-3025:


[~becket_qin], thanks again for the patch. Currently, in Log.append(), we 
always do a second shallow iteration after validateMessagesAndAssignOffsets() 
to check the size of the compressed message. An optimization that we can do is 
that if there is no re-compression, we can skip the second shallow iteration 
since the size of the compressed message won't change. Could you file a 
separate jira to track that? It's not super critical since the shallow 
iteration is cheap.

> KIP-31 (part 1): Add timestamp field to message, configs, and 
> Producer/ConsumerRecord
> 
>
> Key: KAFKA-3025
> URL: https://issues.apache.org/jira/browse/KAFKA-3025
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Anna Povzner
>Assignee: Jiangjie Qin
> Fix For: 0.9.1.0
>
>
> This JIRA is for changes for KIP-32 excluding broker checking and acting on 
> timestamp field in a message.
> This JIRA includes:
> 1. Bump up MessageAndOffset version to version 1 to:
> a. Change absolute offset to relative offset.
> b. Add time field to the message
> Timestamp => int64
> Timestamp is the number of milliseconds since Unix Epoch
> 2. Add time field to both ProducerRecord and Consumer Record
> If a user specifies the timestamp in a ProducerRecord, the ProducerRecord is 
> sent with this timestamp.
> If a user does not specify the timestamp in a ProducerRecord, the producer 
> stamps the ProducerRecord with current time.
> ConsumerRecord will have the timestamp of the message that were stored on 
> broker.
> 3. Add two new configurations to the broker. Configuration is per topic.
> * message.timestamp.type: type of a timestamp. Possible values: CreateTime, 
> LogAppendTime. Default: CreateTime
> * max.message.time.difference.ms: threshold for the acceptable time 
> difference between Timestamp in the message and local time on the broker. 
> Default: Long.MaxValue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3025) KIP-31 (part 1): Add timestamp field to message, configs, and Producer/ConsumerRecord

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154391#comment-15154391
 ] 

ASF GitHub Bot commented on KAFKA-3025:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/764


> KIP-31 (part 1): Add timestamp field to message, configs, and 
> Producer/ConsumerRecord
> 
>
> Key: KAFKA-3025
> URL: https://issues.apache.org/jira/browse/KAFKA-3025
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Anna Povzner
>Assignee: Jiangjie Qin
> Fix For: 0.9.1.0
>
>
> This JIRA is for changes for KIP-32 excluding broker checking and acting on 
> timestamp field in a message.
> This JIRA includes:
> 1. Bump up MessageAndOffset version to version 1 to:
> a. Change absolute offset to relative offset.
> b. Add time field to the message
> Timestamp => int64
> Timestamp is the number of milliseconds since Unix Epoch
> 2. Add time field to both ProducerRecord and Consumer Record
> If a user specifies the timestamp in a ProducerRecord, the ProducerRecord is 
> sent with this timestamp.
> If a user does not specify the timestamp in a ProducerRecord, the producer 
> stamps the ProducerRecord with current time.
> ConsumerRecord will have the timestamp of the message that were stored on 
> broker.
> 3. Add two new configurations to the broker. Configuration is per topic.
> * message.timestamp.type: type of a timestamp. Possible values: CreateTime, 
> LogAppendTime. Default: CreateTime
> * max.message.time.difference.ms: threshold for the acceptable time 
> difference between Timestamp in the message and local time on the broker. 
> Default: Long.MaxValue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3025) KIP-31 (part 1): Add timestamp field to message, configs, and Producer/ConsumerRecord

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096224#comment-15096224
 ] 

ASF GitHub Bot commented on KAFKA-3025:
---

GitHub user becketqin opened a pull request:

https://github.com/apache/kafka/pull/764

KAFKA-3025 Added timetamp to Message and use relative offset.

See KIP-31 and KIP-32 for details.

A few notes on the patch:
1. This patch only implements the server side message format change. It 
also implements the message format conversion.
2. All unit tests passed when set . Some of the tests require specific 
message format versions. I will fix them once the change on o.a.k.clients are 
done.
3. The unit tests were run with new and old message format.
4. When message format conversion occurs during consumption, the consumer 
will not be able to detect the message size too large situation. I did not try 
to fix this because the situation seems rare and only happen during migration 
phase.

The remaining work for KIP-31 and KIP-32 is
1. Add timestamp and relative offset to o.a.k.clients. 
2. Add more unit test and integration test


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/becketqin/kafka KAFKA-3025

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/764.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #764


commit 5604b4a57fff61188c562cc634daca4dee72aa4f
Author: Jiangjie Qin 
Date:   2016-01-07T14:30:39Z

Added timetamp to Message and use relative offset.
See KIP-31 and KIP-32 for details.




> KIP-31 (part 1): Add timestamp field to message, configs, and 
> Producer/ConsumerRecord
> 
>
> Key: KAFKA-3025
> URL: https://issues.apache.org/jira/browse/KAFKA-3025
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Anna Povzner
>Assignee: Jiangjie Qin
>
> This JIRA is for changes for KIP-32 excluding broker checking and acting on 
> timestamp field in a message.
> This JIRA includes:
> 1. Bump up MessageAndOffset version to version 1 to:
> a. Change absolute offset to relative offset.
> b. Add time field to the message
> Timestamp => int64
> Timestamp is the number of milliseconds since Unix Epoch
> 2. Add time field to both ProducerRecord and Consumer Record
> If a user specifies the timestamp in a ProducerRecord, the ProducerRecord is 
> sent with this timestamp.
> If a user does not specify the timestamp in a ProducerRecord, the producer 
> stamps the ProducerRecord with current time.
> ConsumerRecord will have the timestamp of the message that were stored on 
> broker.
> 3. Add two new configurations to the broker. Configuration is per topic.
> * message.timestamp.type: type of a timestamp. Possible values: CreateTime, 
> LogAppendTime. Default: CreateTime
> * max.message.time.difference.ms: threshold for the acceptable time 
> difference between Timestamp in the message and local time on the broker. 
> Default: Long.MaxValue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)