[jira] [Commented] (BEAM-1514) change default timestamp in KafkaIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940760#comment-15940760
 ] 

ASF GitHub Bot commented on BEAM-1514:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2267


> change default timestamp in KafkaIO
> ---
>
> Key: BEAM-1514
> URL: https://issues.apache.org/jira/browse/BEAM-1514
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> When user use Kafka 0.10, the field 'timestamp' from Kafka should be used as 
> the default event timestamp.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1514) change default timestamp in KafkaIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15930645#comment-15930645
 ] 

ASF GitHub Bot commented on BEAM-1514:
--

Github user XuMingmin closed the pull request at:

https://github.com/apache/beam/pull/2135


> change default timestamp in KafkaIO
> ---
>
> Key: BEAM-1514
> URL: https://issues.apache.org/jira/browse/BEAM-1514
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> When user use Kafka 0.10, the field 'timestamp' from Kafka should be used as 
> the default event timestamp.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1514) change default timestamp in KafkaIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15889605#comment-15889605
 ] 

ASF GitHub Bot commented on BEAM-1514:
--

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/2135

BEAM-1514: change default timestamp in KafkaIO

I use Java reflection here to check if timestamp is available and get the 
value, based on the performance test as below. As the function is called for 
every record, it's worth to choose a faster option.
Direct: 1ns
SpEL: 15ns
Reflect: 5ns


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2135.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2135


commit 9b0505ca5f244e28f8c83ca3f1037ae45e516791
Author: mingmxu 
Date:   2017-02-22T00:40:16Z

junit test for both Kafka 0.9 and 0.10

commit 07e207d845943c485e6280e1705cc7c153a2ef35
Author: mingmxu 
Date:   2017-02-22T07:10:45Z

add doc for how to test, and how it works

commit c1d74af9f69cca6d5a7a79a67ef184a13ea6b2f1
Author: XuMingmin 
Date:   2017-02-22T17:58:29Z

Merge pull request #1 from apache/master

pull latest code

commit 71f83b7af6379155e30599f331ac39e68b046783
Author: mingmxu 
Date:   2017-02-22T18:01:29Z

JavaDoc style

commit 931027519be8f15999fef834609db6e2e63e087d
Author: mingmxu 
Date:   2017-02-23T01:14:55Z

revise to keep default kafka version as 0.9.0.1

commit d7093563067a31f268305bf001eb578e899745ec
Author: mingmxu 
Date:   2017-02-24T22:12:21Z

remove changes in TimestampTransform and WindowDoFnOperator

commit 280bc870568aff6825168a3faab3ed62870f6ee7
Author: mingmxu 
Date:   2017-02-24T23:09:12Z

revert TimestampTransform

commit 50d426be9b5cf7cc9d8c68a0a767a8cdcf8ab888
Author: mingmxu 
Date:   2017-02-24T23:10:19Z

revert TimestampTransform

commit d6ef256f7fee618512fccf622c0088d795a32bea
Author: XuMingmin 
Date:   2017-03-01T06:12:28Z

Merge pull request #2 from apache/master

pull latest code from apache/beam

commit ad4c208992f1c8612bd319d82f6a28754032c9ad
Author: mingmxu 
Date:   2017-03-01T06:14:59Z

use Java reflection for timestamp()

commit a95a9a74e08548977569f9b791b6d255468ef5f3
Author: XuMingmin 
Date:   2017-03-01T06:23:01Z

Merge pull request #3 from XuMingmin/BEAM-1514

merge Beam 1514 to master

commit b496ffcfe2aac851ed9c163747edad1a1677fed8
Author: XuMingmin 
Date:   2017-03-01T06:24:58Z

Update pom.xml

commit 220b5af8172bee1319394c2cb4168bc6fbbc059e
Author: XuMingmin 
Date:   2017-03-01T06:28:43Z

Update ConsumerSpEL.java




> change default timestamp in KafkaIO
> ---
>
> Key: BEAM-1514
> URL: https://issues.apache.org/jira/browse/BEAM-1514
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> When user use Kafka 0.10, the field 'timestamp' from Kafka should be used as 
> the default event timestamp.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1514) change default timestamp in KafkaIO

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877176#comment-15877176
 ] 

Xu Mingmin commented on BEAM-1514:
--

[~davor], can I have your help on how to setup the multiple version test? I'm 
not quire familiar with Beam's test settings.
Now I can pass junit tests locally, with 'clean verify 
-Dkafka.clients.version=0.10.1.1' and 'clean verify 
-Dkafka.clients.version=0.9.0.1'. Doubt do I need to change .travis.yml? 

> change default timestamp in KafkaIO
> ---
>
> Key: BEAM-1514
> URL: https://issues.apache.org/jira/browse/BEAM-1514
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> When user use Kafka 0.10, the field 'timestamp' from Kafka should be used as 
> the default event timestamp.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1514) change default timestamp in KafkaIO

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875035#comment-15875035
 ] 

Davor Bonaci commented on BEAM-1514:


SGTM!

> change default timestamp in KafkaIO
> ---
>
> Key: BEAM-1514
> URL: https://issues.apache.org/jira/browse/BEAM-1514
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> When user use Kafka 0.10, the field 'timestamp' from Kafka should be used as 
> the default event timestamp.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1514) change default timestamp in KafkaIO

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875027#comment-15875027
 ] 

Jean-Baptiste Onofré commented on BEAM-1514:


It makes sense to use the default record timestamp (it's what I did in JMS with 
the message timestamp). However, also be able for used to provide his own 
timestamp  fn is useful.

> change default timestamp in KafkaIO
> ---
>
> Key: BEAM-1514
> URL: https://issues.apache.org/jira/browse/BEAM-1514
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> When user use Kafka 0.10, the field 'timestamp' from Kafka should be used as 
> the default event timestamp.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1514) change default timestamp in KafkaIO

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875015#comment-15875015
 ] 

Xu Mingmin commented on BEAM-1514:
--

[~davor], will refer to the naming standard in Cloud PubsubIO.

the lines impacted would be here 
https://github.com/XuMingmin/beam/blob/6aca4d5238165ead825ec6c55202cebc091e900d/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L962-L963
 

curTimestamp = (source.spec.getTimestampFn() == null)
? Instant.now() : source.spec.getTimestampFn().apply(record);

With kafka client 0.10, ConsumerRecord.timestamp is there, instead of using 
Instant.now().

> change default timestamp in KafkaIO
> ---
>
> Key: BEAM-1514
> URL: https://issues.apache.org/jira/browse/BEAM-1514
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> When user use Kafka 0.10, the field 'timestamp' from Kafka should be used as 
> the default event timestamp.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1514) change default timestamp in KafkaIO

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874973#comment-15874973
 ] 

Davor Bonaci commented on BEAM-1514:


Sorry, [~mingmxu], I'm a little uninformed here. Is it a Kafka convention to 
use that specific name? (Also, we should aim for consistency with Cloud Pubsub, 
which has a similar thing.)

FYI [~rangadi].

> change default timestamp in KafkaIO
> ---
>
> Key: BEAM-1514
> URL: https://issues.apache.org/jira/browse/BEAM-1514
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> When user use Kafka 0.10, the field 'timestamp' from Kafka should be used as 
> the default event timestamp.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)