date:20180706

[jira] [Commented] (BEAM-1855) Support Splittable DoFn in Flink Streaming runner

2018-07-06 Thread JIRA



[ 
https://issues.apache.org/jira/browse/BEAM-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16535647#comment-16535647
 ] 

Ismaël Mejía commented on BEAM-1855:


Thanks [~zorro] for reporting. Do you want to do a PR to update the website on 
this? So far Flink support streaming but batch (bounded) support is a WIP in 
case you want to track the progress see BEAM-3648.

> Support Splittable DoFn in Flink Streaming runner
> -
>
> Key: BEAM-1855
> URL: https://issues.apache.org/jira/browse/BEAM-1855
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Eugene Kirpichov
>Assignee: Aljoscha Krettek
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #5521

2018-07-06 Thread Apache Jenkins Server

See 


Changes:

[lcwik] [BEAM-4654] Translate timers as PCollections within the Java SDK.

[ankurgoenka] [BEAM-4644] Use SLF4J in ExecutableStageDoFnOperator

[ccy] Revert #5878 to unbreak Python postcommit

[ankurgoenka] Pull the docker image before starting docker container.

--
[...truncated 1.06 MB...]
test_match_type_variables 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_key_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_value_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_composite_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_simple_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_checks_not_dict 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_value_type_must_be_valid_composite_param 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.GeneratorHintTestCase) 
... ok
test_generator_argument_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_generator_return_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.IterableHintTestCase) 
... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_tuple_compatibility 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_must_be_iterable 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_enforce_kv_type_constraint 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_proxy_to_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_constraint_compatibility 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_repr (apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_proxy_to_union 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_getitem_sequence_not_allowed 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_any_return_type_hint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_primitive_type_or_type_constraint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_single_return_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_no_kwargs_accepted 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_composite_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_simple_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_violation 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_invalid_elem_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_must_be_set 
(apache_beam.typehints.typehints_tes

[jira] [Updated] (BEAM-1855) Support Splittable DoFn in Flink Streaming runner

2018-07-06 Thread JIRA



 [ 
https://issues.apache.org/jira/browse/BEAM-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-1855:
---
Summary: Support Splittable DoFn in Flink Streaming runner  (was: Support 
Splittable DoFn in Flink runner)

> Support Splittable DoFn in Flink Streaming runner
> -
>
> Key: BEAM-1855
> URL: https://issues.apache.org/jira/browse/BEAM-1855
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Eugene Kirpichov
>Assignee: Aljoscha Krettek
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4740) org.apache.beam.sdk.testing.PAssertTest.intentionalFail : java.lang.AssertionError: Intentional fail expected:<0> but was:<1>

2018-07-06 Thread JIRA



 [ 
https://issues.apache.org/jira/browse/BEAM-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-4740:
--

Assignee: (was: Ismaël Mejía)

> org.apache.beam.sdk.testing.PAssertTest.intentionalFail : 
> java.lang.AssertionError: Intentional fail expected:<0> but was:<1>
> -
>
> Key: BEAM-4740
> URL: https://issues.apache.org/jira/browse/BEAM-4740
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>
> http://localhost:8080/job/guinea_job_sdk_java_core_test/22/
> java.lang.AssertionError: Intentional fail expected:<0> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at 
> org.apache.beam.sdk.testing.PAssertTest.intentionalFail(PAssertTest.java:150)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319)
>   at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
>   at 
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at com.sun.proxy.$Proxy1.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.internal.remote.internal.hub

[jira] [Assigned] (BEAM-4740) org.apache.beam.sdk.testing.PAssertTest.intentionalFail : java.lang.AssertionError: Intentional fail expected:<0> but was:<1>

2018-07-06 Thread JIRA



 [ 
https://issues.apache.org/jira/browse/BEAM-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-4740:
--

Assignee: Ismaël Mejía

> org.apache.beam.sdk.testing.PAssertTest.intentionalFail : 
> java.lang.AssertionError: Intentional fail expected:<0> but was:<1>
> -
>
> Key: BEAM-4740
> URL: https://issues.apache.org/jira/browse/BEAM-4740
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Ismaël Mejía
>Priority: Major
>
> http://localhost:8080/job/guinea_job_sdk_java_core_test/22/
> java.lang.AssertionError: Intentional fail expected:<0> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at 
> org.apache.beam.sdk.testing.PAssertTest.intentionalFail(PAssertTest.java:150)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319)
>   at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
>   at 
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at com.sun.proxy.$Proxy1.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.int

[jira] [Resolved] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

2018-07-06 Thread JIRA



 [ 
https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-3587.

   Resolution: Not A Problem
Fix Version/s: Not applicable

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
> Fix For: Not applicable
>
> Attachments: screen1.png, screen2.png
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example]";
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4733) Python portable runner to pass pipeline options to job service

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4733?focusedWorklogId=120196&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120196
 ]

ASF GitHub Bot logged work on BEAM-4733:


Author: ASF GitHub Bot
Created on: 07/Jul/18 05:59
Start Date: 07/Jul/18 05:59
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #5888: [BEAM-4733] Pass 
pipeline options from Python portable runner to job server.
URL: https://github.com/apache/beam/pull/5888#issuecomment-403191429
 
 
   I used `--streaming` as a non-exclusive example for pipeline options that 
the runner needs to understand (we discussed it a while ago and it was 
[implemented 
accordingly](https://github.com/apache/beam/blob/dc1ccfa91204f3c6b6873119656b5302e2be09a5/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobInvocation.java#L105))
   
   Please take a look at FlinkPipelineOptions - there are several other options 
like parallelism and checkpointing interval that a user needs to be able to set.
   
   I think a basic story will be better than no story. Since you have authored 
the approach that is adopted here, are there any issues following it until 
someone has time to come up with something better (I suspect that we have many 
more important things to tackle till then).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120196)
Time Spent: 50m  (was: 40m)

> Python portable runner to pass pipeline options to job service
> --
>
> Key: BEAM-4733
> URL: https://issues.apache.org/jira/browse/BEAM-4733
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink, sdk-py-harness
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> The portable runner client currently drops the pipeline options. In the 
> prototype, it was implemented here: 
> [https://github.com/bsidhom/beam/commit/ce8a79122b98a7cbcf6fea7db4a5fe31b6e8248a#diff-f2f54a3d6ae6ef6a22e5d52621a133ed]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #5895 from charlesccychen/revert-5878

2018-07-06 Thread ccy

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 9c654233f9127d2f2bf4ad27320caf1d0f2ef5d6
Merge: a0e0908 a638e1d
Author: Charles Chen 
AuthorDate: Fri Jul 6 22:44:33 2018 -0700

Merge pull request #5895 from charlesccychen/revert-5878

Revert #5878 to unbreak Python postcommit

 sdks/python/setup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[beam] branch master updated (a0e0908 -> 9c65423)

2018-07-06 Thread ccy

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a0e0908  [BEAM-4654] Treat timers as PCollections within proto 
representation in the Java SDK.
 add a638e1d  Revert #5878 to unbreak Python postcommit
 new 9c65423  Merge pull request #5895 from charlesccychen/revert-5878

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/setup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[jira] [Work logged] (BEAM-4653) Java SDK harness should support user timers

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4653?focusedWorklogId=120181&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120181
 ]

ASF GitHub Bot logged work on BEAM-4653:


Author: ASF GitHub Bot
Created on: 07/Jul/18 02:20
Start Date: 07/Jul/18 02:20
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5898: [BEAM-4653] Add 
support to the Java SDK harness to execute timers.
URL: https://github.com/apache/beam/pull/5898#issuecomment-403182123
 
 
   R: @bsidhom @axelmagn 
   CC: @charlesccychen @tweise 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120181)
Time Spent: 20m  (was: 10m)

> Java SDK harness should support user timers
> ---
>
> Key: BEAM-4653
> URL: https://issues.apache.org/jira/browse/BEAM-4653
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-harness
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Labels: portability
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Wire up the onTimer method in the Java SDK harness FnApiDoFnRunner connecting 
> it to the RemoteGrpcPort read/write that is responsible for 
> producing/consumer timers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4653) Java SDK harness should support user timers

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4653?focusedWorklogId=120180&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120180
 ]

ASF GitHub Bot logged work on BEAM-4653:


Author: ASF GitHub Bot
Created on: 07/Jul/18 02:20
Start Date: 07/Jul/18 02:20
Worklog Time Spent: 10m 
  Work Description: lukecwik opened a new pull request #5898: [BEAM-4653] 
Add support to the Java SDK harness to execute timers.
URL: https://github.com/apache/beam/pull/5898
 
 
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120180)
Time Spent: 10m
Remaining Estimate: 0h

> Java SDK harness should support user timers
> ---
>
> Key: BEAM-4653
> URL: https://issues.apache.org/jira/browse/BEAM-4653
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-harness
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Labels: portability
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Wire up the onTimer method in the Java SDK harness FnApiDoFnRunner connecting

[jira] [Work logged] (BEAM-4654) Update pipeline translation for timers inside Java SDK

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4654?focusedWorklogId=120179&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120179
 ]

ASF GitHub Bot logged work on BEAM-4654:


Author: ASF GitHub Bot
Created on: 07/Jul/18 02:15
Start Date: 07/Jul/18 02:15
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #5883: [BEAM-4654] Treat 
timers as PCollections within proto representation in the Java SDK.
URL: https://github.com/apache/beam/pull/5883
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
index 3da7871d6db..ee93a6fbc8f 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
@@ -23,9 +23,13 @@
 
 import com.google.common.base.Joiner;
 import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSortedSet;
+import com.google.common.collect.Iterables;
 import com.google.common.collect.Sets;
 import java.io.IOException;
+import java.util.Collection;
 import java.util.Collections;
+import java.util.Comparator;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
@@ -36,10 +40,12 @@
 import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
 import org.apache.beam.model.pipeline.v1.RunnerApi.StandardPTransforms;
 import 
org.apache.beam.model.pipeline.v1.RunnerApi.StandardPTransforms.SplittableParDoComponents;
+import 
org.apache.beam.runners.core.construction.ParDoTranslation.ParDoTranslator;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.runners.AppliedPTransform;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.util.common.ReflectHelpers.ObjectsClassComparator;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PInput;
 import org.apache.beam.sdk.values.POutput;
@@ -47,8 +53,8 @@
 import org.apache.beam.sdk.values.TupleTag;
 
 /**
- * Utilities for converting {@link PTransform PTransforms} to and from {@link 
RunnerApi Runner API
- * protocol buffers}.
+ * Utilities for converting {@link PTransform PTransforms} to {@link RunnerApi 
Runner API protocol
+ * buffers}.
  */
 public class PTransformTranslation {
 
@@ -87,31 +93,16 @@
   public static final String MULTIMAP_SIDE_INPUT =
   getUrn(RunnerApi.StandardSideInputTypes.Enum.MULTIMAP);
 
-  private static final Map, 
TransformPayloadTranslator>
-  KNOWN_PAYLOAD_TRANSLATORS = loadTransformPayloadTranslators();
+  private static final Collection> KNOWN_TRANSLATORS =
+  loadKnownTranslators();
 
-  private static Map, TransformPayloadTranslator>
-  loadTransformPayloadTranslators() {
-HashMap, TransformPayloadTranslator> 
translators = new HashMap<>();
-
-for (TransformPayloadTranslatorRegistrar registrar :
-ServiceLoader.load(TransformPayloadTranslatorRegistrar.class)) {
-
-  Map, TransformPayloadTranslator> 
newTranslators =
-  (Map) registrar.getTransformPayloadTranslators();
-
-  Set> alreadyRegistered =
-  Sets.intersection(translators.keySet(), newTranslators.keySet());
-
-  if (!alreadyRegistered.isEmpty()) {
-throw new IllegalArgumentException(
-String.format(
-"Classes already registered: %s", Joiner.on(", 
").join(alreadyRegistered)));
-  }
-
-  translators.putAll(newTranslators);
-}
-return ImmutableMap.copyOf(translators);
+  private static Collection> loadKnownTranslators() {
+return ImmutableSortedSet.>orderedBy(
+(Comparator) ObjectsClassComparator.INSTANCE)
+.add(new RawPTransformTranslator())
+.add(new KnownTransformPayloadTranslator())
+.add(ParDoTranslator.create())
+.build();
   }
 
   private PTransformTranslation() {}
@@ -126,62 +117,13 @@ private PTransformTranslation() {}
   List> subtransforms,
   SdkComponents components)
   throws IOException {
-// TODO include DisplayData https://issues.apache.org/jira/browse/BEAM-2645
-RunnerApi.PTransform.Builder transformBuilder = 
RunnerApi.PTransform.newBuilder();
-for (Map.Entry, PValue> taggedInput : 
appliedPTransform.getInputs().entrySet()) {
-  checkArgument(
-  taggedInput.getValue() instanceof PColle

[jira] [Resolved] (BEAM-4654) Update pipeline translation for timers inside Java SDK

2018-07-06 Thread Luke Cwik (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4654.
-
   Resolution: Fixed
Fix Version/s: 2.6.0

> Update pipeline translation for timers inside Java SDK
> --
>
> Key: BEAM-4654
> URL: https://issues.apache.org/jira/browse/BEAM-4654
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Labels: portability
> Fix For: 2.6.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Add the timer PCollection and treat timers as inputs/outputs of the ParDo 
> PTransform.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (8104095 -> a0e0908)

2018-07-06 Thread lcwik

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 8104095  Merge pull request #5897: Pull the docker image before 
starting docker container.
 add 9fa6043  [BEAM-4654] Translate timers as PCollections within the Java 
SDK.
 add a0e0908  [BEAM-4654] Treat timers as PCollections within proto 
representation in the Java SDK.

No new revisions were added by this update.

Summary of changes:
 .../core/construction/PTransformTranslation.java   | 327 ++---
 .../core/construction/ParDoTranslation.java|  96 --
 .../core/construction/ParDoTranslationTest.java|  40 ++-
 3 files changed, 334 insertions(+), 129 deletions(-)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #643

2018-07-06 Thread Apache Jenkins Server

See 


Changes:

[mariagh] Add query 1 & 2

[mariagh] Clean up

[mariagh] Cleaup

[mariagh] Fix linter errors

[mariagh] Address review commnets; Reuse topic for queries

[ankurgoenka] Fix artifact retrival file system

[matthias] Futurize io subpackage

[matthias] incorporated all feedback for futurize io subpackage

[amaliujia] remove decimal support in BigQuery SQL read.

[iemejia] [BEAM-4016] Invoke Setup and TearDown on SplitRestrictionFn and

[ankurgoenka] Initialization BeamFileSystem in FlinkExecutableStageFunction

--
[...truncated 17.57 MB...]
INFO: Adding 
View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly
 as step s7
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
 as step s8
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
View.AsSingleton/Combine.GloballyAsSingletonView/CreateDataflowView as step s9
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Create123/Read(CreateSource) as step s10
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding OutputSideInputs as step s11
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Window.Into()/Window.Assign as step 
s12
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) as step 
s13
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map 
as step s14
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
Jul 07, 2018 1:58:51 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
Jul 07, 2018 1:58:51 AM 
org.a

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1029

2018-07-06 Thread Apache Jenkins Server

See

[jira] [Work logged] (BEAM-4654) Update pipeline translation for timers inside Java SDK

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4654?focusedWorklogId=120171&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120171
 ]

ASF GitHub Bot logged work on BEAM-4654:


Author: ASF GitHub Bot
Created on: 07/Jul/18 00:39
Start Date: 07/Jul/18 00:39
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #5883: [BEAM-4654] Treat 
timers as PCollections within proto representation in the Java SDK.
URL: https://github.com/apache/beam/pull/5883#issuecomment-403176526
 
 
   Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120171)
Time Spent: 1h 10m  (was: 1h)

> Update pipeline translation for timers inside Java SDK
> --
>
> Key: BEAM-4654
> URL: https://issues.apache.org/jira/browse/BEAM-4654
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Labels: portability
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Add the timer PCollection and treat timers as inputs/outputs of the ParDo 
> PTransform.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (c114445 -> 8104095)

2018-07-06 Thread jkff

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from c114445  [BEAM-4644] Use SLF4J in ExecutableStageDoFnOperator
 add 316a996  Pull the docker image before starting docker container.
 new 8104095  Merge pull request #5897: Pull the docker image before 
starting docker container.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../org/apache/beam/runners/fnexecution/environment/DockerCommand.java | 3 +++
 1 file changed, 3 insertions(+)

[beam] 01/01: Merge pull request #5897: Pull the docker image before starting docker container.

2018-07-06 Thread jkff

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 8104095ddf974b9b39b7dde9fcbb99b86c225526
Merge: c114445 316a996
Author: Eugene Kirpichov 
AuthorDate: Fri Jul 6 17:38:10 2018 -0700

Merge pull request #5897: Pull the docker image before starting docker 
container.

Pull the docker image before starting docker container.

 .../org/apache/beam/runners/fnexecution/environment/DockerCommand.java | 3 +++
 1 file changed, 3 insertions(+)

Build failed in Jenkins: beam_PostCommit_Python_Verify #5520

2018-07-06 Thread Apache Jenkins Server

See 


Changes:

[mariagh] Add query 1 & 2

[mariagh] Clean up

[mariagh] Cleaup

[mariagh] Fix linter errors

[mariagh] Address review commnets; Reuse topic for queries

[ankurgoenka] Fix artifact retrival file system

[matthias] Futurize io subpackage

[matthias] incorporated all feedback for futurize io subpackage

[amaliujia] remove decimal support in BigQuery SQL read.

[iemejia] [BEAM-4016] Invoke Setup and TearDown on SplitRestrictionFn and

[ankurgoenka] Initialization BeamFileSystem in FlinkExecutableStageFunction

--
[...truncated 1.12 MB...]
test_match_type_variables 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_key_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_value_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_composite_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_simple_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_checks_not_dict 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_value_type_must_be_valid_composite_param 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.GeneratorHintTestCase) 
... ok
test_generator_argument_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_generator_return_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.IterableHintTestCase) 
... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_tuple_compatibility 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_must_be_iterable 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_enforce_kv_type_constraint 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_proxy_to_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_constraint_compatibility 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_repr (apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_proxy_to_union 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_getitem_sequence_not_allowed 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_any_return_type_hint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_primitive_type_or_type_constraint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_single_return_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_no_kwargs_accepted 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_composite_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_simple_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_violation 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.SetHintT

[jira] [Resolved] (BEAM-4644) ExecutableStageDoFnOperator.java uses JUL instead of SLF4J

2018-07-06 Thread Luke Cwik (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4644.
-
   Resolution: Fixed
Fix Version/s: 2.6.0

> ExecutableStageDoFnOperator.java uses JUL instead of SLF4J
> --
>
> Key: BEAM-4644
> URL: https://issues.apache.org/jira/browse/BEAM-4644
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Rafael Fernandez
>Assignee: Ankur Goenka
>Priority: Minor
>  Labels: easy-fix
> Fix For: 2.6.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Please see discussion on 
> [https://lists.apache.org/thread.html/ae293f61424589ade7f0a40665900e234dfca8926150f579790bac09@%3Cdev.beam.apache.org%3E]
>  for details.
>  
> If this is an intentional choice, please resolve and comment it on the code. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4644) ExecutableStageDoFnOperator.java uses JUL instead of SLF4J

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4644?focusedWorklogId=120170&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120170
 ]

ASF GitHub Bot logged work on BEAM-4644:


Author: ASF GitHub Bot
Created on: 07/Jul/18 00:22
Start Date: 07/Jul/18 00:22
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #5896: [BEAM-4644] Use 
SLF4J in ExecutableStageDoFnOperator
URL: https://github.com/apache/beam/pull/5896
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
index fbd10edae1e..b24c7aa593a 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
@@ -23,7 +23,6 @@
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
-import java.util.logging.Logger;
 import javax.annotation.concurrent.GuardedBy;
 import org.apache.beam.model.pipeline.v1.RunnerApi;
 import org.apache.beam.runners.core.DoFnRunner;
@@ -47,6 +46,8 @@
 import org.apache.beam.sdk.values.TupleTag;
 import org.apache.beam.sdk.values.WindowingStrategy;
 import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /**
  * ExecutableStageDoFnOperator basic functional implementation without side 
inputs and user state.
@@ -61,7 +62,7 @@
 public class ExecutableStageDoFnOperator extends 
DoFnOperator {
 
   private static final Logger logger =
-  Logger.getLogger(ExecutableStageDoFnOperator.class.getName());
+  LoggerFactory.getLogger(ExecutableStageDoFnOperator.class.getName());
 
   private final RunnerApi.ExecutableStagePayload payload;
   private final JobInfo jobInfo;
@@ -146,7 +147,7 @@ private void 
processElementWithSdkHarness(WindowedValue element) throws
 try (RemoteBundle bundle =
 stageBundleFactory.getBundle(
 new ReceiverFactory(outputManager, outputMap), 
stateRequestHandler, progressHandler)) {
-  logger.finer(String.format("Sending value: %s", element));
+  logger.debug(String.format("Sending value: %s", element));
   bundle.getInputReceiver().accept(element);
 }
   }


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120170)
Time Spent: 20m  (was: 10m)

> ExecutableStageDoFnOperator.java uses JUL instead of SLF4J
> --
>
> Key: BEAM-4644
> URL: https://issues.apache.org/jira/browse/BEAM-4644
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Rafael Fernandez
>Assignee: Ankur Goenka
>Priority: Minor
>  Labels: easy-fix
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Please see discussion on 
> [https://lists.apache.org/thread.html/ae293f61424589ade7f0a40665900e234dfca8926150f579790bac09@%3Cdev.beam.apache.org%3E]
>  for details.
>  
> If this is an intentional choice, please resolve and comment it on the code. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: [BEAM-4644] Use SLF4J in ExecutableStageDoFnOperator

2018-07-06 Thread lcwik

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit c1144459d54fd2865cb0b481e14a3651bcef5c6f
Merge: b439ebe 703d136
Author: Lukasz Cwik 
AuthorDate: Fri Jul 6 17:22:29 2018 -0700

[BEAM-4644] Use SLF4J in ExecutableStageDoFnOperator

 .../wrappers/streaming/ExecutableStageDoFnOperator.java| 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)

[beam] branch master updated (b439ebe -> c114445)

2018-07-06 Thread lcwik

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b439ebe  Merge pull request #5874: Fix artifact retrival file system
 add 703d136  [BEAM-4644] Use SLF4J in ExecutableStageDoFnOperator
 new c114445  [BEAM-4644] Use SLF4J in ExecutableStageDoFnOperator

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../wrappers/streaming/ExecutableStageDoFnOperator.java| 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)

[jira] [Work logged] (BEAM-4654) Update pipeline translation for timers inside Java SDK

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4654?focusedWorklogId=120162&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120162
 ]

ASF GitHub Bot logged work on BEAM-4654:


Author: ASF GitHub Bot
Created on: 07/Jul/18 00:19
Start Date: 07/Jul/18 00:19
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5883: 
[BEAM-4654] Treat timers as PCollections within proto representation in the 
Java SDK.
URL: https://github.com/apache/beam/pull/5883#discussion_r200795883
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
 ##
 @@ -88,39 +91,72 @@
   "urn:beam:windowmappingfn:javasdk:0.1";
 
   /** A {@link TransformPayloadTranslator} for {@link ParDo}. */
-  public static class ParDoPayloadTranslator
-  implements TransformPayloadTranslator> {
-public static TransformPayloadTranslator create() {
-  return new ParDoPayloadTranslator();
+  public static class ParDoTranslator implements 
TransformTranslator> {
+
+public static TransformTranslator create() {
+  return new ParDoTranslator();
 }
 
-private ParDoPayloadTranslator() {}
+private ParDoTranslator() {}
 
 @Override
 public String getUrn(ParDo.MultiOutput transform) {
   return PAR_DO_TRANSFORM_URN;
 }
 
 @Override
-public FunctionSpec translate(
-AppliedPTransform> transform, SdkComponents 
components)
-throws IOException {
-  ParDoPayload payload =
-  translateParDo(transform.getTransform(), transform.getPipeline(), 
components);
-  return RunnerApi.FunctionSpec.newBuilder()
-  .setUrn(PAR_DO_TRANSFORM_URN)
-  .setPayload(payload.toByteString())
-  .build();
+public boolean canTranslate(PTransform pTransform) {
+  return pTransform instanceof ParDo.MultiOutput;
 }
 
-/** Registers {@link ParDoPayloadTranslator}. */
-@AutoService(TransformPayloadTranslatorRegistrar.class)
-public static class Registrar implements 
TransformPayloadTranslatorRegistrar {
-  @Override
-  public Map, ? extends 
TransformPayloadTranslator>
-  getTransformPayloadTranslators() {
-return Collections.singletonMap(ParDo.MultiOutput.class, new 
ParDoPayloadTranslator());
+@Override
+public RunnerApi.PTransform translate(
+AppliedPTransform appliedPTransform,
+List> subtransforms,
+SdkComponents components)
+throws IOException {
+  RunnerApi.PTransform.Builder builder =
+  PTransformTranslation.translateAppliedPTransform(
+  appliedPTransform, subtransforms, components);
+
+  ParDoPayload payload =
+  translateParDo(
+  (ParDo.MultiOutput) appliedPTransform.getTransform(),
+  appliedPTransform.getPipeline(),
+  components);
+  builder.setSpec(
+  RunnerApi.FunctionSpec.newBuilder()
+  .setUrn(PAR_DO_TRANSFORM_URN)
+  .setPayload(payload.toByteString())
+  .build());
+
+  String mainInputId = getMainInputId(builder, payload);
+  PCollection> mainInput =
+  (PCollection) appliedPTransform.getInputs().get(new 
TupleTag(mainInputId));
+
+  // https://s.apache.org/beam-portability-timers
+  // Add a PCollection and coder for each timer. Also treat them as inputs 
and outputs.
+  for (String localTimerName : payload.getTimerSpecsMap().keySet()) {
+PCollection timerPCollection =
 
 Review comment:
   The PCollection has a timer coder on it as defined in 
https://github.com/apache/beam/pull/5789 and an example implementation in 
https://github.com/apache/beam/pull/5794
   For all our current timer PCollections, it is always KV>. 
Java doesn't expose the payload so it is always `null`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120162)
Time Spent: 1h  (was: 50m)

> Update pipeline translation for timers inside Java SDK
> --
>
> Key: BEAM-4654
> URL: https://issues.apache.org/jira/browse/BEAM-4654
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Labels: portability
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Add the timer PCollection and treat timers as inputs/outputs of the ParDo 
> PTransform.

[beam] 01/01: Merge pull request #5874: Fix artifact retrival file system

2018-07-06 Thread jkff

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b439ebe92b6033aa0d93ed0fbd67facd88401909
Merge: 92275d6 cff51fb
Author: Eugene Kirpichov 
AuthorDate: Fri Jul 6 16:47:52 2018 -0700

Merge pull request #5874: Fix artifact retrival file system

Fix artifact retrival file system

 .../java/org/apache/beam/runners/flink/FlinkJobServerDriver.java| 6 ++
 .../flink/translation/functions/FlinkExecutableStageFunction.java   | 5 +
 2 files changed, 7 insertions(+), 4 deletions(-)

[beam] branch master updated (92275d6 -> b439ebe)

2018-07-06 Thread jkff

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 92275d6  Merge pull request #5837 from mariapython/query1_2_pr
 add 6bcaeaa  Fix artifact retrival file system
 add cff51fb  Initialization BeamFileSystem in FlinkExecutableStageFunction
 new b439ebe  Merge pull request #5874: Fix artifact retrival file system

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../java/org/apache/beam/runners/flink/FlinkJobServerDriver.java| 6 ++
 .../flink/translation/functions/FlinkExecutableStageFunction.java   | 5 +
 2 files changed, 7 insertions(+), 4 deletions(-)

[jira] [Work logged] (BEAM-4644) ExecutableStageDoFnOperator.java uses JUL instead of SLF4J

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4644?focusedWorklogId=120146&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120146
 ]

ASF GitHub Bot logged work on BEAM-4644:


Author: ASF GitHub Bot
Created on: 06/Jul/18 23:45
Start Date: 06/Jul/18 23:45
Worklog Time Spent: 10m 
  Work Description: angoenka opened a new pull request #5896: [BEAM-4644] 
Use SLF4J in ExecutableStageDoFnOperator
URL: https://github.com/apache/beam/pull/5896
 
 
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120146)
Time Spent: 10m
Remaining Estimate: 0h

> ExecutableStageDoFnOperator.java uses JUL instead of SLF4J
> --
>
> Key: BEAM-4644
> URL: https://issues.apache.org/jira/browse/BEAM-4644
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Rafael Fernandez
>Assignee: Ankur Goenka
>Priority: Minor
>  Labels: easy-fix
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Please see discussion on 
> [https://lists.apache.org/thread.html

[jira] [Assigned] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

2018-07-06 Thread Ankur Goenka (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankur Goenka reassigned BEAM-4130:
--

Assignee: Ankur Goenka

> Portable Flink runner JobService entry point in a Docker container
> --
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Minor
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4740) org.apache.beam.sdk.testing.PAssertTest.intentionalFail : java.lang.AssertionError: Intentional fail expected:<0> but was:<1>

2018-07-06 Thread Mikhail Gryzykhin (JIRA)

Mikhail Gryzykhin created BEAM-4740:
---

 Summary: org.apache.beam.sdk.testing.PAssertTest.intentionalFail : 
java.lang.AssertionError: Intentional fail expected:<0> but was:<1>
 Key: BEAM-4740
 URL: https://issues.apache.org/jira/browse/BEAM-4740
 Project: Beam
  Issue Type: Bug
  Components: test-failures
Reporter: Mikhail Gryzykhin


http://localhost:8080/job/guinea_job_sdk_java_core_test/22/
java.lang.AssertionError: Intentional fail expected:<0> but was:<1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at 
org.apache.beam.sdk.testing.PAssertTest.intentionalFail(PAssertTest.java:150)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at 
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at 
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy1.processTestClass(Unknown Source)
at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:155)
at 
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:137)
at 
org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
at 
org.

[jira] [Resolved] (BEAM-4023) Log warning for missing worker id in FnApiControlClientPoolService

2018-07-06 Thread Ankur Goenka (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankur Goenka resolved BEAM-4023.

   Resolution: Fixed
Fix Version/s: Not applicable

> Log warning for missing worker id in FnApiControlClientPoolService
> --
>
> Key: BEAM-4023
> URL: https://issues.apache.org/jira/browse/BEAM-4023
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
> Fix For: Not applicable
>
>
> We should log warning for missing worker id when connecting the GRPC channel.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3418) Python Fnapi - Support Multiple SDK workers on a single VM

2018-07-06 Thread Ankur Goenka (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-3418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankur Goenka resolved BEAM-3418.

   Resolution: Fixed
Fix Version/s: 2.6.0

> Python Fnapi - Support Multiple SDK workers on a single VM
> --
>
> Key: BEAM-3418
> URL: https://issues.apache.org/jira/browse/BEAM-3418
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>  Labels: performance, portability
> Fix For: 2.6.0
>
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> Support multiple python SDK process on a VM to fully utilize a machine.
> Each SDK Process will work in isolation and interact with Runner Harness 
> independently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2855) Implement a python version of the nexmark queries

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-2855?focusedWorklogId=120107&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120107
 ]

ASF GitHub Bot logged work on BEAM-2855:


Author: ASF GitHub Bot
Created on: 06/Jul/18 22:32
Start Date: 06/Jul/18 22:32
Worklog Time Spent: 10m 
  Work Description: charlesccychen closed pull request #5837: [BEAM-2855] 
Add queries 1 and 2 to the Python Nexmark suite
URL: https://github.com/apache/beam/pull/5837
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/python/apache_beam/testing/benchmarks/nexmark/nexmark_launcher.py 
b/sdks/python/apache_beam/testing/benchmarks/nexmark/nexmark_launcher.py
index 3616f35c798..2779cb4fa5b 100644
--- a/sdks/python/apache_beam/testing/benchmarks/nexmark/nexmark_launcher.py
+++ b/sdks/python/apache_beam/testing/benchmarks/nexmark/nexmark_launcher.py
@@ -74,6 +74,8 @@
 from apache_beam.options.pipeline_options import TestOptions
 from apache_beam.testing.benchmarks.nexmark.nexmark_util import Command
 from apache_beam.testing.benchmarks.nexmark.queries import query0
+from apache_beam.testing.benchmarks.nexmark.queries import query1
+from apache_beam.testing.benchmarks.nexmark.queries import query2
 
 
 class NexmarkLauncher(object):
@@ -82,6 +84,19 @@ def __init__(self):
 self.uuid = str(uuid.uuid4())
 self.topic_name = self.args.topic_name + self.uuid
 self.subscription_name = self.args.subscription_name + self.uuid
+publish_client = pubsub.Client(project=self.project)
+topic = publish_client.topic(self.topic_name)
+if topic.exists():
+  logging.info('deleting topic %s', self.topic_name)
+  topic.delete()
+logging.info('creating topic %s', self.topic_name)
+topic.create()
+sub = topic.subscription(self.subscription_name)
+if sub.exists():
+  logging.info('deleting sub %s', self.topic_name)
+  sub.delete()
+logging.info('creating sub %s', self.topic_name)
+sub.create()
 
   def parse_args(self):
 parser = argparse.ArgumentParser()
@@ -148,13 +163,7 @@ def parse_args(self):
   def generate_events(self):
 publish_client = pubsub.Client(project=self.project)
 topic = publish_client.topic(self.topic_name)
-if topic.exists():
-  topic.delete()
-topic.create()
 sub = topic.subscription(self.subscription_name)
-if sub.exists():
-  sub.delete()
-sub.create()
 
 logging.info('Generating auction events to topic %s', topic.name)
 
@@ -181,11 +190,12 @@ def generate_events(self):
 
 return raw_events
 
-  def run_query(self, query, query_errors):
+  def run_query(self, query, query_args, query_errors):
 try:
+  self.parse_args()
   self.pipeline = beam.Pipeline(options=self.pipeline_options)
   raw_events = self.generate_events()
-  query.load(raw_events)
+  query.load(raw_events, query_args)
   result = self.pipeline.run()
   job_duration = (
   self.pipeline_options.view_as(TestOptions).wait_until_finish_duration
@@ -211,9 +221,18 @@ def cleanup(self):
   def run(self):
 queries = {
 0: query0,
+1: query1,
+2: query2,
 # TODO(mariagh): Add more queries.
 }
 
+# TODO(mariagh): Move to a config file.
+query_args = {
+2: {
+'auction_id': 'a1003'
+}
+}
+
 query_errors = []
 for i in self.args.query:
   self.parse_args()
@@ -224,13 +243,15 @@ def run(self):
   launch_from_direct_runner = self.pipeline_options.view_as(
   StandardOptions).runner in [None, 'DirectRunner']
 
+  query_duration = 
self.pipeline_options.view_as(TestOptions).wait_until_finish_duration # pylint: 
disable=line-too-long
   if launch_from_direct_runner:
-command = Command(self.run_query, args=[queries[i], query_errors])
-query_duration = 
self.pipeline_options.view_as(TestOptions).wait_until_finish_duration # pylint: 
disable=line-too-long
+command = Command(self.run_query, args=[queries[i],
+query_args.get(i),
+query_errors])
 command.run(timeout=query_duration // 1000)
   else:
 try:
-  self.run_query(queries[i], query_errors=None)
+  self.run_query(queries[i], query_args.get(i), query_errors=None)
 except Exception as exc:
   query_errors.append(exc)
 
diff --git a/sdks/python/apache_beam/testing/benchmarks/nexmark/nexmark_util.py 
b/sdks/python/apache_beam/testing/benchmarks/nexmark/nexmark_util.py
index dd8c142f97f..688d60272cb 100644
--- a/sdks/python/apache_beam/testing/benchmarks

[jira] [Work logged] (BEAM-2855) Implement a python version of the nexmark queries

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-2855?focusedWorklogId=120106&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120106
 ]

ASF GitHub Bot logged work on BEAM-2855:


Author: ASF GitHub Bot
Created on: 06/Jul/18 22:32
Start Date: 06/Jul/18 22:32
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #5837: [BEAM-2855] 
Add queries 1 and 2 to the Python Nexmark suite
URL: https://github.com/apache/beam/pull/5837#issuecomment-403162385
 
 
   Thanks, LGTM.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120106)
Time Spent: 13.5h  (was: 13h 20m)

> Implement a python version of the nexmark queries
> -
>
> Key: BEAM-2855
> URL: https://issues.apache.org/jira/browse/BEAM-2855
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-nexmark, sdk-py-core, testing
>Reporter: Ismaël Mejía
>Assignee: María GH
>Priority: Minor
>  Labels: newbie, nexmark, starter
>  Time Spent: 13.5h
>  Remaining Estimate: 0h
>
> Currently we have a Java only implementation of Nexmark, a python based 
> implementation would be nice to have to validate the direct and dataflow 
> runners, but also to validate the new support of multiple SDKs in multiple 
> runners via the runner/fn API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (9cf9fec -> 92275d6)

2018-07-06 Thread ccy

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 9cf9fec  Merge pull request #5894: [BEAM-4016] Invoke Setup and 
TearDown on SplitRestrictionFn and PairWithRestrictionFn
 add db29d7e  Add query 1 & 2
 add fcb5a89  Clean up
 add b845798  Cleaup
 add cd9eb2f  Fix linter errors
 add c667b64  Address review commnets; Reuse topic for queries
 new 92275d6  Merge pull request #5837 from mariapython/query1_2_pr

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../testing/benchmarks/nexmark/nexmark_launcher.py | 43 --
 .../testing/benchmarks/nexmark/nexmark_util.py |  6 ++-
 .../testing/benchmarks/nexmark/queries/query0.py   |  6 ++-
 .../nexmark/queries/{query0.py => query1.py}   | 26 +
 .../nexmark/queries/{query0.py => query2.py}   | 19 +++---
 5 files changed, 73 insertions(+), 27 deletions(-)
 copy sdks/python/apache_beam/testing/benchmarks/nexmark/queries/{query0.py => 
query1.py} (54%)
 copy sdks/python/apache_beam/testing/benchmarks/nexmark/queries/{query0.py => 
query2.py} (59%)

[beam] 01/01: Merge pull request #5837 from mariapython/query1_2_pr

2018-07-06 Thread ccy

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 92275d64a29e71bc27e2bf7ca215fc96c1a9e480
Merge: 9cf9fec c667b64
Author: Charles Chen 
AuthorDate: Fri Jul 6 15:32:31 2018 -0700

Merge pull request #5837 from mariapython/query1_2_pr

[BEAM-2855] Add queries 1 and 2 to the Python Nexmark suite

 .../testing/benchmarks/nexmark/nexmark_launcher.py | 43 --
 .../testing/benchmarks/nexmark/nexmark_util.py |  6 ++-
 .../testing/benchmarks/nexmark/queries/query0.py   |  6 ++-
 .../nexmark/queries/{query0.py => query1.py}   | 26 +
 .../nexmark/queries/{query0.py => query2.py}   | 19 +++---
 5 files changed, 73 insertions(+), 27 deletions(-)

[jira] [Closed] (BEAM-4739) org.apache.beam.sdk.testing.PAssertTest.intentionalFail : java.lang.AssertionError: Intentional fail expected:<0> but was:<1>

2018-07-06 Thread Mikhail Gryzykhin (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Gryzykhin closed BEAM-4739.
---

> org.apache.beam.sdk.testing.PAssertTest.intentionalFail : 
> java.lang.AssertionError: Intentional fail expected:<0> but was:<1>
> -
>
> Key: BEAM-4739
> URL: https://issues.apache.org/jira/browse/BEAM-4739
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
> Fix For: Not applicable
>
>
> http://localhost:8080/job/guinea_job_sdk_java_core_test/16/
> java.lang.AssertionError: Intentional fail expected:<0> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at 
> org.apache.beam.sdk.testing.PAssertTest.intentionalFail(PAssertTest.java:150)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319)
>   at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
>   at 
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at com.sun.proxy.$Proxy1.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.internal.remote.internal.

[jira] [Work logged] (BEAM-4016) @SplitRestriction should execute after @Setup on SplittableDoFn

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4016?focusedWorklogId=120103&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120103
 ]

ASF GitHub Bot logged work on BEAM-4016:


Author: ASF GitHub Bot
Created on: 06/Jul/18 22:28
Start Date: 06/Jul/18 22:28
Worklog Time Spent: 10m 
  Work Description: iemejia closed pull request #5894: [BEAM-4016] Invoke 
Setup and TearDown on SplitRestrictionFn and PairWithRestrictionFn
URL: https://github.com/apache/beam/pull/5894
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SplittableParDo.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SplittableParDo.java
index b581eecf414..8d8da216afa 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SplittableParDo.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SplittableParDo.java
@@ -415,6 +415,7 @@ public String apply(T input) {
 @Setup
 public void setup() {
   invoker = DoFnInvokers.invokerFor(fn);
+  invoker.invokeSetup();
 }
 
 @ProcessElement
@@ -422,6 +423,12 @@ public void processElement(ProcessContext context) {
   context.output(
   KV.of(context.element(), 
invoker.invokeGetInitialRestriction(context.element(;
 }
+
+@Teardown
+public void tearDown() {
+  invoker.invokeTeardown();
+  invoker = null;
+}
   }
 
   /** Splits the restriction using the given {@link SplitRestriction} method. 
*/
@@ -439,6 +446,7 @@ public void processElement(ProcessContext context) {
 @Setup
 public void setup() {
   invoker = DoFnInvokers.invokerFor(splittableFn);
+  invoker.invokeSetup();
 }
 
 @ProcessElement
@@ -459,5 +467,11 @@ public void outputWithTimestamp(RestrictionT part, Instant 
timestamp) {
 }
   });
 }
+
+@Teardown
+public void tearDown() {
+  invoker.invokeTeardown();
+  invoker = null;
+}
   }
 }
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java
index b7f0c10d046..fe33b1ab5b8 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java
@@ -513,18 +513,19 @@ public void testLateData() throws Exception {
 
 private State state = State.BEFORE_SETUP;
 
-@ProcessElement
-public void processElement(ProcessContext c, OffsetRangeTracker tracker) {
-  assertEquals(State.INSIDE_BUNDLE, state);
-  assertTrue(tracker.tryClaim(0L));
-  c.output(c.element());
-}
-
 @GetInitialRestriction
 public OffsetRange getInitialRestriction(String value) {
+  assertEquals(State.OUTSIDE_BUNDLE, state);
   return new OffsetRange(0, 1);
 }
 
+@SplitRestriction
+public void splitRestriction(
+String value, OffsetRange range, OutputReceiver receiver) 
{
+  assertEquals(State.OUTSIDE_BUNDLE, state);
+  receiver.output(range);
+}
+
 @Setup
 public void setUp() {
   assertEquals(State.BEFORE_SETUP, state);
@@ -537,6 +538,13 @@ public void startBundle() {
   state = State.INSIDE_BUNDLE;
 }
 
+@ProcessElement
+public void processElement(ProcessContext c, OffsetRangeTracker tracker) {
+  assertEquals(State.INSIDE_BUNDLE, state);
+  assertTrue(tracker.tryClaim(0L));
+  c.output(c.element());
+}
+
 @FinishBundle
 public void finishBundle() {
   assertEquals(State.INSIDE_BUNDLE, state);
@@ -553,12 +561,9 @@ public void tearDown() {
   @Test
   @Category({ValidatesRunner.class, UsesSplittableParDo.class})
   public void testLifecycleMethods() throws Exception {
-
 PCollection res =
 p.apply(Create.of("a", "b", "c")).apply(ParDo.of(new 
SDFWithLifecycle()));
-
 PAssert.that(res).containsInAnyOrder("a", "b", "c");
-
 p.run();
   }
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120103)
Time Spent: 0.5h  (was: 20m)

> @SplitRestriction should execute after @Setup on SplittableDoFn
> -

[beam] branch master updated (85bcd37 -> 9cf9fec)

2018-07-06 Thread iemejia

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 85bcd37  [BEAM-4738] [SQL] Remove decimal support in BigQuery SQL read.
 add 59345ff  [BEAM-4016] Invoke Setup and TearDown on SplitRestrictionFn 
and PairWithRestrictionFn
 new 9cf9fec  Merge pull request #5894: [BEAM-4016] Invoke Setup and 
TearDown on SplitRestrictionFn and PairWithRestrictionFn

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../runners/core/construction/SplittableParDo.java | 14 
 .../beam/sdk/transforms/SplittableDoFnTest.java| 25 +-
 2 files changed, 29 insertions(+), 10 deletions(-)

[jira] [Resolved] (BEAM-4016) @SplitRestriction should execute after @Setup on SplittableDoFn

2018-07-06 Thread JIRA



 [ 
https://issues.apache.org/jira/browse/BEAM-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-4016.

   Resolution: Fixed
Fix Version/s: 2.6.0

> @SplitRestriction should execute after @Setup on SplittableDoFn
> ---
>
> Key: BEAM-4016
> URL: https://issues.apache.org/jira/browse/BEAM-4016
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 2.4.0, 2.5.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Major
> Fix For: 2.6.0
>
> Attachments: sdf-splitrestriction-lifeycle-test.patch
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The method annotated with @SplitRestriction is the method where we can define 
> the RestrictionTrackers (splits) in advance in a SDF. It makes sense to 
> execute this after the @Setup method given that usually connections are 
> established at Setup and can be used to ask the different data stores about 
> the partitioning strategy. I added a test for this in the 
> SplittableDoFnTest.SDFWithLifecycle test.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #5894: [BEAM-4016] Invoke Setup and TearDown on SplitRestrictionFn and PairWithRestrictionFn

2018-07-06 Thread iemejia

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 9cf9fecb314938ea10d8556dd818c59a19dc7ff9
Merge: 85bcd37 59345ff
Author: Ismaël Mejía 
AuthorDate: Sat Jul 7 00:28:32 2018 +0200

Merge pull request #5894: [BEAM-4016] Invoke Setup and TearDown on 
SplitRestrictionFn and PairWithRestrictionFn

 .../runners/core/construction/SplittableParDo.java | 14 
 .../beam/sdk/transforms/SplittableDoFnTest.java| 25 +-
 2 files changed, 29 insertions(+), 10 deletions(-)

[jira] [Resolved] (BEAM-4739) org.apache.beam.sdk.testing.PAssertTest.intentionalFail : java.lang.AssertionError: Intentional fail expected:<0> but was:<1>

2018-07-06 Thread Mikhail Gryzykhin (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Gryzykhin resolved BEAM-4739.
-
   Resolution: Not A Problem
Fix Version/s: Not applicable

Testing JiraTestResultsReporter plugin.

> org.apache.beam.sdk.testing.PAssertTest.intentionalFail : 
> java.lang.AssertionError: Intentional fail expected:<0> but was:<1>
> -
>
> Key: BEAM-4739
> URL: https://issues.apache.org/jira/browse/BEAM-4739
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
> Fix For: Not applicable
>
>
> http://localhost:8080/job/guinea_job_sdk_java_core_test/16/
> java.lang.AssertionError: Intentional fail expected:<0> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at 
> org.apache.beam.sdk.testing.PAssertTest.intentionalFail(PAssertTest.java:150)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319)
>   at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
>   at 
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at com.sun.proxy.$Proxy1.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.internal.dis

[jira] [Created] (BEAM-4739) org.apache.beam.sdk.testing.PAssertTest.intentionalFail : java.lang.AssertionError: Intentional fail expected:<0> but was:<1>

2018-07-06 Thread Mikhail Gryzykhin (JIRA)

Mikhail Gryzykhin created BEAM-4739:
---

 Summary: org.apache.beam.sdk.testing.PAssertTest.intentionalFail : 
java.lang.AssertionError: Intentional fail expected:<0> but was:<1>
 Key: BEAM-4739
 URL: https://issues.apache.org/jira/browse/BEAM-4739
 Project: Beam
  Issue Type: Bug
  Components: test-failures
Reporter: Mikhail Gryzykhin


http://localhost:8080/job/guinea_job_sdk_java_core_test/16/
java.lang.AssertionError: Intentional fail expected:<0> but was:<1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at 
org.apache.beam.sdk.testing.PAssertTest.intentionalFail(PAssertTest.java:150)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at 
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at 
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy1.processTestClass(Unknown Source)
at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:155)
at 
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:137)
at 
org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
at 
org.

[jira] [Work logged] (BEAM-4654) Update pipeline translation for timers inside Java SDK

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4654?focusedWorklogId=120101&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120101
 ]

ASF GitHub Bot logged work on BEAM-4654:


Author: ASF GitHub Bot
Created on: 06/Jul/18 22:13
Start Date: 06/Jul/18 22:13
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5883: 
[BEAM-4654] Treat timers as PCollections within proto representation in the 
Java SDK.
URL: https://github.com/apache/beam/pull/5883#discussion_r200781061
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
 ##
 @@ -88,39 +91,72 @@
   "urn:beam:windowmappingfn:javasdk:0.1";
 
   /** A {@link TransformPayloadTranslator} for {@link ParDo}. */
-  public static class ParDoPayloadTranslator
-  implements TransformPayloadTranslator> {
-public static TransformPayloadTranslator create() {
-  return new ParDoPayloadTranslator();
+  public static class ParDoTranslator implements 
TransformTranslator> {
+
+public static TransformTranslator create() {
+  return new ParDoTranslator();
 }
 
-private ParDoPayloadTranslator() {}
+private ParDoTranslator() {}
 
 @Override
 public String getUrn(ParDo.MultiOutput transform) {
   return PAR_DO_TRANSFORM_URN;
 }
 
 @Override
-public FunctionSpec translate(
-AppliedPTransform> transform, SdkComponents 
components)
-throws IOException {
-  ParDoPayload payload =
-  translateParDo(transform.getTransform(), transform.getPipeline(), 
components);
-  return RunnerApi.FunctionSpec.newBuilder()
-  .setUrn(PAR_DO_TRANSFORM_URN)
-  .setPayload(payload.toByteString())
-  .build();
+public boolean canTranslate(PTransform pTransform) {
+  return pTransform instanceof ParDo.MultiOutput;
 }
 
-/** Registers {@link ParDoPayloadTranslator}. */
-@AutoService(TransformPayloadTranslatorRegistrar.class)
-public static class Registrar implements 
TransformPayloadTranslatorRegistrar {
-  @Override
-  public Map, ? extends 
TransformPayloadTranslator>
-  getTransformPayloadTranslators() {
-return Collections.singletonMap(ParDo.MultiOutput.class, new 
ParDoPayloadTranslator());
+@Override
+public RunnerApi.PTransform translate(
+AppliedPTransform appliedPTransform,
+List> subtransforms,
+SdkComponents components)
+throws IOException {
+  RunnerApi.PTransform.Builder builder =
+  PTransformTranslation.translateAppliedPTransform(
+  appliedPTransform, subtransforms, components);
+
+  ParDoPayload payload =
+  translateParDo(
+  (ParDo.MultiOutput) appliedPTransform.getTransform(),
+  appliedPTransform.getPipeline(),
+  components);
+  builder.setSpec(
+  RunnerApi.FunctionSpec.newBuilder()
+  .setUrn(PAR_DO_TRANSFORM_URN)
+  .setPayload(payload.toByteString())
+  .build());
+
+  String mainInputId = getMainInputId(builder, payload);
+  PCollection> mainInput =
+  (PCollection) appliedPTransform.getInputs().get(new 
TupleTag(mainInputId));
+
+  // https://s.apache.org/beam-portability-timers
+  // Add a PCollection and coder for each timer. Also treat them as inputs 
and outputs.
+  for (String localTimerName : payload.getTimerSpecsMap().keySet()) {
+PCollection timerPCollection =
 
 Review comment:
   How is TimerSpec encoded in this PCollectoon?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120101)
Time Spent: 50m  (was: 40m)

> Update pipeline translation for timers inside Java SDK
> --
>
> Key: BEAM-4654
> URL: https://issues.apache.org/jira/browse/BEAM-4654
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Labels: portability
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Add the timer PCollection and treat timers as inputs/outputs of the ParDo 
> PTransform.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4016) @SplitRestriction should execute after @Setup on SplittableDoFn

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4016?focusedWorklogId=120099&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120099
 ]

ASF GitHub Bot logged work on BEAM-4016:


Author: ASF GitHub Bot
Created on: 06/Jul/18 22:07
Start Date: 06/Jul/18 22:07
Worklog Time Spent: 10m 
  Work Description: jkff commented on issue #5894: [BEAM-4016] Invoke Setup 
and TearDown on SplitRestrictionFn and PairWithRestrictionFn
URL: https://github.com/apache/beam/pull/5894#issuecomment-403158680
 
 
   LGTM, thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120099)
Time Spent: 20m  (was: 10m)

> @SplitRestriction should execute after @Setup on SplittableDoFn
> ---
>
> Key: BEAM-4016
> URL: https://issues.apache.org/jira/browse/BEAM-4016
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 2.4.0, 2.5.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Major
> Attachments: sdf-splitrestriction-lifeycle-test.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The method annotated with @SplitRestriction is the method where we can define 
> the RestrictionTrackers (splits) in advance in a SDF. It makes sense to 
> execute this after the @Setup method given that usually connections are 
> established at Setup and can be used to ask the different data stores about 
> the partitioning strategy. I added a test for this in the 
> SplittableDoFnTest.SDFWithLifecycle test.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4016) @SplitRestriction should execute after @Setup on SplittableDoFn

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4016?focusedWorklogId=120096&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120096
 ]

ASF GitHub Bot logged work on BEAM-4016:


Author: ASF GitHub Bot
Created on: 06/Jul/18 22:05
Start Date: 06/Jul/18 22:05
Worklog Time Spent: 10m 
  Work Description: iemejia opened a new pull request #5894: [BEAM-4016] 
Invoke Setup and TearDown on SplitRestrictionFn and PairWithRestrictionFn
URL: https://github.com/apache/beam/pull/5894
 
 
   R: @jkff 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120096)
Time Spent: 10m
Remaining Estimate: 0h

> @SplitRestriction should execute after @Setup on SplittableDoFn
> ---
>
> Key: BEAM-4016
> URL: https://issues.apache.org/jira/browse/BEAM-4016
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 2.4.0, 2.5.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Major
> Attachments: sdf-splitrestriction-lifeycle-test.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The method annotated with @SplitRestriction is the method where we can define 
> the RestrictionTrackers (splits) in advance in a SDF. It makes sense to 
> execute this after the @Setup method given that usually connections are 
> established at Setup and can be used to ask the different data stores about 
> the partitioning strategy. I added a test for this in the 
> SplittableDoFnTest.SDFWithLifecycle test.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4016) @SplitRestriction should execute after @Setup on SplittableDoFn

2018-07-06 Thread JIRA



 [ 
https://issues.apache.org/jira/browse/BEAM-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-4016:
---
Summary: @SplitRestriction should execute after @Setup on SplittableDoFn  
(was: Direct runner incorrect lifecycle, @SplitRestriction should execute after 
@Setup on SplittableDoFn)

> @SplitRestriction should execute after @Setup on SplittableDoFn
> ---
>
> Key: BEAM-4016
> URL: https://issues.apache.org/jira/browse/BEAM-4016
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 2.4.0, 2.5.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Major
> Attachments: sdf-splitrestriction-lifeycle-test.patch
>
>
> The method annotated with @SplitRestriction is the method where we can define 
> the RestrictionTrackers (splits) in advance in a SDF. It makes sense to 
> execute this after the @Setup method given that usually connections are 
> established at Setup and can be used to ask the different data stores about 
> the partitioning strategy. I added a test for this in the 
> SplittableDoFnTest.SDFWithLifecycle test.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4733) Python portable runner to pass pipeline options to job service

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4733?focusedWorklogId=120092&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120092
 ]

ASF GitHub Bot logged work on BEAM-4733:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:58
Start Date: 06/Jul/18 21:58
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5888: [BEAM-4733] Pass 
pipeline options from Python portable runner to job server.
URL: https://github.com/apache/beam/pull/5888#issuecomment-403156795
 
 
   A runner can look at whether any of the PCollections are unbounded instead 
of relying on `--streaming`. The SDK should pass options through but for now we 
are attempting to prevent the runner from inspecting them until someone has 
time to think of a good story around how runners and PipelineOptions interact.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120092)
Time Spent: 40m  (was: 0.5h)

> Python portable runner to pass pipeline options to job service
> --
>
> Key: BEAM-4733
> URL: https://issues.apache.org/jira/browse/BEAM-4733
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink, sdk-py-harness
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The portable runner client currently drops the pipeline options. In the 
> prototype, it was implemented here: 
> [https://github.com/bsidhom/beam/commit/ce8a79122b98a7cbcf6fea7db4a5fe31b6e8248a#diff-f2f54a3d6ae6ef6a22e5d52621a133ed]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-4738) org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQL java.nio.HeapByteBuffer cannot be cast to java.lang.Double

2018-07-06 Thread Rui Wang (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang closed BEAM-4738.
--
   Resolution: Fixed
Fix Version/s: Not applicable

The investigation work is tracked in BEAM-4736

> org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQL
>  java.nio.HeapByteBuffer cannot be cast to java.lang.Double
> 
>
> Key: BEAM-4738
> URL: https://issues.apache.org/jira/browse/BEAM-4738
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Luke Cwik
>Assignee: Rui Wang
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQLRead
>  fails a lot:
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1022/testReport/]
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1023/testReport/]
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1027/testReport/]
> {code:java}
> java.lang.ClassCastException: java.nio.HeapByteBuffer cannot be cast to 
> java.lang.Double at 
> org.apache.beam.sdk.io.gcp.bigquery.AvroUtils.convertAvroPrimitiveTypes(AvroUtils.java:102)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.AvroUtils.convertAvroFormat(AvroUtils.java:41)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.toBeamRow(BigQueryUtils.java:207)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils$ToBeamRow.apply(BigQueryUtils.java:199)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils$ToBeamRow.apply(BigQueryUtils.java:186)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase$1.apply(BigQuerySourceBase.java:222)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase$1.apply(BigQuerySourceBase.java:215)
>  at 
> org.apache.beam.sdk.io.AvroSource$AvroBlock.readNextRecord(AvroSource.java:567)
>  at 
> org.apache.beam.sdk.io.BlockBasedSource$BlockBasedReader.readNextRecord(BlockBasedSource.java:209)
>  at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:484)
>  at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:479)
>  at 
> org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:249)
>  at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:147)
>  at 
> org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:160)
>  at 
> org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:124)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4738) org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQL java.nio.HeapByteBuffer cannot be cast to java.lang.Double

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4738?focusedWorklogId=120091&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120091
 ]

ASF GitHub Bot logged work on BEAM-4738:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:54
Start Date: 06/Jul/18 21:54
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #5893: [BEAM-4738] Ignore 
test that fails a lot.
URL: https://github.com/apache/beam/pull/5893#issuecomment-403156120
 
 
   Thanks for the merge!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120091)
Time Spent: 1h 10m  (was: 1h)

> org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQL
>  java.nio.HeapByteBuffer cannot be cast to java.lang.Double
> 
>
> Key: BEAM-4738
> URL: https://issues.apache.org/jira/browse/BEAM-4738
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Luke Cwik
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQLRead
>  fails a lot:
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1022/testReport/]
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1023/testReport/]
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1027/testReport/]
> {code:java}
> java.lang.ClassCastException: java.nio.HeapByteBuffer cannot be cast to 
> java.lang.Double at 
> org.apache.beam.sdk.io.gcp.bigquery.AvroUtils.convertAvroPrimitiveTypes(AvroUtils.java:102)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.AvroUtils.convertAvroFormat(AvroUtils.java:41)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.toBeamRow(BigQueryUtils.java:207)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils$ToBeamRow.apply(BigQueryUtils.java:199)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils$ToBeamRow.apply(BigQueryUtils.java:186)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase$1.apply(BigQuerySourceBase.java:222)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase$1.apply(BigQuerySourceBase.java:215)
>  at 
> org.apache.beam.sdk.io.AvroSource$AvroBlock.readNextRecord(AvroSource.java:567)
>  at 
> org.apache.beam.sdk.io.BlockBasedSource$BlockBasedReader.readNextRecord(BlockBasedSource.java:209)
>  at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:484)
>  at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:479)
>  at 
> org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:249)
>  at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:147)
>  at 
> org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:160)
>  at 
> org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:124)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4738) org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQL java.nio.HeapByteBuffer cannot be cast to java.lang.Double

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4738?focusedWorklogId=120088&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120088
 ]

ASF GitHub Bot logged work on BEAM-4738:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:53
Start Date: 06/Jul/18 21:53
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #5892: [BEAM-4738] [SQL] 
Remove decimal support in BigQuery SQL read.
URL: https://github.com/apache/beam/pull/5892
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryReadWriteIT.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryReadWriteIT.java
index c39da40939e..8b3b4a64769 100644
--- 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryReadWriteIT.java
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryReadWriteIT.java
@@ -19,7 +19,6 @@
 
 import static org.apache.beam.sdk.schemas.Schema.FieldType.BOOLEAN;
 import static org.apache.beam.sdk.schemas.Schema.FieldType.BYTE;
-import static org.apache.beam.sdk.schemas.Schema.FieldType.DECIMAL;
 import static org.apache.beam.sdk.schemas.Schema.FieldType.DOUBLE;
 import static org.apache.beam.sdk.schemas.Schema.FieldType.FLOAT;
 import static org.apache.beam.sdk.schemas.Schema.FieldType.INT16;
@@ -32,7 +31,6 @@
 
 import com.google.common.collect.ImmutableMap;
 import java.io.Serializable;
-import java.math.BigDecimal;
 import java.util.Arrays;
 import java.util.List;
 import org.apache.beam.sdk.Pipeline;
@@ -77,7 +75,6 @@
   .addNullableField("c_integer", INT32)
   .addNullableField("c_float", FLOAT)
   .addNullableField("c_double", DOUBLE)
-  .addNullableField("c_decimal", DECIMAL)
   .addNullableField("c_boolean", BOOLEAN)
   .addNullableField("c_timestamp", 
FieldType.DATETIME.withMetadata("TS"))
   .addNullableField("c_varchar", STRING)
@@ -102,7 +99,6 @@ public void testSQLRead() {
 + "   c_integer INTEGER, \n"
 + "   c_float FLOAT, \n"
 + "   c_double DOUBLE, \n"
-+ "   c_decimal DECIMAL, \n"
 + "   c_boolean BOOLEAN, \n"
 + "   c_timestamp TIMESTAMP, \n"
 + "   c_varchar VARCHAR, \n "
@@ -123,7 +119,6 @@ public void testSQLRead() {
 + "2147483647, "
 + "1.0, "
 + "1.0, "
-+ "123.45, "
 + "TRUE, "
 + "TIMESTAMP '2018-05-28 20:17:40.123', "
 + "'varchar', "
@@ -149,7 +144,6 @@ public void testSQLRead() {
 2147483647,
 (float) 1.0,
 1.0,
-BigDecimal.valueOf(123.45),
 true,
 new DateTime(2018, 05, 28, 20, 17, 40, 123, 
ISOChronology.getInstanceUTC()),
 "varchar",
@@ -171,7 +165,6 @@ public void testSQLTypes() {
 + "   c_integer INTEGER, \n"
 + "   c_float FLOAT, \n"
 + "   c_double DOUBLE, \n"
-+ "   c_decimal DECIMAL, \n"
 + "   c_boolean BOOLEAN, \n"
 + "   c_timestamp TIMESTAMP, \n"
 + "   c_varchar VARCHAR, \n "
@@ -192,7 +185,6 @@ public void testSQLTypes() {
 + "2147483647, "
 + "1.0, "
 + "1.0, "
-+ "123.45, "
 + "TRUE, "
 + "TIMESTAMP '2018-05-28 20:17:40.123', "
 + "'varchar', "
@@ -214,7 +206,6 @@ public void testSQLTypes() {
 2147483647,
 (float) 1.0,
 1.0,
-BigDecimal.valueOf(123.45),
 true,
 new DateTime(2018, 05, 28, 20, 17, 40, 123, 
ISOChronology.getInstanceUTC()),
 "varchar",
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroUtils.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroUtils.java
index c9d9c9fe17b..e36700d0726 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroUtils.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroUtils.java
@@ -17,7 +17,6 @@
  */
 package org.apache.beam.sdk.io.gcp.bigquery;
 
-import java.math.BigDecimal;
 import java.util.ArrayList;
 import java.util.List;
 import org.apache.beam.sdk.schemas.Schema.Field;
@@ -35,7 +34,6 @@ public static Objec

[jira] [Work logged] (BEAM-4738) org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQL java.nio.HeapByteBuffer cannot be cast to java.lang.Double

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4738?focusedWorklogId=120089&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120089
 ]

ASF GitHub Bot logged work on BEAM-4738:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:53
Start Date: 06/Jul/18 21:53
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5893: [BEAM-4738] Ignore 
test that fails a lot.
URL: https://github.com/apache/beam/pull/5893#issuecomment-403156039
 
 
   Merged, #5892, closed this.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120089)
Time Spent: 50m  (was: 40m)

> org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQL
>  java.nio.HeapByteBuffer cannot be cast to java.lang.Double
> 
>
> Key: BEAM-4738
> URL: https://issues.apache.org/jira/browse/BEAM-4738
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Luke Cwik
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQLRead
>  fails a lot:
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1022/testReport/]
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1023/testReport/]
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1027/testReport/]
> {code:java}
> java.lang.ClassCastException: java.nio.HeapByteBuffer cannot be cast to 
> java.lang.Double at 
> org.apache.beam.sdk.io.gcp.bigquery.AvroUtils.convertAvroPrimitiveTypes(AvroUtils.java:102)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.AvroUtils.convertAvroFormat(AvroUtils.java:41)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.toBeamRow(BigQueryUtils.java:207)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils$ToBeamRow.apply(BigQueryUtils.java:199)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils$ToBeamRow.apply(BigQueryUtils.java:186)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase$1.apply(BigQuerySourceBase.java:222)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase$1.apply(BigQuerySourceBase.java:215)
>  at 
> org.apache.beam.sdk.io.AvroSource$AvroBlock.readNextRecord(AvroSource.java:567)
>  at 
> org.apache.beam.sdk.io.BlockBasedSource$BlockBasedReader.readNextRecord(BlockBasedSource.java:209)
>  at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:484)
>  at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:479)
>  at 
> org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:249)
>  at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:147)
>  at 
> org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:160)
>  at 
> org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:124)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4738) org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQL java.nio.HeapByteBuffer cannot be cast to java.lang.Double

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4738?focusedWorklogId=120090&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120090
 ]

ASF GitHub Bot logged work on BEAM-4738:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:53
Start Date: 06/Jul/18 21:53
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #5893: [BEAM-4738] Ignore 
test that fails a lot.
URL: https://github.com/apache/beam/pull/5893
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryReadWriteIT.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryReadWriteIT.java
index c39da40939e..b087352f1a2 100644
--- 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryReadWriteIT.java
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryReadWriteIT.java
@@ -54,6 +54,7 @@
 import org.joda.time.DateTime;
 import org.joda.time.Duration;
 import org.joda.time.chrono.ISOChronology;
+import org.junit.Ignore;
 import org.junit.Rule;
 import org.junit.Test;
 import org.junit.runner.RunWith;
@@ -91,6 +92,7 @@
   @Rule public transient TestBigQuery bigQueryTestingTypes = 
TestBigQuery.create(SOURCE_SCHEMA_TWO);
 
   @Test
+  @Ignore("BEAM-4738")
   public void testSQLRead() {
 BeamSqlEnv sqlEnv = BeamSqlEnv.inMemory(new BigQueryTableProvider());
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120090)
Time Spent: 1h  (was: 50m)

> org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQL
>  java.nio.HeapByteBuffer cannot be cast to java.lang.Double
> 
>
> Key: BEAM-4738
> URL: https://issues.apache.org/jira/browse/BEAM-4738
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Luke Cwik
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQLRead
>  fails a lot:
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1022/testReport/]
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1023/testReport/]
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1027/testReport/]
> {code:java}
> java.lang.ClassCastException: java.nio.HeapByteBuffer cannot be cast to 
> java.lang.Double at 
> org.apache.beam.sdk.io.gcp.bigquery.AvroUtils.convertAvroPrimitiveTypes(AvroUtils.java:102)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.AvroUtils.convertAvroFormat(AvroUtils.java:41)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.toBeamRow(BigQueryUtils.java:207)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils$ToBeamRow.apply(BigQueryUtils.java:199)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils$ToBeamRow.apply(BigQueryUtils.java:186)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase$1.apply(BigQuerySourceBase.java:222)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase$1.apply(BigQuerySourceBase.java:215)
>  at 
> org.apache.beam.sdk.io.AvroSource$AvroBlock.readNextRecord(AvroSource.java:567)
>  at 
> org.apache.beam.sdk.io.BlockBasedSource$BlockBasedReader.readNextRecord(BlockBasedSource.java:209)
>  at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:484)
>  at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:479)
>  at 
> org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:249)
>  at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:147)
>  at 
> org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:160)
>  at 
> org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:124)
>  at java.util.concurrent.Executor

[beam] 01/01: [BEAM-4738] [SQL] Remove decimal support in BigQuery SQL read.

2018-07-06 Thread lcwik

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 85bcd37933b5d11bc87f82ac2c22017944837e45
Merge: 6f6feaa 2dc55c3
Author: Lukasz Cwik 
AuthorDate: Fri Jul 6 14:53:23 2018 -0700

[BEAM-4738] [SQL] Remove decimal support in BigQuery SQL read.

 .../sql/meta/provider/bigquery/BigQueryReadWriteIT.java  | 9 -
 .../main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroUtils.java | 6 +++---
 2 files changed, 3 insertions(+), 12 deletions(-)

[beam] branch master updated (6f6feaa -> 85bcd37)

2018-07-06 Thread lcwik

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 6f6feaa  Merge pull request #5715 from Fematich/io
 add 2dc55c3  remove decimal support in BigQuery SQL read.
 new 85bcd37  [BEAM-4738] [SQL] Remove decimal support in BigQuery SQL read.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../sql/meta/provider/bigquery/BigQueryReadWriteIT.java  | 9 -
 .../main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroUtils.java | 6 +++---
 2 files changed, 3 insertions(+), 12 deletions(-)

[jira] [Work logged] (BEAM-4738) org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQL java.nio.HeapByteBuffer cannot be cast to java.lang.Double

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4738?focusedWorklogId=120087&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120087
 ]

ASF GitHub Bot logged work on BEAM-4738:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:50
Start Date: 06/Jul/18 21:50
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #5893: [BEAM-4738] Ignore 
test that fails a lot.
URL: https://github.com/apache/beam/pull/5893#issuecomment-403155472
 
 
   #5892 can fix the break if it can be merged. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120087)
Time Spent: 0.5h  (was: 20m)

> org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQL
>  java.nio.HeapByteBuffer cannot be cast to java.lang.Double
> 
>
> Key: BEAM-4738
> URL: https://issues.apache.org/jira/browse/BEAM-4738
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Luke Cwik
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQLRead
>  fails a lot:
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1022/testReport/]
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1023/testReport/]
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1027/testReport/]
> {code:java}
> java.lang.ClassCastException: java.nio.HeapByteBuffer cannot be cast to 
> java.lang.Double at 
> org.apache.beam.sdk.io.gcp.bigquery.AvroUtils.convertAvroPrimitiveTypes(AvroUtils.java:102)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.AvroUtils.convertAvroFormat(AvroUtils.java:41)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.toBeamRow(BigQueryUtils.java:207)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils$ToBeamRow.apply(BigQueryUtils.java:199)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils$ToBeamRow.apply(BigQueryUtils.java:186)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase$1.apply(BigQuerySourceBase.java:222)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase$1.apply(BigQuerySourceBase.java:215)
>  at 
> org.apache.beam.sdk.io.AvroSource$AvroBlock.readNextRecord(AvroSource.java:567)
>  at 
> org.apache.beam.sdk.io.BlockBasedSource$BlockBasedReader.readNextRecord(BlockBasedSource.java:209)
>  at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:484)
>  at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:479)
>  at 
> org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:249)
>  at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:147)
>  at 
> org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:160)
>  at 
> org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:124)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4738) org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQL java.nio.HeapByteBuffer cannot be cast to java.lang.Double

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4738?focusedWorklogId=120085&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120085
 ]

ASF GitHub Bot logged work on BEAM-4738:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:46
Start Date: 06/Jul/18 21:46
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5893: [BEAM-4738] Ignore 
test that fails a lot.
URL: https://github.com/apache/beam/pull/5893#issuecomment-403154653
 
 
   Commit-then-review for post commit break.
   
   CC: @kennknowles @amaliujia 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120085)
Time Spent: 20m  (was: 10m)

> org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQL
>  java.nio.HeapByteBuffer cannot be cast to java.lang.Double
> 
>
> Key: BEAM-4738
> URL: https://issues.apache.org/jira/browse/BEAM-4738
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Luke Cwik
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQLRead
>  fails a lot:
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1022/testReport/]
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1023/testReport/]
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1027/testReport/]
> {code:java}
> java.lang.ClassCastException: java.nio.HeapByteBuffer cannot be cast to 
> java.lang.Double at 
> org.apache.beam.sdk.io.gcp.bigquery.AvroUtils.convertAvroPrimitiveTypes(AvroUtils.java:102)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.AvroUtils.convertAvroFormat(AvroUtils.java:41)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.toBeamRow(BigQueryUtils.java:207)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils$ToBeamRow.apply(BigQueryUtils.java:199)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils$ToBeamRow.apply(BigQueryUtils.java:186)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase$1.apply(BigQuerySourceBase.java:222)
>  at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase$1.apply(BigQuerySourceBase.java:215)
>  at 
> org.apache.beam.sdk.io.AvroSource$AvroBlock.readNextRecord(AvroSource.java:567)
>  at 
> org.apache.beam.sdk.io.BlockBasedSource$BlockBasedReader.readNextRecord(BlockBasedSource.java:209)
>  at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:484)
>  at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:479)
>  at 
> org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:249)
>  at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:147)
>  at 
> org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:160)
>  at 
> org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:124)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4601) BigQuery reads basic types from pure SQL

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4601?focusedWorklogId=120086&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120086
 ]

ASF GitHub Bot logged work on BEAM-4601:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:46
Start Date: 06/Jul/18 21:46
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5830: [BEAM-4601][SQL] 
Support BigQuery read from SQL.
URL: https://github.com/apache/beam/pull/5830#issuecomment-403154661
 
 
   I think that is what #5892 is trying to address?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120086)
Time Spent: 6h 40m  (was: 6.5h)

> BigQuery reads basic types from pure SQL
> 
>
> Key: BEAM-4601
> URL: https://issues.apache.org/jira/browse/BEAM-4601
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>
> Right now Beam SQL can created a BigQuery table, however, read from BigQuery 
> table is not supported yet. We want to support reading basic data types from 
> BigQuery table (not including ROW type).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4738) org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQL java.nio.HeapByteBuffer cannot be cast to java.lang.Double

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4738?focusedWorklogId=120084&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120084
 ]

ASF GitHub Bot logged work on BEAM-4738:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:45
Start Date: 06/Jul/18 21:45
Worklog Time Spent: 10m 
  Work Description: lukecwik opened a new pull request #5893: [BEAM-4738] 
Ignore test that fails a lot.
URL: https://github.com/apache/beam/pull/5893
 
 
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120084)
Time Spent: 10m
Remaining Estimate: 0h

> org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQL
>  java.nio.HeapByteBuffer cannot be cast to java.lang.Double
> 
>
> Key: BEAM-4738
> URL: https://issues.apache.org/jira/browse/BEAM-4738
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Luke Cwik
>Assignee: Rui Wang
>Priority: Major
>

[jira] [Work logged] (BEAM-2732) State tracking in Python is inefficient and has duplicated code

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-2732?focusedWorklogId=120079&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120079
 ]

ASF GitHub Bot logged work on BEAM-2732:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:38
Start Date: 06/Jul/18 21:38
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #5356: 
[BEAM-2732][BEAM-4028] Logging relies on StateSampler for context
URL: https://github.com/apache/beam/pull/5356#issuecomment-403153187
 
 
   Thanks Pablo!  This LGTM.  Can you rebase?  It looks like this is a great 
performance improvement too, cutting down processing overhead.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120079)
Time Spent: 18h 10m  (was: 18h)

> State tracking in Python is inefficient and has duplicated code
> ---
>
> Key: BEAM-2732
> URL: https://issues.apache.org/jira/browse/BEAM-2732
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 18h 10m
>  Remaining Estimate: 0h
>
> e.g logging and metrics keep state separately. State tracking should be 
> unified.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4738) org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQL java.nio.HeapByteBuffer cannot be cast to java.lang.Double

2018-07-06 Thread Luke Cwik (JIRA)

Luke Cwik created BEAM-4738:
---

 Summary: 
org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQL
 java.nio.HeapByteBuffer cannot be cast to java.lang.Double
 Key: BEAM-4738
 URL: https://issues.apache.org/jira/browse/BEAM-4738
 Project: Beam
  Issue Type: Bug
  Components: io-java-gcp
Reporter: Luke Cwik
Assignee: Rui Wang


org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQLRead
 fails a lot:
[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1022/testReport/]
[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1023/testReport/]
[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1027/testReport/]
{code:java}
java.lang.ClassCastException: java.nio.HeapByteBuffer cannot be cast to 
java.lang.Double at 
org.apache.beam.sdk.io.gcp.bigquery.AvroUtils.convertAvroPrimitiveTypes(AvroUtils.java:102)
 at 
org.apache.beam.sdk.io.gcp.bigquery.AvroUtils.convertAvroFormat(AvroUtils.java:41)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.toBeamRow(BigQueryUtils.java:207)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils$ToBeamRow.apply(BigQueryUtils.java:199)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils$ToBeamRow.apply(BigQueryUtils.java:186)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase$1.apply(BigQuerySourceBase.java:222)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase$1.apply(BigQuerySourceBase.java:215)
 at 
org.apache.beam.sdk.io.AvroSource$AvroBlock.readNextRecord(AvroSource.java:567) 
at 
org.apache.beam.sdk.io.BlockBasedSource$BlockBasedReader.readNextRecord(BlockBasedSource.java:209)
 at 
org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:484)
 at 
org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:479)
 at 
org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:249)
 at 
org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:147)
 at 
org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:160)
 at 
org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:124)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at 
java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
at java.lang.Thread.run(Thread.java:748){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4601) BigQuery reads basic types from pure SQL

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4601?focusedWorklogId=120076&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120076
 ]

ASF GitHub Bot logged work on BEAM-4601:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:35
Start Date: 06/Jul/18 21:35
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5830: [BEAM-4601][SQL] 
Support BigQuery read from SQL.
URL: https://github.com/apache/beam/pull/5830#issuecomment-403152652
 
 
   
org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryReadWriteIT.testSQLRead
 fails a lot:
   
https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1022/testReport/
   
https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1023/testReport/
   
https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1027/testReport/


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120076)
Time Spent: 6.5h  (was: 6h 20m)

> BigQuery reads basic types from pure SQL
> 
>
> Key: BEAM-4601
> URL: https://issues.apache.org/jira/browse/BEAM-4601
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>
> Right now Beam SQL can created a BigQuery table, however, read from BigQuery 
> table is not supported yet. We want to support reading basic data types from 
> BigQuery table (not including ROW type).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4020) Add HBaseIO.readAll() based on SDF

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4020?focusedWorklogId=120075&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120075
 ]

ASF GitHub Bot logged work on BEAM-4020:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:31
Start Date: 06/Jul/18 21:31
Worklog Time Spent: 10m 
  Work Description: iemejia commented on a change in pull request #5212: 
[BEAM-4020] Add HBaseIO.readAll() based on SDF
URL: https://github.com/apache/beam/pull/5212#discussion_r200775086
 
 

 ##
 File path: 
sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseIO.java
 ##
 @@ -107,6 +107,16 @@
  * .withFilter(filter));
  * }
  *
+ * {@link HBaseIO#readAll()} allows to execute multiple {@link Scan}s to 
multiple {@link Table}s.
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120075)
Time Spent: 4h 20m  (was: 4h 10m)

> Add HBaseIO.readAll() based on SDF
> --
>
> Key: BEAM-4020
> URL: https://issues.apache.org/jira/browse/BEAM-4020
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-hbase
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> Since the support from runners is still limited, it is probably wise to 
> create a first IO based on the current SDF batch implementation in Java to 
> validate/test it with a real data-store. Since HBase partitioning model is 
> quite straightforward it is a perfect candidate.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3443) Enable running the build within a sandbox without needing to have network access

2018-07-06 Thread Luke Cwik (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-3443.
-
   Resolution: Fixed
Fix Version/s: 2.5.0

> Enable running the build within a sandbox without needing to have network 
> access
> 
>
> Key: BEAM-3443
> URL: https://issues.apache.org/jira/browse/BEAM-3443
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: 2.5.0
>
>
> Being able to build the project without needing network access allows one to 
> hack on the project within an offline environment or build and run demos 
> without network access.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4020) Add HBaseIO.readAll() based on SDF

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4020?focusedWorklogId=120073&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120073
 ]

ASF GitHub Bot logged work on BEAM-4020:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:24
Start Date: 06/Jul/18 21:24
Worklog Time Spent: 10m 
  Work Description: iemejia commented on a change in pull request #5212: 
[BEAM-4020] Add HBaseIO.readAll() based on SDF
URL: https://github.com/apache/beam/pull/5212#discussion_r200773277
 
 

 ##
 File path: 
sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseReadSplittableDoFn.java
 ##
 @@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.hbase;
+
+import java.io.IOException;
+import java.util.List;
+import org.apache.beam.sdk.io.hadoop.SerializableConfiguration;
+import org.apache.beam.sdk.io.range.ByteKey;
+import org.apache.beam.sdk.io.range.ByteKeyRange;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.DoFn.BoundedPerElement;
+import org.apache.beam.sdk.transforms.splittabledofn.ByteKeyRangeTracker;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+
+/** A SplittableDoFn to read from HBase. */
+@BoundedPerElement
+class HBaseReadSplittableDoFn extends DoFn {
+  private final SerializableConfiguration serializableConfiguration;
+
+  private transient Connection connection;
+
+  HBaseReadSplittableDoFn(SerializableConfiguration serializableConfiguration) 
{
+this.serializableConfiguration = serializableConfiguration;
+  }
+
+  @Setup
+  public void setup() throws Exception {
+connection = 
ConnectionFactory.createConnection(serializableConfiguration.get());
+  }
+
+  private static Scan newScanInRange(Scan scan, ByteKeyRange range) throws 
IOException {
+return new Scan(scan)
+.setStartRow(range.getStartKey().getBytes())
+.setStopRow(range.getEndKey().getBytes());
+  }
+
+  @ProcessElement
+  public void processElement(ProcessContext c, ByteKeyRangeTracker tracker) 
throws Exception {
+final HBaseQuery query = c.element();
+TableName tableName = TableName.valueOf(query.getTableId());
+Table table = connection.getTable(tableName);
+final ByteKeyRange range = tracker.currentRestriction();
+try (ResultScanner scanner = 
table.getScanner(newScanInRange(query.getScan(), range))) {
+  for (Result result : scanner) {
+ByteKey key = ByteKey.copyFrom(result.getRow());
+if (!tracker.tryClaim(key)) {
+  return;
+}
+c.output(result);
+  }
+  tracker.markDone();
+}
+  }
+
+  @GetInitialRestriction
+  public ByteKeyRange getInitialRestriction(HBaseQuery query) {
+return ByteKeyRange.of(
+ByteKey.copyFrom(query.getScan().getStartRow()),
+ByteKey.copyFrom(query.getScan().getStopRow()));
+  }
+
+  @SplitRestriction
+  public void splitRestriction(
+  HBaseQuery query, ByteKeyRange range, OutputReceiver 
receiver)
+  throws Exception {
+// TODO remove once BEAM-4016 is fixed (lifecycle ignores calling setup 
before splitRestriction)
 
 Review comment:
   Since we seem to have a fix for BEAM-4016 will remove it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120073)
Time Spent: 4h 10m  (was: 4h)

> Add HBaseIO.readAll() based on SDF
> --
>
> Key: BEAM-4020
> URL: https://issues.apach

[jira] [Updated] (BEAM-4016) Direct runner incorrect lifecycle, @SplitRestriction should execute after @Setup on SplittableDoFn

2018-07-06 Thread JIRA



 [ 
https://issues.apache.org/jira/browse/BEAM-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-4016:
---
Affects Version/s: 2.5.0

> Direct runner incorrect lifecycle, @SplitRestriction should execute after 
> @Setup on SplittableDoFn
> --
>
> Key: BEAM-4016
> URL: https://issues.apache.org/jira/browse/BEAM-4016
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 2.4.0, 2.5.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Major
> Attachments: sdf-splitrestriction-lifeycle-test.patch
>
>
> The method annotated with @SplitRestriction is the method where we can define 
> the RestrictionTrackers (splits) in advance in a SDF. It makes sense to 
> execute this after the @Setup method given that usually connections are 
> established at Setup and can be used to ask the different data stores about 
> the partitioning strategy. I added a test for this in the 
> SplittableDoFnTest.SDFWithLifecycle test.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4736) Support reading decimal from BigQuery in Beam SQL

2018-07-06 Thread Rui Wang (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang updated BEAM-4736:
---
Description: 
Right now BigQueryIO returns avro format data and Beam SQL parses it and 
convert to Row. More specifically, decimal value is saved into HeapByteBuffer

 

Need to investigate a right way to convert byte array from byte buffer to 
BigDecimal.

  was:
Right now BigQueryIO returns avro format data and Beam SQL parses it and 
convert to Row. More specifically, decimal value is saved into HeapByteBuffer.

 

Need to investigate a right way to convert byte array from byte buffer to 
BigDecimal.


> Support reading decimal from BigQuery in Beam SQL
> -
>
> Key: BEAM-4736
> URL: https://issues.apache.org/jira/browse/BEAM-4736
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>
> Right now BigQueryIO returns avro format data and Beam SQL parses it and 
> convert to Row. More specifically, decimal value is saved into HeapByteBuffer
>  
> Need to investigate a right way to convert byte array from byte buffer to 
> BigDecimal.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4016) Direct runner incorrect lifecycle, @SplitRestriction should execute after @Setup on SplittableDoFn

2018-07-06 Thread JIRA



 [ 
https://issues.apache.org/jira/browse/BEAM-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-4016:
--

Assignee: Ismaël Mejía

> Direct runner incorrect lifecycle, @SplitRestriction should execute after 
> @Setup on SplittableDoFn
> --
>
> Key: BEAM-4016
> URL: https://issues.apache.org/jira/browse/BEAM-4016
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 2.4.0, 2.5.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Major
> Attachments: sdf-splitrestriction-lifeycle-test.patch
>
>
> The method annotated with @SplitRestriction is the method where we can define 
> the RestrictionTrackers (splits) in advance in a SDF. It makes sense to 
> execute this after the @Setup method given that usually connections are 
> established at Setup and can be used to ask the different data stores about 
> the partitioning strategy. I added a test for this in the 
> SplittableDoFnTest.SDFWithLifecycle test.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4736) Support reading decimal from BigQuery in Beam SQL

2018-07-06 Thread Rui Wang (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang updated BEAM-4736:
---
Description: 
Right now BigQueryIO returns avro format data and Beam SQL parses it and 
convert to Row. More specifically, decimal value is saved into HeapByteBuffer

 

Need to investigate what's the way to convert from byte buffer to BigDecimal.

  was:
Right now BigQueryIO returns avro format data and Beam SQL parses it and 
convert to Row. More specifically, decimal value is saved into HeapByteBuffer

 

Need to investigate a right way to convert byte array from byte buffer to 
BigDecimal.


> Support reading decimal from BigQuery in Beam SQL
> -
>
> Key: BEAM-4736
> URL: https://issues.apache.org/jira/browse/BEAM-4736
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>
> Right now BigQueryIO returns avro format data and Beam SQL parses it and 
> convert to Row. More specifically, decimal value is saved into HeapByteBuffer
>  
> Need to investigate what's the way to convert from byte buffer to BigDecimal.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120072&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120072
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:20
Start Date: 06/Jul/18 21:20
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200722785
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ByteBuddyUtils.java
 ##
 @@ -0,0 +1,563 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.lang.reflect.ParameterizedType;
+import java.lang.reflect.Type;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import net.bytebuddy.ByteBuddy;
+import net.bytebuddy.description.type.TypeDescription;
+import net.bytebuddy.description.type.TypeDescription.ForLoadedType;
+import net.bytebuddy.dynamic.DynamicType;
+import net.bytebuddy.implementation.FixedValue;
+import net.bytebuddy.implementation.Implementation;
+import net.bytebuddy.implementation.bytecode.Duplication;
+import net.bytebuddy.implementation.bytecode.StackManipulation;
+import net.bytebuddy.implementation.bytecode.StackManipulation.Compound;
+import net.bytebuddy.implementation.bytecode.TypeCreation;
+import net.bytebuddy.implementation.bytecode.assign.Assigner;
+import net.bytebuddy.implementation.bytecode.assign.Assigner.Typing;
+import net.bytebuddy.implementation.bytecode.assign.TypeCasting;
+import net.bytebuddy.implementation.bytecode.collection.ArrayFactory;
+import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeParameter;
+import org.apache.beam.sdk.values.reflect.FieldValueGetter;
+import org.apache.beam.sdk.values.reflect.FieldValueSetter;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.commons.lang3.ClassUtils;
+import org.joda.time.DateTime;
+import org.joda.time.ReadableDateTime;
+import org.joda.time.ReadableInstant;
+
+class ByteBuddyUtils {
+  private static final ForLoadedType ARRAYS_TYPE = new 
ForLoadedType(Arrays.class);
+  private static final ForLoadedType ARRAY_UTILS_TYPE = new 
ForLoadedType(ArrayUtils.class);
+  private static final ForLoadedType BYTE_ARRAY_TYPE = new 
ForLoadedType(byte[].class);
+  private static final ForLoadedType BYTE_BUFFER_TYPE = new 
ForLoadedType(ByteBuffer.class);
+  private static final ForLoadedType CHAR_SEQUENCE_TYPE = new 
ForLoadedType(CharSequence.class);
+  private static final ForLoadedType DATE_TIME_TYPE = new 
ForLoadedType(DateTime.class);
+  private static final ForLoadedType LIST_TYPE = new ForLoadedType(List.class);
+  private static final ForLoadedType READABLE_INSTANT_TYPE =
+  new ForLoadedType(ReadableInstant.class);
+
+  // Create a new FieldValueGetter subclass.
+  @SuppressWarnings("unchecked")
+  static DynamicType.Builder subclassGetterInterface(
+  ByteBuddy byteBuddy, Type objectType, Type fieldType) {
+TypeDescription.Generic getterGenericType =
+TypeDescription.Generic.Builder.parameterizedType(
+FieldValueGetter.class, objectType, fieldType)
+.build();
+return (DynamicType.Builder) 
byteBuddy.subclass(getterGenericType);
+  }
+
+  // Create a new FieldValueSetter subclass.
+  @SuppressWarnings("unchecked")
+  static DynamicType.Builder subclassSetterInterface(
+  ByteBuddy byteBuddy, Type objectType, Type fieldType) {
+TypeDescription.Generic setterGenericType =
+TypeDescription.Generic.Builder.parameterizedType(
+FieldValueSetter.class, objectType, fieldType)
+.build();
+return

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120068&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120068
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200757953
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ReflectUtils.java
 ##
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.lang.reflect.Method;
+import java.lang.reflect.Modifier;
+import java.util.Arrays;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.schemas.Schema;
+
+/** A set of reflection helper methods. */
+public class ReflectUtils {
+  static class ClassWithSchema {
+Class clazz;
+Schema schema;
+
+public ClassWithSchema(Class clazz, Schema schema) {
+  this.clazz = clazz;
+  this.schema = schema;
+}
+
+@Override
+public boolean equals(Object o) {
+  if (this == o) {
+return true;
+  }
+  if (o == null || getClass() != o.getClass()) {
+return false;
+  }
+  ClassWithSchema that = (ClassWithSchema) o;
+  return Objects.equals(clazz, that.clazz) && Objects.equals(schema, 
that.schema);
+}
+
+@Override
+public int hashCode() {
+  return Objects.hash(clazz, schema);
+}
+  }
+
+  private static final Map> DECLARED_METHODS = 
Maps.newHashMap();
+  private static final Map> DECLARED_FIELDS = 
Maps.newHashMap();
+
+  /** Returns the list of public, non-static methods in the class, caching the 
results. */
+  static List getMethods(Class clazz) throws IOException {
+return DECLARED_METHODS.computeIfAbsent(
+clazz,
+c -> {
+  return Arrays.stream(c.getDeclaredMethods())
+  .filter(m -> Modifier.isPublic(m.getModifiers()))
+  .filter(m -> !Modifier.isStatic(m.getModifiers()))
+  .collect(Collectors.toList());
+});
+  }
+
+  // Get all public, non-static, non-transient fields.
+  static List getFields(Class clazz) {
+return DECLARED_FIELDS.computeIfAbsent(
+clazz,
+c -> {
+  Map types = new LinkedHashMap<>();
+  do {
+if (c.getPackage() != null && 
c.getPackage().getName().startsWith("java.")) {
+  break; // skip java built-in classes
+}
+for (java.lang.reflect.Field field : c.getDeclaredFields()) {
+  if ((field.getModifiers() & (Modifier.TRANSIENT | 
Modifier.STATIC)) == 0) {
+if ((field.getModifiers() & Modifier.PUBLIC) != 0) {
+  boolean nullable = field.getAnnotation(Nullable.class) != 
null;
+  checkArgument(
+  types.put(field.getName(), field) == null,
+  c.getSimpleName() + " contains two fields named: " + 
field);
+}
+  }
+}
+c = c.getSuperclass();
+  } while (c != null);
+  return Lists.newArrayList(types.values());
+});
+  }
+
+  static boolean isGetter(Method method) {
+if (Void.TYPE.equals(method.getReturnType())) {
 
 Review comment:
   Should this check include `method.getParameterCount() != 0`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For que

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120070&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120070
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200759781
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/StaticSchemaInference.java
 ##
 @@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.utils;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.collect.ImmutableMap;
+import java.lang.reflect.Field;
+import java.lang.reflect.Method;
+import java.lang.reflect.ParameterizedType;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.joda.time.ReadableInstant;
+
+/** A set of utilities for inferring a Beam {@link Schema} from static Java 
types. */
+public class StaticSchemaInference {
+  enum MethodType {
+GETTER,
+SETTER
+  }
+
+  private static final Map PRIMITIVE_TYPES =
+  ImmutableMap.builder()
+  .put(Byte.class, FieldType.BYTE)
+  .put(byte.class, FieldType.BYTE)
+  .put(Short.class, FieldType.INT16)
+  .put(short.class, FieldType.INT16)
+  .put(Integer.class, FieldType.INT32)
+  .put(int.class, FieldType.INT32)
+  .put(Long.class, FieldType.INT64)
+  .put(long.class, FieldType.INT64)
+  .put(Float.class, FieldType.FLOAT)
+  .put(float.class, FieldType.FLOAT)
+  .put(Double.class, FieldType.DOUBLE)
+  .put(double.class, FieldType.DOUBLE)
+  .put(Boolean.class, FieldType.BOOLEAN)
+  .put(boolean.class, FieldType.BOOLEAN)
+  .put(BigDecimal.class, FieldType.DECIMAL)
+  .build();
+
+  /** Relevant information about a Java type. */
+  public static class TypeInformation {
+private String name;
+private TypeDescriptor type;
+private boolean nullable;
+
+/** Construct a {@link TypeInformation}. */
+public TypeInformation(String name, TypeDescriptor type, boolean nullable) 
{
+  this.name = name;
+  this.type = type;
+  this.nullable = nullable;
+}
+
+/** Construct a {@link TypeInformation} from a class member variable. */
+public TypeInformation(Field field) {
+  this(
+  field.getName(),
+  TypeDescriptor.of(field.getGenericType()),
+  field.getAnnotation(Nullable.class) != null);
+}
+
+/** Construct a {@link TypeInformation} from a class getter or setter. */
+public TypeInformation(Method method, MethodType methodType) {
 
 Review comment:
   Should this be a Factory Method instead of a constructor?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120070)
Time Spent: 3h 40m  (was: 3.5h)

> Provide automatic schema registration for POJOs
> ---
>
> Key: BEAM-4453
> URL: https://issues.apache.org/jira/browse/BEAM-4453
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: R

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120057&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120057
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200754840
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/JavaBeanUtils.java
 ##
 @@ -0,0 +1,335 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import net.bytebuddy.ByteBuddy;
+import net.bytebuddy.description.method.MethodDescription.ForLoadedMethod;
+import net.bytebuddy.dynamic.DynamicType;
+import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import net.bytebuddy.implementation.FixedValue;
+import net.bytebuddy.implementation.Implementation;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender.Size;
+import net.bytebuddy.implementation.bytecode.StackManipulation;
+import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import net.bytebuddy.implementation.bytecode.member.MethodReturn;
+import net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
+import net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertType;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertValueForGetter;
+import org.apache.beam.sdk.schemas.utils.ReflectUtils.ClassWithSchema;
+import org.apache.beam.sdk.schemas.utils.StaticSchemaInference.MethodType;
+import org.apache.beam.sdk.schemas.utils.StaticSchemaInference.TypeInformation;
+import org.apache.beam.sdk.values.reflect.FieldValueGetter;
+import org.apache.beam.sdk.values.reflect.FieldValueSetter;
+
+/** A set of utilities yo generate getter and setter classes for JavaBean 
objects. */
+@Experimental(Kind.SCHEMAS)
+public class JavaBeanUtils {
+  /** Create a {@link Schema} for a Java Bean class. */
+  public static Schema schemaFromJavaBeanClass(Class clazz) {
+return StaticSchemaInference.schemaFromClass(clazz, 
JavaBeanUtils::typeInformationFromClass);
+  }
+
+  private static List typeInformationFromClass(Class 
clazz) {
+try {
+  List getterTypes =
+  ReflectUtils.getMethods(clazz)
+  .stream()
+  .filter(ReflectUtils::isGetter)
+  .map(m -> new TypeInformation(m, MethodType.GETTER))
+  .collect(Collectors.toList());
+
+  Map setterTypes =
+  ReflectUtils.getMethods(clazz)
+  .stream()
+  .filter(ReflectUtils::isSetter)
+  .map(m -> new TypeInformation(m, MethodType.SETTER))
+  .collect(Collectors.toMap(TypeInformation::getName, 
Function.identity()));
+  validateJavaBean(getterTypes, setterTypes);
+  return getterTypes;
+} catch (IOException e) {
+  throw new RuntimeException(e);
+}
+  }
+
+  // Make sure that there are matching setters and getters.
+  private static void validateJavaBean(
+  List getters, Map setters) {
+for (TypeInformation type : getters) {
+  TypeInformation setterType = setters.get(type.getName());
+  if (setterType == null) {
+throw new RuntimeException(
+"JavaBean contained a setter for field "
++ type.getName()

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120054&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120054
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200739672
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/JavaBeanUtils.java
 ##
 @@ -0,0 +1,335 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import net.bytebuddy.ByteBuddy;
+import net.bytebuddy.description.method.MethodDescription.ForLoadedMethod;
+import net.bytebuddy.dynamic.DynamicType;
+import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import net.bytebuddy.implementation.FixedValue;
+import net.bytebuddy.implementation.Implementation;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender.Size;
+import net.bytebuddy.implementation.bytecode.StackManipulation;
+import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import net.bytebuddy.implementation.bytecode.member.MethodReturn;
+import net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
+import net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertType;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertValueForGetter;
+import org.apache.beam.sdk.schemas.utils.ReflectUtils.ClassWithSchema;
+import org.apache.beam.sdk.schemas.utils.StaticSchemaInference.MethodType;
+import org.apache.beam.sdk.schemas.utils.StaticSchemaInference.TypeInformation;
+import org.apache.beam.sdk.values.reflect.FieldValueGetter;
+import org.apache.beam.sdk.values.reflect.FieldValueSetter;
+
+/** A set of utilities yo generate getter and setter classes for JavaBean 
objects. */
+@Experimental(Kind.SCHEMAS)
+public class JavaBeanUtils {
+  /** Create a {@link Schema} for a Java Bean class. */
+  public static Schema schemaFromJavaBeanClass(Class clazz) {
+return StaticSchemaInference.schemaFromClass(clazz, 
JavaBeanUtils::typeInformationFromClass);
+  }
+
+  private static List typeInformationFromClass(Class 
clazz) {
+try {
+  List getterTypes =
+  ReflectUtils.getMethods(clazz)
+  .stream()
+  .filter(ReflectUtils::isGetter)
+  .map(m -> new TypeInformation(m, MethodType.GETTER))
+  .collect(Collectors.toList());
+
+  Map setterTypes =
+  ReflectUtils.getMethods(clazz)
+  .stream()
+  .filter(ReflectUtils::isSetter)
+  .map(m -> new TypeInformation(m, MethodType.SETTER))
+  .collect(Collectors.toMap(TypeInformation::getName, 
Function.identity()));
+  validateJavaBean(getterTypes, setterTypes);
+  return getterTypes;
+} catch (IOException e) {
+  throw new RuntimeException(e);
+}
+  }
+
+  // Make sure that there are matching setters and getters.
+  private static void validateJavaBean(
+  List getters, Map setters) {
+for (TypeInformation type : getters) {
+  TypeInformation setterType = setters.get(type.getName());
+  if (setterType == null) {
+throw new RuntimeException(
+"JavaBean contained a setter for field "
++ type.getName()

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120043&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120043
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200728080
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ByteBuddyUtils.java
 ##
 @@ -0,0 +1,563 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.lang.reflect.ParameterizedType;
+import java.lang.reflect.Type;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import net.bytebuddy.ByteBuddy;
+import net.bytebuddy.description.type.TypeDescription;
+import net.bytebuddy.description.type.TypeDescription.ForLoadedType;
+import net.bytebuddy.dynamic.DynamicType;
+import net.bytebuddy.implementation.FixedValue;
+import net.bytebuddy.implementation.Implementation;
+import net.bytebuddy.implementation.bytecode.Duplication;
+import net.bytebuddy.implementation.bytecode.StackManipulation;
+import net.bytebuddy.implementation.bytecode.StackManipulation.Compound;
+import net.bytebuddy.implementation.bytecode.TypeCreation;
+import net.bytebuddy.implementation.bytecode.assign.Assigner;
+import net.bytebuddy.implementation.bytecode.assign.Assigner.Typing;
+import net.bytebuddy.implementation.bytecode.assign.TypeCasting;
+import net.bytebuddy.implementation.bytecode.collection.ArrayFactory;
+import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeParameter;
+import org.apache.beam.sdk.values.reflect.FieldValueGetter;
+import org.apache.beam.sdk.values.reflect.FieldValueSetter;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.commons.lang3.ClassUtils;
+import org.joda.time.DateTime;
+import org.joda.time.ReadableDateTime;
+import org.joda.time.ReadableInstant;
+
+class ByteBuddyUtils {
+  private static final ForLoadedType ARRAYS_TYPE = new 
ForLoadedType(Arrays.class);
+  private static final ForLoadedType ARRAY_UTILS_TYPE = new 
ForLoadedType(ArrayUtils.class);
+  private static final ForLoadedType BYTE_ARRAY_TYPE = new 
ForLoadedType(byte[].class);
+  private static final ForLoadedType BYTE_BUFFER_TYPE = new 
ForLoadedType(ByteBuffer.class);
+  private static final ForLoadedType CHAR_SEQUENCE_TYPE = new 
ForLoadedType(CharSequence.class);
+  private static final ForLoadedType DATE_TIME_TYPE = new 
ForLoadedType(DateTime.class);
+  private static final ForLoadedType LIST_TYPE = new ForLoadedType(List.class);
+  private static final ForLoadedType READABLE_INSTANT_TYPE =
+  new ForLoadedType(ReadableInstant.class);
+
+  // Create a new FieldValueGetter subclass.
+  @SuppressWarnings("unchecked")
+  static DynamicType.Builder subclassGetterInterface(
+  ByteBuddy byteBuddy, Type objectType, Type fieldType) {
+TypeDescription.Generic getterGenericType =
+TypeDescription.Generic.Builder.parameterizedType(
+FieldValueGetter.class, objectType, fieldType)
+.build();
+return (DynamicType.Builder) 
byteBuddy.subclass(getterGenericType);
+  }
+
+  // Create a new FieldValueSetter subclass.
+  @SuppressWarnings("unchecked")
+  static DynamicType.Builder subclassSetterInterface(
+  ByteBuddy byteBuddy, Type objectType, Type fieldType) {
+TypeDescription.Generic setterGenericType =
+TypeDescription.Generic.Builder.parameterizedType(
+FieldValueSetter.class, objectType, fieldType)
+.build();
+return

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120055&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120055
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200758970
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ReflectUtils.java
 ##
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.lang.reflect.Method;
+import java.lang.reflect.Modifier;
+import java.util.Arrays;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.schemas.Schema;
+
+/** A set of reflection helper methods. */
+public class ReflectUtils {
+  static class ClassWithSchema {
+Class clazz;
+Schema schema;
+
+public ClassWithSchema(Class clazz, Schema schema) {
+  this.clazz = clazz;
+  this.schema = schema;
+}
+
+@Override
+public boolean equals(Object o) {
+  if (this == o) {
+return true;
+  }
+  if (o == null || getClass() != o.getClass()) {
+return false;
+  }
+  ClassWithSchema that = (ClassWithSchema) o;
+  return Objects.equals(clazz, that.clazz) && Objects.equals(schema, 
that.schema);
+}
+
+@Override
+public int hashCode() {
+  return Objects.hash(clazz, schema);
+}
+  }
+
+  private static final Map> DECLARED_METHODS = 
Maps.newHashMap();
+  private static final Map> DECLARED_FIELDS = 
Maps.newHashMap();
+
+  /** Returns the list of public, non-static methods in the class, caching the 
results. */
+  static List getMethods(Class clazz) throws IOException {
+return DECLARED_METHODS.computeIfAbsent(
+clazz,
+c -> {
+  return Arrays.stream(c.getDeclaredMethods())
+  .filter(m -> Modifier.isPublic(m.getModifiers()))
+  .filter(m -> !Modifier.isStatic(m.getModifiers()))
+  .collect(Collectors.toList());
+});
+  }
+
+  // Get all public, non-static, non-transient fields.
+  static List getFields(Class clazz) {
+return DECLARED_FIELDS.computeIfAbsent(
+clazz,
+c -> {
+  Map types = new LinkedHashMap<>();
+  do {
+if (c.getPackage() != null && 
c.getPackage().getName().startsWith("java.")) {
+  break; // skip java built-in classes
+}
+for (java.lang.reflect.Field field : c.getDeclaredFields()) {
+  if ((field.getModifiers() & (Modifier.TRANSIENT | 
Modifier.STATIC)) == 0) {
+if ((field.getModifiers() & Modifier.PUBLIC) != 0) {
+  boolean nullable = field.getAnnotation(Nullable.class) != 
null;
+  checkArgument(
+  types.put(field.getName(), field) == null,
+  c.getSimpleName() + " contains two fields named: " + 
field);
+}
+  }
+}
+c = c.getSuperclass();
+  } while (c != null);
+  return Lists.newArrayList(types.values());
+});
+  }
+
+  static boolean isGetter(Method method) {
+if (Void.TYPE.equals(method.getReturnType())) {
+  return false;
+}
+if (method.getName().startsWith("get") && method.getName().length() > 3) {
+  return true;
+}
+return (method.getName().startsWith("is")
+&& method.getName().length() > 2
+&& (Boolean.TYPE.equals(method.getReturnType())
+|| Boolean.class.equals

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120042&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120042
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200726955
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ByteBuddyUtils.java
 ##
 @@ -0,0 +1,563 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.lang.reflect.ParameterizedType;
+import java.lang.reflect.Type;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import net.bytebuddy.ByteBuddy;
+import net.bytebuddy.description.type.TypeDescription;
+import net.bytebuddy.description.type.TypeDescription.ForLoadedType;
+import net.bytebuddy.dynamic.DynamicType;
+import net.bytebuddy.implementation.FixedValue;
+import net.bytebuddy.implementation.Implementation;
+import net.bytebuddy.implementation.bytecode.Duplication;
+import net.bytebuddy.implementation.bytecode.StackManipulation;
+import net.bytebuddy.implementation.bytecode.StackManipulation.Compound;
+import net.bytebuddy.implementation.bytecode.TypeCreation;
+import net.bytebuddy.implementation.bytecode.assign.Assigner;
+import net.bytebuddy.implementation.bytecode.assign.Assigner.Typing;
+import net.bytebuddy.implementation.bytecode.assign.TypeCasting;
+import net.bytebuddy.implementation.bytecode.collection.ArrayFactory;
+import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeParameter;
+import org.apache.beam.sdk.values.reflect.FieldValueGetter;
+import org.apache.beam.sdk.values.reflect.FieldValueSetter;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.commons.lang3.ClassUtils;
+import org.joda.time.DateTime;
+import org.joda.time.ReadableDateTime;
+import org.joda.time.ReadableInstant;
+
+class ByteBuddyUtils {
+  private static final ForLoadedType ARRAYS_TYPE = new 
ForLoadedType(Arrays.class);
+  private static final ForLoadedType ARRAY_UTILS_TYPE = new 
ForLoadedType(ArrayUtils.class);
+  private static final ForLoadedType BYTE_ARRAY_TYPE = new 
ForLoadedType(byte[].class);
+  private static final ForLoadedType BYTE_BUFFER_TYPE = new 
ForLoadedType(ByteBuffer.class);
+  private static final ForLoadedType CHAR_SEQUENCE_TYPE = new 
ForLoadedType(CharSequence.class);
+  private static final ForLoadedType DATE_TIME_TYPE = new 
ForLoadedType(DateTime.class);
+  private static final ForLoadedType LIST_TYPE = new ForLoadedType(List.class);
+  private static final ForLoadedType READABLE_INSTANT_TYPE =
+  new ForLoadedType(ReadableInstant.class);
+
+  // Create a new FieldValueGetter subclass.
+  @SuppressWarnings("unchecked")
+  static DynamicType.Builder subclassGetterInterface(
+  ByteBuddy byteBuddy, Type objectType, Type fieldType) {
+TypeDescription.Generic getterGenericType =
+TypeDescription.Generic.Builder.parameterizedType(
+FieldValueGetter.class, objectType, fieldType)
+.build();
+return (DynamicType.Builder) 
byteBuddy.subclass(getterGenericType);
+  }
+
+  // Create a new FieldValueSetter subclass.
+  @SuppressWarnings("unchecked")
+  static DynamicType.Builder subclassSetterInterface(
+  ByteBuddy byteBuddy, Type objectType, Type fieldType) {
+TypeDescription.Generic setterGenericType =
+TypeDescription.Generic.Builder.parameterizedType(
+FieldValueSetter.class, objectType, fieldType)
+.build();
+return

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120046&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120046
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200729058
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ByteBuddyUtils.java
 ##
 @@ -0,0 +1,563 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.lang.reflect.ParameterizedType;
+import java.lang.reflect.Type;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import net.bytebuddy.ByteBuddy;
+import net.bytebuddy.description.type.TypeDescription;
+import net.bytebuddy.description.type.TypeDescription.ForLoadedType;
+import net.bytebuddy.dynamic.DynamicType;
+import net.bytebuddy.implementation.FixedValue;
+import net.bytebuddy.implementation.Implementation;
+import net.bytebuddy.implementation.bytecode.Duplication;
+import net.bytebuddy.implementation.bytecode.StackManipulation;
+import net.bytebuddy.implementation.bytecode.StackManipulation.Compound;
+import net.bytebuddy.implementation.bytecode.TypeCreation;
+import net.bytebuddy.implementation.bytecode.assign.Assigner;
+import net.bytebuddy.implementation.bytecode.assign.Assigner.Typing;
+import net.bytebuddy.implementation.bytecode.assign.TypeCasting;
+import net.bytebuddy.implementation.bytecode.collection.ArrayFactory;
+import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeParameter;
+import org.apache.beam.sdk.values.reflect.FieldValueGetter;
+import org.apache.beam.sdk.values.reflect.FieldValueSetter;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.commons.lang3.ClassUtils;
+import org.joda.time.DateTime;
+import org.joda.time.ReadableDateTime;
+import org.joda.time.ReadableInstant;
+
+class ByteBuddyUtils {
+  private static final ForLoadedType ARRAYS_TYPE = new 
ForLoadedType(Arrays.class);
+  private static final ForLoadedType ARRAY_UTILS_TYPE = new 
ForLoadedType(ArrayUtils.class);
+  private static final ForLoadedType BYTE_ARRAY_TYPE = new 
ForLoadedType(byte[].class);
+  private static final ForLoadedType BYTE_BUFFER_TYPE = new 
ForLoadedType(ByteBuffer.class);
+  private static final ForLoadedType CHAR_SEQUENCE_TYPE = new 
ForLoadedType(CharSequence.class);
+  private static final ForLoadedType DATE_TIME_TYPE = new 
ForLoadedType(DateTime.class);
+  private static final ForLoadedType LIST_TYPE = new ForLoadedType(List.class);
+  private static final ForLoadedType READABLE_INSTANT_TYPE =
+  new ForLoadedType(ReadableInstant.class);
+
+  // Create a new FieldValueGetter subclass.
+  @SuppressWarnings("unchecked")
+  static DynamicType.Builder subclassGetterInterface(
+  ByteBuddy byteBuddy, Type objectType, Type fieldType) {
+TypeDescription.Generic getterGenericType =
+TypeDescription.Generic.Builder.parameterizedType(
+FieldValueGetter.class, objectType, fieldType)
+.build();
+return (DynamicType.Builder) 
byteBuddy.subclass(getterGenericType);
+  }
+
+  // Create a new FieldValueSetter subclass.
+  @SuppressWarnings("unchecked")
+  static DynamicType.Builder subclassSetterInterface(
+  ByteBuddy byteBuddy, Type objectType, Type fieldType) {
+TypeDescription.Generic setterGenericType =
+TypeDescription.Generic.Builder.parameterizedType(
+FieldValueSetter.class, objectType, fieldType)
+.build();
+return

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120052&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120052
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200759285
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/StaticSchemaInference.java
 ##
 @@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.utils;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.collect.ImmutableMap;
+import java.lang.reflect.Field;
+import java.lang.reflect.Method;
+import java.lang.reflect.ParameterizedType;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.joda.time.ReadableInstant;
+
+/** A set of utilities for inferring a Beam {@link Schema} from static Java 
types. */
+public class StaticSchemaInference {
+  enum MethodType {
+GETTER,
+SETTER
+  }
+
+  private static final Map PRIMITIVE_TYPES =
+  ImmutableMap.builder()
+  .put(Byte.class, FieldType.BYTE)
+  .put(byte.class, FieldType.BYTE)
+  .put(Short.class, FieldType.INT16)
+  .put(short.class, FieldType.INT16)
+  .put(Integer.class, FieldType.INT32)
+  .put(int.class, FieldType.INT32)
+  .put(Long.class, FieldType.INT64)
+  .put(long.class, FieldType.INT64)
+  .put(Float.class, FieldType.FLOAT)
+  .put(float.class, FieldType.FLOAT)
+  .put(Double.class, FieldType.DOUBLE)
+  .put(double.class, FieldType.DOUBLE)
+  .put(Boolean.class, FieldType.BOOLEAN)
+  .put(boolean.class, FieldType.BOOLEAN)
+  .put(BigDecimal.class, FieldType.DECIMAL)
+  .build();
+
+  /** Relevant information about a Java type. */
+  public static class TypeInformation {
+private String name;
 
 Review comment:
   `final` for these?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120052)
Time Spent: 2h 10m  (was: 2h)

> Provide automatic schema registration for POJOs
> ---
>
> Key: BEAM-4453
> URL: https://issues.apache.org/jira/browse/BEAM-4453
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120051&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120051
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200738661
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/JavaBeanUtils.java
 ##
 @@ -0,0 +1,335 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import net.bytebuddy.ByteBuddy;
+import net.bytebuddy.description.method.MethodDescription.ForLoadedMethod;
+import net.bytebuddy.dynamic.DynamicType;
+import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import net.bytebuddy.implementation.FixedValue;
+import net.bytebuddy.implementation.Implementation;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender.Size;
+import net.bytebuddy.implementation.bytecode.StackManipulation;
+import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import net.bytebuddy.implementation.bytecode.member.MethodReturn;
+import net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
+import net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertType;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertValueForGetter;
+import org.apache.beam.sdk.schemas.utils.ReflectUtils.ClassWithSchema;
+import org.apache.beam.sdk.schemas.utils.StaticSchemaInference.MethodType;
+import org.apache.beam.sdk.schemas.utils.StaticSchemaInference.TypeInformation;
+import org.apache.beam.sdk.values.reflect.FieldValueGetter;
+import org.apache.beam.sdk.values.reflect.FieldValueSetter;
+
+/** A set of utilities yo generate getter and setter classes for JavaBean 
objects. */
+@Experimental(Kind.SCHEMAS)
+public class JavaBeanUtils {
+  /** Create a {@link Schema} for a Java Bean class. */
+  public static Schema schemaFromJavaBeanClass(Class clazz) {
+return StaticSchemaInference.schemaFromClass(clazz, 
JavaBeanUtils::typeInformationFromClass);
+  }
+
+  private static List typeInformationFromClass(Class 
clazz) {
+try {
+  List getterTypes =
+  ReflectUtils.getMethods(clazz)
+  .stream()
+  .filter(ReflectUtils::isGetter)
+  .map(m -> new TypeInformation(m, MethodType.GETTER))
+  .collect(Collectors.toList());
+
+  Map setterTypes =
+  ReflectUtils.getMethods(clazz)
+  .stream()
+  .filter(ReflectUtils::isSetter)
+  .map(m -> new TypeInformation(m, MethodType.SETTER))
+  .collect(Collectors.toMap(TypeInformation::getName, 
Function.identity()));
+  validateJavaBean(getterTypes, setterTypes);
+  return getterTypes;
+} catch (IOException e) {
+  throw new RuntimeException(e);
+}
+  }
+
+  // Make sure that there are matching setters and getters.
+  private static void validateJavaBean(
+  List getters, Map setters) {
+for (TypeInformation type : getters) {
+  TypeInformation setterType = setters.get(type.getName());
+  if (setterType == null) {
+throw new RuntimeException(
+"JavaBean contained a setter for field "
 
 Review comment:
   This string

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120038&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120038
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r199864866
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/DefaultSchema.java
 ##
 @@ -83,10 +83,7 @@ private SchemaProvider getSchemaProvider(TypeDescriptor 
typeDescriptor) {
 Class providerClass = annotation.value();
 checkArgument(
 providerClass != null,
-"Type "
-+ type
-+ " has a @DefaultSchemaProvider annotation "
-+ " with a null argument.");
+"Type " + type + " has a @DefaultSchema annotation " + " with 
a null argument.");
 
 Review comment:
   Looks like spotless messed up your string. (Unnecessary `" + "` in the 
middle.)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120038)
Time Spent: 20m  (was: 10m)

> Provide automatic schema registration for POJOs
> ---
>
> Key: BEAM-4453
> URL: https://issues.apache.org/jira/browse/BEAM-4453
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120061&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120061
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200770146
 
 

 ##
 File path: sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
 ##
 @@ -197,7 +199,15 @@ public Byte getByte(int idx) {
   }
 
   /**
-   * Get a {@link TypeName#INT16} value by field index, {@link 
ClassCastException} is thrown if
+   * Get a {@link TypeName#BYTES} value by field index, {@link 
ClassCastException} is thrown if
+   * schema doesn't match.
+   */
+  public byte[] getBytes(int idx) {
+return getValue(idx);
+  }
+
+  /**
+   * Get a {@link TypeName#INT16 16} value by field index, {@link 
ClassCastException} is thrown if
 
 Review comment:
   Looks like there is an extra `16`? (This line should be unchanged.)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120061)
Time Spent: 2h 50m  (was: 2h 40m)

> Provide automatic schema registration for POJOs
> ---
>
> Key: BEAM-4453
> URL: https://issues.apache.org/jira/browse/BEAM-4453
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120059&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120059
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200757807
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ReflectUtils.java
 ##
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.lang.reflect.Method;
+import java.lang.reflect.Modifier;
+import java.util.Arrays;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.schemas.Schema;
+
+/** A set of reflection helper methods. */
+public class ReflectUtils {
+  static class ClassWithSchema {
+Class clazz;
+Schema schema;
+
+public ClassWithSchema(Class clazz, Schema schema) {
+  this.clazz = clazz;
+  this.schema = schema;
+}
+
+@Override
+public boolean equals(Object o) {
+  if (this == o) {
+return true;
+  }
+  if (o == null || getClass() != o.getClass()) {
+return false;
+  }
+  ClassWithSchema that = (ClassWithSchema) o;
+  return Objects.equals(clazz, that.clazz) && Objects.equals(schema, 
that.schema);
+}
+
+@Override
+public int hashCode() {
+  return Objects.hash(clazz, schema);
+}
+  }
+
+  private static final Map> DECLARED_METHODS = 
Maps.newHashMap();
+  private static final Map> DECLARED_FIELDS = 
Maps.newHashMap();
+
+  /** Returns the list of public, non-static methods in the class, caching the 
results. */
+  static List getMethods(Class clazz) throws IOException {
+return DECLARED_METHODS.computeIfAbsent(
+clazz,
+c -> {
+  return Arrays.stream(c.getDeclaredMethods())
+  .filter(m -> Modifier.isPublic(m.getModifiers()))
+  .filter(m -> !Modifier.isStatic(m.getModifiers()))
+  .collect(Collectors.toList());
+});
+  }
+
+  // Get all public, non-static, non-transient fields.
+  static List getFields(Class clazz) {
+return DECLARED_FIELDS.computeIfAbsent(
+clazz,
+c -> {
+  Map types = new LinkedHashMap<>();
+  do {
+if (c.getPackage() != null && 
c.getPackage().getName().startsWith("java.")) {
+  break; // skip java built-in classes
+}
+for (java.lang.reflect.Field field : c.getDeclaredFields()) {
+  if ((field.getModifiers() & (Modifier.TRANSIENT | 
Modifier.STATIC)) == 0) {
+if ((field.getModifiers() & Modifier.PUBLIC) != 0) {
+  boolean nullable = field.getAnnotation(Nullable.class) != 
null;
+  checkArgument(
+  types.put(field.getName(), field) == null,
+  c.getSimpleName() + " contains two fields named: " + 
field);
+}
+  }
+}
+c = c.getSuperclass();
+  } while (c != null);
+  return Lists.newArrayList(types.values());
+});
+  }
+
+  static boolean isGetter(Method method) {
+if (Void.TYPE.equals(method.getReturnType())) {
+  return false;
+}
+if (method.getName().startsWith("get") && method.getName().length() > 3) {
+  return true;
+}
+return (method.getName().startsWith("is")
+&& method.getName().length() > 2
+&& (Boolean.TYPE.equals(method.getReturnType())
+|| Boolean.class.equals

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120039&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120039
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r199974567
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/GetterBasedSchemaProvider.java
 ##
 @@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Type;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.schemas.Schema.TypeName;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.RowWithGetters;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.reflect.FieldValueGetterFactory;
+import org.apache.beam.sdk.values.reflect.FieldValueSetter;
+import org.apache.beam.sdk.values.reflect.FieldValueSetterFactory;
+
+/**
+ * A {@link SchemaProvider} base class that vends schemas and rows based on 
{@link
+ * org.apache.beam.sdk.values.reflect.FieldValueGetter}s.
+ */
+@Experimental(Kind.SCHEMAS)
+public abstract class GetterBasedSchemaProvider implements SchemaProvider {
+  /** Implementing class should override to return a getter factory. */
+  abstract FieldValueGetterFactory fieldValueGetterFactory();
+
+  /** Implementing class should override to return a setter factory. */
+  abstract FieldValueSetterFactory fieldValueSetterFactory();
+
+  @Override
+  public  SerializableFunction toRowFunction(TypeDescriptor 
typeDescriptor) {
+return o ->
+Row.withSchema(schemaFor(typeDescriptor))
+.withFieldValueGetters(fieldValueGetterFactory(), o)
+.build();
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public  SerializableFunction fromRowFunction(TypeDescriptor 
typeDescriptor) {
+return r -> {
+  if (r instanceof RowWithGetters) {
+// Efficient path: simply extract the underlying POJO instead of 
creating a new one.
+return (T) ((RowWithGetters) r).getGetterTarget();
+  } else {
+// Use the setters to copy values from the Row to a new instance of 
the class.
+return fromRow(r, (Class) typeDescriptor.getType());
+  }
+};
+  }
+
+  private  T fromRow(Row row, Class clazz) {
+T object;
+try {
+  object = clazz.getDeclaredConstructor().newInstance();
+} catch (NoSuchMethodException
+| IllegalAccessException
+| InvocationTargetException
+| InstantiationException e) {
+  throw new RuntimeException("Failed to instantiate object ", e);
+}
+
+List setters =
+fieldValueSetterFactory().createSetters(clazz, row.getSchema());
+checkState(
+setters.size() == row.getFieldCount(),
+"Did not have a matching number of setters and fields.");
+
+// Iterate over the row, and set (possibly recursively) each field in the 
underlying object
+// using the setter.
+Schema schema = row.getSchema();
 
 Review comment:
   You are also calling `row.getSchema()` a few lines up. If getSchema is 
expensive it makes sense to move this before `createSetters()` and use it there.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comm

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120053&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120053
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200754870
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/POJOUtils.java
 ##
 @@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import com.google.common.collect.Maps;
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import net.bytebuddy.ByteBuddy;
+import net.bytebuddy.description.field.FieldDescription.ForLoadedField;
+import net.bytebuddy.dynamic.DynamicType;
+import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import net.bytebuddy.implementation.FixedValue;
+import net.bytebuddy.implementation.Implementation;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender.Size;
+import net.bytebuddy.implementation.bytecode.StackManipulation;
+import net.bytebuddy.implementation.bytecode.member.FieldAccess;
+import net.bytebuddy.implementation.bytecode.member.MethodReturn;
+import net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
+import net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertType;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertValueForGetter;
+import org.apache.beam.sdk.schemas.utils.ReflectUtils.ClassWithSchema;
+import org.apache.beam.sdk.schemas.utils.StaticSchemaInference.TypeInformation;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.reflect.FieldValueGetter;
+import org.apache.beam.sdk.values.reflect.FieldValueSetter;
+
+/** A set of utilities yo generate getter and setter classes for POJOs. */
+@Experimental(Kind.SCHEMAS)
+public class POJOUtils {
+  public static Schema schemaFromPojoClass(Class clazz) {
+// We should cache the field order.
+Function> getTypesForClass =
+c ->
+ReflectUtils.getFields(c)
+.stream()
+.map(TypeInformation::new)
+.collect(Collectors.toList());
+return StaticSchemaInference.schemaFromClass(clazz, getTypesForClass);
+  }
+
+  // Static ByteBuddy instance used by all helpers.
+  private static final ByteBuddy BYTE_BUDDY = new ByteBuddy();
+
+  // The list of getters for a class is cached, so we only create the classes 
the first time
+  // getSetters is called.
+  public static final Map> 
CACHED_GETTERS =
+  Maps.newConcurrentMap();
+
+  public static List getGetters(Class clazz, Schema 
schema) {
+// Return the getters ordered by their position in the schema.
+return CACHED_GETTERS.computeIfAbsent(
+new ClassWithSchema(clazz, schema),
+c -> {
+  Map getterMap =
+  ReflectUtils.getFields(clazz)
+  .stream()
+  .map(POJOUtils::createGetter)
+  .collect(Collectors.toMap(FieldValueGetter::name, 
Function.identity()));
+  return schema
+  .getFields()
+  .stream()
+  .map(f -> getterMap.get(f.getName()))
+  .collect(Collectors.toList());
+});
+  }
+
+  /**
+   * Generate the following {@link FieldValueSetter} class for the {@link 
Field}.
+   *
+   * 
+   *   class Getter implements {@literal FieldValueG

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120048&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120048
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200729565
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/JavaBeanUtils.java
 ##
 @@ -0,0 +1,335 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import net.bytebuddy.ByteBuddy;
+import net.bytebuddy.description.method.MethodDescription.ForLoadedMethod;
+import net.bytebuddy.dynamic.DynamicType;
+import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import net.bytebuddy.implementation.FixedValue;
+import net.bytebuddy.implementation.Implementation;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender.Size;
+import net.bytebuddy.implementation.bytecode.StackManipulation;
+import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import net.bytebuddy.implementation.bytecode.member.MethodReturn;
+import net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
+import net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertType;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertValueForGetter;
+import org.apache.beam.sdk.schemas.utils.ReflectUtils.ClassWithSchema;
+import org.apache.beam.sdk.schemas.utils.StaticSchemaInference.MethodType;
+import org.apache.beam.sdk.schemas.utils.StaticSchemaInference.TypeInformation;
+import org.apache.beam.sdk.values.reflect.FieldValueGetter;
+import org.apache.beam.sdk.values.reflect.FieldValueSetter;
+
+/** A set of utilities yo generate getter and setter classes for JavaBean 
objects. */
 
 Review comment:
   s/yo/to/


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120048)

> Provide automatic schema registration for POJOs
> ---
>
> Key: BEAM-4453
> URL: https://issues.apache.org/jira/browse/BEAM-4453
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120049&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120049
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200754028
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/JavaBeanUtils.java
 ##
 @@ -0,0 +1,335 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import net.bytebuddy.ByteBuddy;
+import net.bytebuddy.description.method.MethodDescription.ForLoadedMethod;
+import net.bytebuddy.dynamic.DynamicType;
+import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import net.bytebuddy.implementation.FixedValue;
+import net.bytebuddy.implementation.Implementation;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender.Size;
+import net.bytebuddy.implementation.bytecode.StackManipulation;
+import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import net.bytebuddy.implementation.bytecode.member.MethodReturn;
+import net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
+import net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertType;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertValueForGetter;
+import org.apache.beam.sdk.schemas.utils.ReflectUtils.ClassWithSchema;
+import org.apache.beam.sdk.schemas.utils.StaticSchemaInference.MethodType;
+import org.apache.beam.sdk.schemas.utils.StaticSchemaInference.TypeInformation;
+import org.apache.beam.sdk.values.reflect.FieldValueGetter;
+import org.apache.beam.sdk.values.reflect.FieldValueSetter;
+
+/** A set of utilities yo generate getter and setter classes for JavaBean 
objects. */
+@Experimental(Kind.SCHEMAS)
+public class JavaBeanUtils {
+  /** Create a {@link Schema} for a Java Bean class. */
+  public static Schema schemaFromJavaBeanClass(Class clazz) {
+return StaticSchemaInference.schemaFromClass(clazz, 
JavaBeanUtils::typeInformationFromClass);
+  }
+
+  private static List typeInformationFromClass(Class 
clazz) {
+try {
+  List getterTypes =
+  ReflectUtils.getMethods(clazz)
+  .stream()
+  .filter(ReflectUtils::isGetter)
+  .map(m -> new TypeInformation(m, MethodType.GETTER))
+  .collect(Collectors.toList());
+
+  Map setterTypes =
+  ReflectUtils.getMethods(clazz)
+  .stream()
+  .filter(ReflectUtils::isSetter)
+  .map(m -> new TypeInformation(m, MethodType.SETTER))
+  .collect(Collectors.toMap(TypeInformation::getName, 
Function.identity()));
+  validateJavaBean(getterTypes, setterTypes);
+  return getterTypes;
+} catch (IOException e) {
+  throw new RuntimeException(e);
+}
+  }
+
+  // Make sure that there are matching setters and getters.
+  private static void validateJavaBean(
+  List getters, Map setters) {
+for (TypeInformation type : getters) {
+  TypeInformation setterType = setters.get(type.getName());
+  if (setterType == null) {
+throw new RuntimeException(
+"JavaBean contained a setter for field "
++ type.getName()

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120060&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120060
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200760467
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/StaticSchemaInference.java
 ##
 @@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.utils;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.collect.ImmutableMap;
+import java.lang.reflect.Field;
+import java.lang.reflect.Method;
+import java.lang.reflect.ParameterizedType;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.joda.time.ReadableInstant;
+
+/** A set of utilities for inferring a Beam {@link Schema} from static Java 
types. */
+public class StaticSchemaInference {
+  enum MethodType {
+GETTER,
+SETTER
+  }
+
+  private static final Map PRIMITIVE_TYPES =
+  ImmutableMap.builder()
+  .put(Byte.class, FieldType.BYTE)
+  .put(byte.class, FieldType.BYTE)
+  .put(Short.class, FieldType.INT16)
+  .put(short.class, FieldType.INT16)
+  .put(Integer.class, FieldType.INT32)
+  .put(int.class, FieldType.INT32)
+  .put(Long.class, FieldType.INT64)
+  .put(long.class, FieldType.INT64)
+  .put(Float.class, FieldType.FLOAT)
+  .put(float.class, FieldType.FLOAT)
+  .put(Double.class, FieldType.DOUBLE)
+  .put(double.class, FieldType.DOUBLE)
+  .put(Boolean.class, FieldType.BOOLEAN)
+  .put(boolean.class, FieldType.BOOLEAN)
+  .put(BigDecimal.class, FieldType.DECIMAL)
+  .build();
+
+  /** Relevant information about a Java type. */
+  public static class TypeInformation {
+private String name;
+private TypeDescriptor type;
+private boolean nullable;
+
+/** Construct a {@link TypeInformation}. */
+public TypeInformation(String name, TypeDescriptor type, boolean nullable) 
{
+  this.name = name;
+  this.type = type;
+  this.nullable = nullable;
+}
+
+/** Construct a {@link TypeInformation} from a class member variable. */
+public TypeInformation(Field field) {
+  this(
+  field.getName(),
+  TypeDescriptor.of(field.getGenericType()),
+  field.getAnnotation(Nullable.class) != null);
+}
+
+/** Construct a {@link TypeInformation} from a class getter or setter. */
+public TypeInformation(Method method, MethodType methodType) {
+  switch (methodType) {
 
 Review comment:
   This code would be much simpler if you removed the `methodType` and had a 
different Factory Method for each type (`TypeInformation ofGetter(Method 
method)` and `TypeInformation ofSetter(Method method)`)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120060)
Time Spent: 2h 50m  (was: 2h 40m)

> Provide automatic schema registration for POJOs
> ---
>
> Key: BEAM-4453
> URL: https://issues.apache.org

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120066&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120066
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200771517
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowWithStorage.java
 ##
 @@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.values;
+
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.schemas.Schema;
+
+/** Concrete subclass of {@link Row} that explicitly stores all fields of the 
row. */
+public class RowWithStorage extends Row {
+  private List values;
 
 Review comment:
   `final`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120066)
Time Spent: 3h 10m  (was: 3h)

> Provide automatic schema registration for POJOs
> ---
>
> Key: BEAM-4453
> URL: https://issues.apache.org/jira/browse/BEAM-4453
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120044&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120044
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200728790
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ByteBuddyUtils.java
 ##
 @@ -0,0 +1,563 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.lang.reflect.ParameterizedType;
+import java.lang.reflect.Type;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import net.bytebuddy.ByteBuddy;
+import net.bytebuddy.description.type.TypeDescription;
+import net.bytebuddy.description.type.TypeDescription.ForLoadedType;
+import net.bytebuddy.dynamic.DynamicType;
+import net.bytebuddy.implementation.FixedValue;
+import net.bytebuddy.implementation.Implementation;
+import net.bytebuddy.implementation.bytecode.Duplication;
+import net.bytebuddy.implementation.bytecode.StackManipulation;
+import net.bytebuddy.implementation.bytecode.StackManipulation.Compound;
+import net.bytebuddy.implementation.bytecode.TypeCreation;
+import net.bytebuddy.implementation.bytecode.assign.Assigner;
+import net.bytebuddy.implementation.bytecode.assign.Assigner.Typing;
+import net.bytebuddy.implementation.bytecode.assign.TypeCasting;
+import net.bytebuddy.implementation.bytecode.collection.ArrayFactory;
+import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeParameter;
+import org.apache.beam.sdk.values.reflect.FieldValueGetter;
+import org.apache.beam.sdk.values.reflect.FieldValueSetter;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.commons.lang3.ClassUtils;
+import org.joda.time.DateTime;
+import org.joda.time.ReadableDateTime;
+import org.joda.time.ReadableInstant;
+
+class ByteBuddyUtils {
+  private static final ForLoadedType ARRAYS_TYPE = new 
ForLoadedType(Arrays.class);
+  private static final ForLoadedType ARRAY_UTILS_TYPE = new 
ForLoadedType(ArrayUtils.class);
+  private static final ForLoadedType BYTE_ARRAY_TYPE = new 
ForLoadedType(byte[].class);
+  private static final ForLoadedType BYTE_BUFFER_TYPE = new 
ForLoadedType(ByteBuffer.class);
+  private static final ForLoadedType CHAR_SEQUENCE_TYPE = new 
ForLoadedType(CharSequence.class);
+  private static final ForLoadedType DATE_TIME_TYPE = new 
ForLoadedType(DateTime.class);
+  private static final ForLoadedType LIST_TYPE = new ForLoadedType(List.class);
+  private static final ForLoadedType READABLE_INSTANT_TYPE =
+  new ForLoadedType(ReadableInstant.class);
+
+  // Create a new FieldValueGetter subclass.
+  @SuppressWarnings("unchecked")
+  static DynamicType.Builder subclassGetterInterface(
+  ByteBuddy byteBuddy, Type objectType, Type fieldType) {
+TypeDescription.Generic getterGenericType =
+TypeDescription.Generic.Builder.parameterizedType(
+FieldValueGetter.class, objectType, fieldType)
+.build();
+return (DynamicType.Builder) 
byteBuddy.subclass(getterGenericType);
+  }
+
+  // Create a new FieldValueSetter subclass.
+  @SuppressWarnings("unchecked")
+  static DynamicType.Builder subclassSetterInterface(
+  ByteBuddy byteBuddy, Type objectType, Type fieldType) {
+TypeDescription.Generic setterGenericType =
+TypeDescription.Generic.Builder.parameterizedType(
+FieldValueSetter.class, objectType, fieldType)
+.build();
+return

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120062&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120062
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200759658
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/StaticSchemaInference.java
 ##
 @@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.utils;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.collect.ImmutableMap;
+import java.lang.reflect.Field;
+import java.lang.reflect.Method;
+import java.lang.reflect.ParameterizedType;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.joda.time.ReadableInstant;
+
+/** A set of utilities for inferring a Beam {@link Schema} from static Java 
types. */
+public class StaticSchemaInference {
+  enum MethodType {
+GETTER,
+SETTER
+  }
+
+  private static final Map PRIMITIVE_TYPES =
+  ImmutableMap.builder()
+  .put(Byte.class, FieldType.BYTE)
+  .put(byte.class, FieldType.BYTE)
+  .put(Short.class, FieldType.INT16)
+  .put(short.class, FieldType.INT16)
+  .put(Integer.class, FieldType.INT32)
+  .put(int.class, FieldType.INT32)
+  .put(Long.class, FieldType.INT64)
+  .put(long.class, FieldType.INT64)
+  .put(Float.class, FieldType.FLOAT)
+  .put(float.class, FieldType.FLOAT)
+  .put(Double.class, FieldType.DOUBLE)
+  .put(double.class, FieldType.DOUBLE)
+  .put(Boolean.class, FieldType.BOOLEAN)
+  .put(boolean.class, FieldType.BOOLEAN)
+  .put(BigDecimal.class, FieldType.DECIMAL)
+  .build();
+
+  /** Relevant information about a Java type. */
+  public static class TypeInformation {
+private String name;
+private TypeDescriptor type;
+private boolean nullable;
+
+/** Construct a {@link TypeInformation}. */
+public TypeInformation(String name, TypeDescriptor type, boolean nullable) 
{
 
 Review comment:
   Should this be `private`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120062)

> Provide automatic schema registration for POJOs
> ---
>
> Key: BEAM-4453
> URL: https://issues.apache.org/jira/browse/BEAM-4453
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120063&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120063
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200756476
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ReflectUtils.java
 ##
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.lang.reflect.Method;
+import java.lang.reflect.Modifier;
+import java.util.Arrays;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.schemas.Schema;
+
+/** A set of reflection helper methods. */
+public class ReflectUtils {
+  static class ClassWithSchema {
+Class clazz;
 
 Review comment:
   should both of these be `private final`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120063)
Time Spent: 3h  (was: 2h 50m)

> Provide automatic schema registration for POJOs
> ---
>
> Key: BEAM-4453
> URL: https://issues.apache.org/jira/browse/BEAM-4453
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120045&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120045
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200728335
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ByteBuddyUtils.java
 ##
 @@ -0,0 +1,563 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.lang.reflect.ParameterizedType;
+import java.lang.reflect.Type;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import net.bytebuddy.ByteBuddy;
+import net.bytebuddy.description.type.TypeDescription;
+import net.bytebuddy.description.type.TypeDescription.ForLoadedType;
+import net.bytebuddy.dynamic.DynamicType;
+import net.bytebuddy.implementation.FixedValue;
+import net.bytebuddy.implementation.Implementation;
+import net.bytebuddy.implementation.bytecode.Duplication;
+import net.bytebuddy.implementation.bytecode.StackManipulation;
+import net.bytebuddy.implementation.bytecode.StackManipulation.Compound;
+import net.bytebuddy.implementation.bytecode.TypeCreation;
+import net.bytebuddy.implementation.bytecode.assign.Assigner;
+import net.bytebuddy.implementation.bytecode.assign.Assigner.Typing;
+import net.bytebuddy.implementation.bytecode.assign.TypeCasting;
+import net.bytebuddy.implementation.bytecode.collection.ArrayFactory;
+import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeParameter;
+import org.apache.beam.sdk.values.reflect.FieldValueGetter;
+import org.apache.beam.sdk.values.reflect.FieldValueSetter;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.commons.lang3.ClassUtils;
+import org.joda.time.DateTime;
+import org.joda.time.ReadableDateTime;
+import org.joda.time.ReadableInstant;
+
+class ByteBuddyUtils {
+  private static final ForLoadedType ARRAYS_TYPE = new 
ForLoadedType(Arrays.class);
+  private static final ForLoadedType ARRAY_UTILS_TYPE = new 
ForLoadedType(ArrayUtils.class);
+  private static final ForLoadedType BYTE_ARRAY_TYPE = new 
ForLoadedType(byte[].class);
+  private static final ForLoadedType BYTE_BUFFER_TYPE = new 
ForLoadedType(ByteBuffer.class);
+  private static final ForLoadedType CHAR_SEQUENCE_TYPE = new 
ForLoadedType(CharSequence.class);
+  private static final ForLoadedType DATE_TIME_TYPE = new 
ForLoadedType(DateTime.class);
+  private static final ForLoadedType LIST_TYPE = new ForLoadedType(List.class);
+  private static final ForLoadedType READABLE_INSTANT_TYPE =
+  new ForLoadedType(ReadableInstant.class);
+
+  // Create a new FieldValueGetter subclass.
+  @SuppressWarnings("unchecked")
+  static DynamicType.Builder subclassGetterInterface(
+  ByteBuddy byteBuddy, Type objectType, Type fieldType) {
+TypeDescription.Generic getterGenericType =
+TypeDescription.Generic.Builder.parameterizedType(
+FieldValueGetter.class, objectType, fieldType)
+.build();
+return (DynamicType.Builder) 
byteBuddy.subclass(getterGenericType);
+  }
+
+  // Create a new FieldValueSetter subclass.
+  @SuppressWarnings("unchecked")
+  static DynamicType.Builder subclassSetterInterface(
+  ByteBuddy byteBuddy, Type objectType, Type fieldType) {
+TypeDescription.Generic setterGenericType =
+TypeDescription.Generic.Builder.parameterizedType(
+FieldValueSetter.class, objectType, fieldType)
+.build();
+return

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120047&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120047
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200739701
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/JavaBeanUtils.java
 ##
 @@ -0,0 +1,335 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import net.bytebuddy.ByteBuddy;
+import net.bytebuddy.description.method.MethodDescription.ForLoadedMethod;
+import net.bytebuddy.dynamic.DynamicType;
+import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import net.bytebuddy.implementation.FixedValue;
+import net.bytebuddy.implementation.Implementation;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender.Size;
+import net.bytebuddy.implementation.bytecode.StackManipulation;
+import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import net.bytebuddy.implementation.bytecode.member.MethodReturn;
+import net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
+import net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertType;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertValueForGetter;
+import org.apache.beam.sdk.schemas.utils.ReflectUtils.ClassWithSchema;
+import org.apache.beam.sdk.schemas.utils.StaticSchemaInference.MethodType;
+import org.apache.beam.sdk.schemas.utils.StaticSchemaInference.TypeInformation;
+import org.apache.beam.sdk.values.reflect.FieldValueGetter;
+import org.apache.beam.sdk.values.reflect.FieldValueSetter;
+
+/** A set of utilities yo generate getter and setter classes for JavaBean 
objects. */
+@Experimental(Kind.SCHEMAS)
+public class JavaBeanUtils {
+  /** Create a {@link Schema} for a Java Bean class. */
+  public static Schema schemaFromJavaBeanClass(Class clazz) {
+return StaticSchemaInference.schemaFromClass(clazz, 
JavaBeanUtils::typeInformationFromClass);
+  }
+
+  private static List typeInformationFromClass(Class 
clazz) {
+try {
+  List getterTypes =
+  ReflectUtils.getMethods(clazz)
+  .stream()
+  .filter(ReflectUtils::isGetter)
+  .map(m -> new TypeInformation(m, MethodType.GETTER))
+  .collect(Collectors.toList());
+
+  Map setterTypes =
+  ReflectUtils.getMethods(clazz)
+  .stream()
+  .filter(ReflectUtils::isSetter)
+  .map(m -> new TypeInformation(m, MethodType.SETTER))
+  .collect(Collectors.toMap(TypeInformation::getName, 
Function.identity()));
+  validateJavaBean(getterTypes, setterTypes);
+  return getterTypes;
+} catch (IOException e) {
+  throw new RuntimeException(e);
+}
+  }
+
+  // Make sure that there are matching setters and getters.
+  private static void validateJavaBean(
+  List getters, Map setters) {
+for (TypeInformation type : getters) {
+  TypeInformation setterType = setters.get(type.getName());
+  if (setterType == null) {
+throw new RuntimeException(
+"JavaBean contained a setter for field "
++ type.getName()

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120041&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120041
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200722785
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ByteBuddyUtils.java
 ##
 @@ -0,0 +1,563 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.lang.reflect.ParameterizedType;
+import java.lang.reflect.Type;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import net.bytebuddy.ByteBuddy;
+import net.bytebuddy.description.type.TypeDescription;
+import net.bytebuddy.description.type.TypeDescription.ForLoadedType;
+import net.bytebuddy.dynamic.DynamicType;
+import net.bytebuddy.implementation.FixedValue;
+import net.bytebuddy.implementation.Implementation;
+import net.bytebuddy.implementation.bytecode.Duplication;
+import net.bytebuddy.implementation.bytecode.StackManipulation;
+import net.bytebuddy.implementation.bytecode.StackManipulation.Compound;
+import net.bytebuddy.implementation.bytecode.TypeCreation;
+import net.bytebuddy.implementation.bytecode.assign.Assigner;
+import net.bytebuddy.implementation.bytecode.assign.Assigner.Typing;
+import net.bytebuddy.implementation.bytecode.assign.TypeCasting;
+import net.bytebuddy.implementation.bytecode.collection.ArrayFactory;
+import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeParameter;
+import org.apache.beam.sdk.values.reflect.FieldValueGetter;
+import org.apache.beam.sdk.values.reflect.FieldValueSetter;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.commons.lang3.ClassUtils;
+import org.joda.time.DateTime;
+import org.joda.time.ReadableDateTime;
+import org.joda.time.ReadableInstant;
+
+class ByteBuddyUtils {
+  private static final ForLoadedType ARRAYS_TYPE = new 
ForLoadedType(Arrays.class);
+  private static final ForLoadedType ARRAY_UTILS_TYPE = new 
ForLoadedType(ArrayUtils.class);
+  private static final ForLoadedType BYTE_ARRAY_TYPE = new 
ForLoadedType(byte[].class);
+  private static final ForLoadedType BYTE_BUFFER_TYPE = new 
ForLoadedType(ByteBuffer.class);
+  private static final ForLoadedType CHAR_SEQUENCE_TYPE = new 
ForLoadedType(CharSequence.class);
+  private static final ForLoadedType DATE_TIME_TYPE = new 
ForLoadedType(DateTime.class);
+  private static final ForLoadedType LIST_TYPE = new ForLoadedType(List.class);
+  private static final ForLoadedType READABLE_INSTANT_TYPE =
+  new ForLoadedType(ReadableInstant.class);
+
+  // Create a new FieldValueGetter subclass.
+  @SuppressWarnings("unchecked")
+  static DynamicType.Builder subclassGetterInterface(
+  ByteBuddy byteBuddy, Type objectType, Type fieldType) {
+TypeDescription.Generic getterGenericType =
+TypeDescription.Generic.Builder.parameterizedType(
+FieldValueGetter.class, objectType, fieldType)
+.build();
+return (DynamicType.Builder) 
byteBuddy.subclass(getterGenericType);
+  }
+
+  // Create a new FieldValueSetter subclass.
+  @SuppressWarnings("unchecked")
+  static DynamicType.Builder subclassSetterInterface(
+  ByteBuddy byteBuddy, Type objectType, Type fieldType) {
+TypeDescription.Generic setterGenericType =
+TypeDescription.Generic.Builder.parameterizedType(
+FieldValueSetter.class, objectType, fieldType)
+.build();
+return

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120065&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120065
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200771130
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowWithGetters.java
 ##
 @@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.values;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.schemas.Schema.TypeName;
+import org.apache.beam.sdk.values.reflect.FieldValueGetter;
+import org.apache.beam.sdk.values.reflect.FieldValueGetterFactory;
+
+/**
+ * A Concrete subclass of {@link Row} that delegates to a set of provided 
{@link FieldValueGetter}s.
+ *
+ * This allows us to have {@link Row} objects for which the actual storage 
is in another object.
+ * For example, the user's type may be a POJO, in which case the provided 
getters will simple read
+ * the appropriate fields from the POJO.
+ */
+public class RowWithGetters extends Row {
+  private FieldValueGetterFactory fieldValueGetterFactory;
 
 Review comment:
   All these things appear to be `final`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120065)
Time Spent: 3h  (was: 2h 50m)

> Provide automatic schema registration for POJOs
> ---
>
> Key: BEAM-4453
> URL: https://issues.apache.org/jira/browse/BEAM-4453
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120071&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120071
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200769726
 
 

 ##
 File path: sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
 ##
 @@ -51,38 +54,29 @@
  * {see @link Schema#getRowCoder()}.
  */
 @Experimental
-@AutoValue
 public abstract class Row implements Serializable {
-  /** Creates a {@link Row} from the list of values and {@link #getSchema()}. 
*/
-  public static  Collector, Row> toRow(Schema schema) {
-return Collector.of(
-() -> new ArrayList<>(schema.getFieldCount()),
-List::add,
-(left, right) -> {
-  left.addAll(right);
-  return left;
-},
-values -> Row.withSchema(schema).addValues(values).build());
-  }
+  private Schema schema;
 
 Review comment:
   `final`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 120071)
Time Spent: 3h 40m  (was: 3.5h)

> Provide automatic schema registration for POJOs
> ---
>
> Key: BEAM-4453
> URL: https://issues.apache.org/jira/browse/BEAM-4453
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120058&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120058
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200754903
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/POJOUtils.java
 ##
 @@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import com.google.common.collect.Maps;
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import net.bytebuddy.ByteBuddy;
+import net.bytebuddy.description.field.FieldDescription.ForLoadedField;
+import net.bytebuddy.dynamic.DynamicType;
+import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import net.bytebuddy.implementation.FixedValue;
+import net.bytebuddy.implementation.Implementation;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender.Size;
+import net.bytebuddy.implementation.bytecode.StackManipulation;
+import net.bytebuddy.implementation.bytecode.member.FieldAccess;
+import net.bytebuddy.implementation.bytecode.member.MethodReturn;
+import net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
+import net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertType;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertValueForGetter;
+import org.apache.beam.sdk.schemas.utils.ReflectUtils.ClassWithSchema;
+import org.apache.beam.sdk.schemas.utils.StaticSchemaInference.TypeInformation;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.reflect.FieldValueGetter;
+import org.apache.beam.sdk.values.reflect.FieldValueSetter;
+
+/** A set of utilities yo generate getter and setter classes for POJOs. */
+@Experimental(Kind.SCHEMAS)
+public class POJOUtils {
+  public static Schema schemaFromPojoClass(Class clazz) {
+// We should cache the field order.
+Function> getTypesForClass =
+c ->
+ReflectUtils.getFields(c)
+.stream()
+.map(TypeInformation::new)
+.collect(Collectors.toList());
+return StaticSchemaInference.schemaFromClass(clazz, getTypesForClass);
+  }
+
+  // Static ByteBuddy instance used by all helpers.
+  private static final ByteBuddy BYTE_BUDDY = new ByteBuddy();
+
+  // The list of getters for a class is cached, so we only create the classes 
the first time
+  // getSetters is called.
+  public static final Map> 
CACHED_GETTERS =
+  Maps.newConcurrentMap();
+
+  public static List getGetters(Class clazz, Schema 
schema) {
+// Return the getters ordered by their position in the schema.
+return CACHED_GETTERS.computeIfAbsent(
+new ClassWithSchema(clazz, schema),
+c -> {
+  Map getterMap =
+  ReflectUtils.getFields(clazz)
+  .stream()
+  .map(POJOUtils::createGetter)
+  .collect(Collectors.toMap(FieldValueGetter::name, 
Function.identity()));
+  return schema
+  .getFields()
+  .stream()
+  .map(f -> getterMap.get(f.getName()))
+  .collect(Collectors.toList());
+});
+  }
+
+  /**
+   * Generate the following {@link FieldValueSetter} class for the {@link 
Field}.
+   *
+   * 
+   *   class Getter implements {@literal FieldValueG

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120050&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120050
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200738655
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/JavaBeanUtils.java
 ##
 @@ -0,0 +1,335 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import net.bytebuddy.ByteBuddy;
+import net.bytebuddy.description.method.MethodDescription.ForLoadedMethod;
+import net.bytebuddy.dynamic.DynamicType;
+import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import net.bytebuddy.implementation.FixedValue;
+import net.bytebuddy.implementation.Implementation;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender.Size;
+import net.bytebuddy.implementation.bytecode.StackManipulation;
+import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import net.bytebuddy.implementation.bytecode.member.MethodReturn;
+import net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
+import net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertType;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertValueForGetter;
+import org.apache.beam.sdk.schemas.utils.ReflectUtils.ClassWithSchema;
+import org.apache.beam.sdk.schemas.utils.StaticSchemaInference.MethodType;
+import org.apache.beam.sdk.schemas.utils.StaticSchemaInference.TypeInformation;
+import org.apache.beam.sdk.values.reflect.FieldValueGetter;
+import org.apache.beam.sdk.values.reflect.FieldValueSetter;
+
+/** A set of utilities yo generate getter and setter classes for JavaBean 
objects. */
+@Experimental(Kind.SCHEMAS)
+public class JavaBeanUtils {
+  /** Create a {@link Schema} for a Java Bean class. */
+  public static Schema schemaFromJavaBeanClass(Class clazz) {
+return StaticSchemaInference.schemaFromClass(clazz, 
JavaBeanUtils::typeInformationFromClass);
+  }
+
+  private static List typeInformationFromClass(Class 
clazz) {
+try {
+  List getterTypes =
+  ReflectUtils.getMethods(clazz)
+  .stream()
+  .filter(ReflectUtils::isGetter)
+  .map(m -> new TypeInformation(m, MethodType.GETTER))
+  .collect(Collectors.toList());
+
+  Map setterTypes =
+  ReflectUtils.getMethods(clazz)
+  .stream()
+  .filter(ReflectUtils::isSetter)
+  .map(m -> new TypeInformation(m, MethodType.SETTER))
+  .collect(Collectors.toMap(TypeInformation::getName, 
Function.identity()));
+  validateJavaBean(getterTypes, setterTypes);
+  return getterTypes;
+} catch (IOException e) {
+  throw new RuntimeException(e);
+}
+  }
+
+  // Make sure that there are matching setters and getters.
+  private static void validateJavaBean(
+  List getters, Map setters) {
+for (TypeInformation type : getters) {
 
 Review comment:
   This only validates that there is a setter for every getter. Should there be 
a check (possibly that counts match?) to ensure there aren't setters without 
matching getters?

-

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-06 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=120067&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-120067
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 06/Jul/18 21:19
Start Date: 06/Jul/18 21:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5873: 
[BEAM-4453] Add schema support for Java POJOs and Java Beans
URL: https://github.com/apache/beam/pull/5873#discussion_r200755007
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/JavaBeanUtils.java
 ##
 @@ -0,0 +1,335 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.utils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import net.bytebuddy.ByteBuddy;
+import net.bytebuddy.description.method.MethodDescription.ForLoadedMethod;
+import net.bytebuddy.dynamic.DynamicType;
+import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import net.bytebuddy.implementation.FixedValue;
+import net.bytebuddy.implementation.Implementation;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender.Size;
+import net.bytebuddy.implementation.bytecode.StackManipulation;
+import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import net.bytebuddy.implementation.bytecode.member.MethodReturn;
+import net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
+import net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertType;
+import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.ConvertValueForGetter;
+import org.apache.beam.sdk.schemas.utils.ReflectUtils.ClassWithSchema;
+import org.apache.beam.sdk.schemas.utils.StaticSchemaInference.MethodType;
+import org.apache.beam.sdk.schemas.utils.StaticSchemaInference.TypeInformation;
+import org.apache.beam.sdk.values.reflect.FieldValueGetter;
+import org.apache.beam.sdk.values.reflect.FieldValueSetter;
+
+/** A set of utilities yo generate getter and setter classes for JavaBean 
objects. */
+@Experimental(Kind.SCHEMAS)
+public class JavaBeanUtils {
+  /** Create a {@link Schema} for a Java Bean class. */
+  public static Schema schemaFromJavaBeanClass(Class clazz) {
+return StaticSchemaInference.schemaFromClass(clazz, 
JavaBeanUtils::typeInformationFromClass);
+  }
+
+  private static List typeInformationFromClass(Class 
clazz) {
+try {
+  List getterTypes =
+  ReflectUtils.getMethods(clazz)
+  .stream()
+  .filter(ReflectUtils::isGetter)
+  .map(m -> new TypeInformation(m, MethodType.GETTER))
+  .collect(Collectors.toList());
+
+  Map setterTypes =
+  ReflectUtils.getMethods(clazz)
+  .stream()
+  .filter(ReflectUtils::isSetter)
+  .map(m -> new TypeInformation(m, MethodType.SETTER))
+  .collect(Collectors.toMap(TypeInformation::getName, 
Function.identity()));
+  validateJavaBean(getterTypes, setterTypes);
+  return getterTypes;
+} catch (IOException e) {
+  throw new RuntimeException(e);
+}
+  }
+
+  // Make sure that there are matching setters and getters.
+  private static void validateJavaBean(
+  List getters, Map setters) {
+for (TypeInformation type : getters) {
+  TypeInformation setterType = setters.get(type.getName());
+  if (setterType == null) {
+throw new RuntimeException(
+"JavaBean contained a setter for field "
++ type.getName()

1 2 >

1 - 100 of 191 matches

Mail list logo