[jira] [Closed] (BEAM-2743) rename size() in BeamRecord

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2743.

   Resolution: Duplicate
Fix Version/s: Not applicable

> rename size() in BeamRecord
> ---
>
> Key: BEAM-2743
> URL: https://issues.apache.org/jira/browse/BEAM-2743
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>  Labels: dsl_sql_merge, dsl_sql_review
> Fix For: Not applicable
>
>
> refer to 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTrrtwc



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2742) Switch from primitives to Java types

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2742.

   Resolution: Fixed
Fix Version/s: Not applicable

> Switch from primitives to Java types
> 
>
> Key: BEAM-2742
> URL: https://issues.apache.org/jira/browse/BEAM-2742
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>  Labels: dsl_sql_merge, dsl_sql_review
> Fix For: Not applicable
>
>
> Refer to 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTrrtwc



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2730) make BeamRecord an immutable type

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2730.

   Resolution: Fixed
Fix Version/s: Not applicable

> make BeamRecord an immutable type
> -
>
> Key: BEAM-2730
> URL: https://issues.apache.org/jira/browse/BEAM-2730
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge, dsl_sql_review
> Fix For: Not applicable
>
>
> Refer to discussion 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTnbIjE



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2749) update BeamSqlExample

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119420#comment-16119420
 ] 

ASF GitHub Bot commented on BEAM-2749:
--

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/3706

[BEAM-2749] update BeamSqlExample

R: @takidau 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam BEAM-2749

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3706.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3706


commit f03cc869db3801b7913a87f7950e16241b877a5a
Author: mingmxu 
Date:   2017-08-09T04:52:54Z

update example.




> update BeamSqlExample
> -
>
> Key: BEAM-2749
> URL: https://issues.apache.org/jira/browse/BEAM-2749
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge, dsl_sql_review
>
> change usage to reflect the latest status, and add an example with GROUP_BY



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3706: [BEAM-2749] update BeamSqlExample

[XuMingmin]

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/3706

[BEAM-2749] update BeamSqlExample

R: @takidau 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam BEAM-2749

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3706.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3706


commit f03cc869db3801b7913a87f7950e16241b877a5a
Author: mingmxu 
Date:   2017-08-09T04:52:54Z

update example.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

svn commit: r20909 - in /dev/beam: ./ 2.1.0/

[jbonofre]

Author: jbonofre
Date: Wed Aug  9 04:59:42 2017
New Revision: 20909

Log:
Publish Apache Beam 2.1.0, RC3

Added:
dev/beam/latest   (with props)
Modified:
dev/beam/2.1.0/apache-beam-2.1.0-python.zip
dev/beam/2.1.0/apache-beam-2.1.0-python.zip.asc
dev/beam/2.1.0/apache-beam-2.1.0-python.zip.md5
dev/beam/2.1.0/apache-beam-2.1.0-python.zip.sha1
dev/beam/2.1.0/apache-beam-2.1.0-source-release.zip
dev/beam/2.1.0/apache-beam-2.1.0-source-release.zip.asc
dev/beam/2.1.0/apache-beam-2.1.0-source-release.zip.md5
dev/beam/2.1.0/apache-beam-2.1.0-source-release.zip.sha1

Modified: dev/beam/2.1.0/apache-beam-2.1.0-python.zip
==
Binary files - no diff available.

Modified: dev/beam/2.1.0/apache-beam-2.1.0-python.zip.asc
==
--- dev/beam/2.1.0/apache-beam-2.1.0-python.zip.asc (original)
+++ dev/beam/2.1.0/apache-beam-2.1.0-python.zip.asc Wed Aug  9 04:59:42 2017
@@ -1,17 +1,17 @@
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1
 
-iQIcBAABAgAGBQJZbjYeAAoJEL/y7kLIKC52PjAQAJv9zsWQgtwxT9vngrj9c0D/
-jUTymNKe5x8G5FE4ZmoZtmFECDbLxIhf4DqIjAPjhb9yIE8qe7yNvAXf9xboegvl
-jtbpcfqRrodsOjsOhgw3JhB6qyEi87PZr0wLDxEIJkEweId09EGFrMxntezpA4SE
-IxsJjnV1CSm/xCJ52Jjv1tG9BMFcehJGsyIbr0RjCxnH6qxWS5TrKewXyrG3h0V8
-rvitv0tcc2BaJu9ZKY7RKoEJUafpgEz6lBYtsJAunEsChX4VwMNIs2IlMwRGnUgm
-OjOJitIqFRL1W4mMnbvTKkKFqkXG0YaHBow+RHbx4H4y+Cn/2ybwbODcMbRiV00e
-o6SikSS96xhW1PLuYrgZu315P1MA8z1cXgxztRrMf52wUun2jjMCFx1C5NyrJIyf
-Xdu9A84XspM0LDtlsORXZ8yxv/6jxMsU37ONZE3W7a06lMMtJWMdj1CgZjlu1FHN
-aI2x22lMMOw/fCNgr3bZfSi1ZwhW1hjR4AHDUXgsR+dzznIfaGdc22nktgFm0oB6
-EWFib4PXc2rp6bafWrQT2vAeo7xkBekXajGcIxj53gwQkhYTZCzs/O2B6y3/0TwF
-HdvBI7T1owOoKRnuLdZvRfSCwtbZMUFy9agdMKvj5W4L3KyyXjPGdgTgynLui8dC
-n0CLc2dzuolO1EK/qyHr
-=k+gW
+iQIcBAABAgAGBQJZipYNAAoJEL/y7kLIKC52GroP/jtquMaM0d6rlJndGow1dg9y
+LzuKKjU1O8ftBhbgTiMs/7XWT4LDnN63Ql2LtDkISDb8fPNtOTlBO972ztwxJ1aq
+qzxKHvvh0UwZ/chFYkdNVUeFrZoJaCrIwd9F1rSNADhusoPlkPXPwGWLp4IUVoEl
+s7+XLcfux6zZ5YIaja8prohq9gQKx9Ob2ZJSp0YGorxEUbHu/rVNg0Dt67MEXuW8
+OblHSlY0vmU6DSqw4ck1kVMb4nu0KeN5zdSwRj9BWq+VFDO/vOzEqHuTmVUo+A/r
+dslzC6MwDTH5MN+b3cY+eHbmT5Vd08/yG0ZuLxlYHbYZ2lptLE9lt+EpNwBPnscp
+cXc7OFp2d0F4JlIG+pRO1tmRjbjood5xRPgS+PRTfsuY6SrtEd9Opt6X+P9wcitK
+EzG7gfss/XbfB0sU6yT+jLREhdtP/+uDZcLN91T8D2RPzwgKPiinjJT3b3PXCmuq
+MSjTJ557Xh/jD9/z17la0vssiGtUtbuBvIkp+GozR74eHBpbXVyRZhqg8luqj+uO
+BgeQJ/HS1DI6tr733GEJnoY0bAzBkMbnGRtjAyS49TBPc0odLM/jX6PJrY54rjg8
+8iPxeNfOM7+mxzjrLAlA2LxjYcxzPVbYikeRrCD6g3QfFG22KcKSlXNdkRp6X4FR
+fMOcP5JAD5JHbdy/mRQ+
+=P4qe
 -END PGP SIGNATURE-

Modified: dev/beam/2.1.0/apache-beam-2.1.0-python.zip.md5
==
--- dev/beam/2.1.0/apache-beam-2.1.0-python.zip.md5 (original)
+++ dev/beam/2.1.0/apache-beam-2.1.0-python.zip.md5 Wed Aug  9 04:59:42 2017
@@ -1 +1 @@
-24b0e777836cc7059f1ce231a48caea0  apache-beam-2.1.0-python.zip
+0122b39da8f5b73334d75298ff8e37af  apache-beam-2.1.0-python.zip

Modified: dev/beam/2.1.0/apache-beam-2.1.0-python.zip.sha1
==
--- dev/beam/2.1.0/apache-beam-2.1.0-python.zip.sha1 (original)
+++ dev/beam/2.1.0/apache-beam-2.1.0-python.zip.sha1 Wed Aug  9 04:59:42 2017
@@ -1 +1 @@
-ca78ae3e6ec36f393d41240a73e246ddc325c28e  apache-beam-2.1.0-python.zip
+cc5da6cecc538540a5f08098a03acc72a0c322c1  apache-beam-2.1.0-python.zip

Modified: dev/beam/2.1.0/apache-beam-2.1.0-source-release.zip
==
Binary files - no diff available.

Modified: dev/beam/2.1.0/apache-beam-2.1.0-source-release.zip.asc
==
--- dev/beam/2.1.0/apache-beam-2.1.0-source-release.zip.asc (original)
+++ dev/beam/2.1.0/apache-beam-2.1.0-source-release.zip.asc Wed Aug  9 04:59:42 
2017
@@ -1,17 +1,17 @@
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1
 
-iQIcBAABAgAGBQJZbhrsAAoJEL/y7kLIKC52aikQAIiQkhtIO7uaSVlDZzAWz3xR
-IzqLfML21V7KEonVMde2H0fiCJOnkwWCWN3J/xcxiVejASzSzi2sVA9u+PKLjoOh
-Xj8VH3UnA1bu+0UtDplB7+brsZtB84jJ97VQGvmU5XHYqouR6xhn0HR5XZNXG4El
-To6snSWFb4AZfK5LdLnFja32sY4sOqrzhK08K6yGtjUW7mLL24AeTBxKBBrHDe1/
-PjdhP8VZ25ORMs4jkPiTv1rHSqL7SZ74P1Hu+DF5zC7WG/I6KSe7bRdZhlgX3rD7
-QWd4hzYjQ3G7fmKpSUtqAnmvvDc+Njpqd6Tvc4B3QRomdiyhTtkkqorE3VkzsLRG
-rTRNdYhRa4k3KHG/GdwJO0YtviPTN4jODhUg2W9txpBGfo28SuaeQ1IHmrYKZYLS
-Nri9JZAPCH1HlbJ7Vd+xN50Mi9esQSNbwotj6ZWYvEdlCyXAVfeSZOOZxZwBe0cT
-ezU4AsbWqOP0AFsUlowab8ZIdZ7OYxzGo6d6WY2qrq+CEgakPn9Zy40ZoCeY1zZx
-63rxNbRN6Iopx/aof/9R9/IDGyfL2/qS3EhJ/xG3Sr637Kz46SUYOGUdZuBHaAdC
-FUed3tlAEcAg97FODs27u/lXxk1vsn7AV0AJuGL3b4f4XZJ7XeMTyBmuF3tCDufe
-08Mz06Rd6YGaPuLaSk3x
-=jEOJ
+iQIcBAABAgAGBQJZifpwAAoJEL/y7kLIKC52WVIQAKtW68nr0RYHs+JkKBrOiBuf
+l2lzeamYm3kkf3TPS2CTFdWbvjLkyrdl/92UbqhRzeCrQy4hps++VhlZpEpmalM5

[jira] [Commented] (BEAM-2730) make BeamRecord an immutable type

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119406#comment-16119406
 ] 

ASF GitHub Bot commented on BEAM-2730:
--

Github user XuMingmin closed the pull request at:

https://github.com/apache/beam/pull/3692


> make BeamRecord an immutable type
> -
>
> Key: BEAM-2730
> URL: https://issues.apache.org/jira/browse/BEAM-2730
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge, dsl_sql_review
>
> Refer to discussion 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTnbIjE



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3692: [BEAM-2730] make BeamRecord an immutable type

[XuMingmin]

Github user XuMingmin closed the pull request at:

https://github.com/apache/beam/pull/3692


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3734

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2742) Switch from primitives to Java types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119328#comment-16119328
 ] 

ASF GitHub Bot commented on BEAM-2742:
--

Github user xumingming closed the pull request at:

https://github.com/apache/beam/pull/3699


> Switch from primitives to Java types
> 
>
> Key: BEAM-2742
> URL: https://issues.apache.org/jira/browse/BEAM-2742
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>  Labels: dsl_sql_merge, dsl_sql_review
>
> Refer to 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTrrtwc



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3699: [BEAM-2742] change Field type from primitives to bo...

[xumingming]

Github user xumingming closed the pull request at:

https://github.com/apache/beam/pull/3699


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-2709) Add TezRunner

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He updated BEAM-2709:
-
Component/s: runner-tez

> Add TezRunner
> -
>
> Key: BEAM-2709
> URL: https://issues.apache.org/jira/browse/BEAM-2709
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-ideas, runner-tez
>Reporter: Brandon Scheller
>Assignee: Brandon Scheller
>
> Add a TezRunner to Beam



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-165) Add Hadoop MapReduce runner

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He reassigned BEAM-165:
---

   Assignee: Pei He  (was: Jean-Baptiste Onofré)
Component/s: runner-mapreduce
 Issue Type: New Feature  (was: Wish)

> Add Hadoop MapReduce runner
> ---
>
> Key: BEAM-165
> URL: https://issues.apache.org/jira/browse/BEAM-165
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-ideas, runner-mapreduce
>Reporter: Jean-Baptiste Onofré
>Assignee: Pei He
>
> I think a MapReduce runner could be a good addition to Beam. It would allow 
> users to smoothly "migrate" from MapReduce to Spark or Flink.
> Of course, the MapReduce runner will run in batch mode (not stream).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-165) Add Hadoop MapReduce runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119313#comment-16119313
 ] 

ASF GitHub Bot commented on BEAM-165:
-

GitHub user peihe opened a pull request:

https://github.com/apache/beam/pull/3705

[BEAM-165] Initial implementation of the MapReduce runner.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam mr-runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3705.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3705


commit 9fffd554f1e5fd6465989bb3568dfb6f2d854eeb
Author: Pei He 
Date:   2017-07-06T02:22:27Z

Initial commit for MapReduceRunner.

commit 3bacc3e6099718bbcb672ab738ad607204fa8487
Author: Pei He 
Date:   2017-07-11T02:45:11Z

MapReduceRunner: add Graph and its visitors.

commit b62238545c1ba95e9857710d91609431cd0a2f93
Author: Pei He 
Date:   2017-07-13T06:09:10Z

MapReduceRunner: add unit tests for GraphConverter and GraphPlanner.

commit 64548dc949d0251949efdd02df68eed6032a64f4
Author: Pei He 
Date:   2017-07-21T05:46:36Z

mr-runner: support BoundedSource with BeamInputFormat.

commit 3070fded4bc0dde8f08b63e53f94342d21d4bc53
Author: Pei He 
Date:   2017-07-24T12:15:37Z

mr-runner: add JobPrototype and translate it to a MR job.

commit 0e16c52463278c6c4f9db61253c6b8287c4718ff
Author: Pei He 
Date:   2017-07-25T13:44:34Z

mr-runner: add ParDoOperation and support ParDos chaining.

commit 72a50aa508726e34110475448e9bb52381711faf
Author: Pei He 
Date:   2017-07-26T13:19:30Z

mr-runner: add BeamReducer and support GroupByKey.

commit 1b449b0981ae2bb2e1b397113b48eec1df53a4b1
Author: Pei He 
Date:   2017-07-27T07:01:22Z

core-java: InMemoryTimerInternals expose getTimers() for timer firings in 
mr-runner.

commit 6d152a623550446b06bde91ad0c54df1f7e5c60b
Author: Pei He 
Date:   2017-07-27T02:52:32Z

mr-runner: support reduce side ParDos and WordCount.

commit 1ef0dec520ee301328007f99419c25b7a7b5b46f
Author: Pei He 
Date:   2017-07-27T07:05:06Z

mr-runner: add JarClassInstanceFactory to run ValidatesRunner tests.

commit 02c77375cc114a210f99079cf3efec3d2426941e
Author: Pei He 
Date:   2017-07-28T08:31:41Z

mr-runner: refactors and creates Graph data structures to handle general 
Beam pipelines.

commit bb3349e10c0cfacd81b610880ddfec030fedf34d
Author: Pei He 
Date:   2017-08-02T11:19:14Z

mr-runner: support graph visualization with dotfiles.

commit 0fd2f15847e1f9bdd42f4388f6de6e566f9b64ef
Author: Pei He 
Date:   2017-08-02T13:59:21Z

mr-runner: hack to get around that ViewAsXXX.expand() return wrong output 
PValue.

commit 5079322c2e2a092a85b9740d04a7ca9bd887460e
Author: Pei He 
Date:   2017-08-08T03:30:29Z

mr-runner: support PCollections materialization with multiple MR jobs.

commit ad4cd2d5ea2af795bba86319d6447e7f8c415bf2
Author: Pei He 
Date:   2017-08-08T07:49:04Z

mr-runner: support multiple SourceOperations by composing and partitioning.

commit de2859e1092bfc3fdd036c3becf9e79fbb8fc8fa
Author: Pei He 
Date:   2017-08-08T09:38:58Z

mr-runner: support side inputs by reading in all views contents.

commit 69ee0f92bf170f0628d788d5dabeb339e7f1ad0c
Author: Pei He 
Date:   2017-08-08T14:07:12Z

mr-runner: setup file paths for read and write sides of materialization.




> Add Hadoop MapReduce runner
> 

[jira] [Commented] (BEAM-2733) update JavaDoc of BeamSql

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119312#comment-16119312
 ] 

ASF GitHub Bot commented on BEAM-2733:
--

Github user xumingming closed the pull request at:

https://github.com/apache/beam/pull/3704


> update JavaDoc of BeamSql
> -
>
> Key: BEAM-2733
> URL: https://issues.apache.org/jira/browse/BEAM-2733
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>  Labels: dsl_sql_merge, dsl_sql_review
>
> refer to discussions:
> 1. 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmFcik
> 2. 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhfc
> 3. 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhfU
> 4. 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhfY
> 5. 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhfQ



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3704: [BEAM-2733] update javadoc for BeamSql

[xumingming]

Github user xumingming closed the pull request at:

https://github.com/apache/beam/pull/3704


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3705: [BEAM-165] Initial implementation of the MapReduce ...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/beam/pull/3705

[BEAM-165] Initial implementation of the MapReduce runner.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam mr-runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3705.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3705


commit 9fffd554f1e5fd6465989bb3568dfb6f2d854eeb
Author: Pei He 
Date:   2017-07-06T02:22:27Z

Initial commit for MapReduceRunner.

commit 3bacc3e6099718bbcb672ab738ad607204fa8487
Author: Pei He 
Date:   2017-07-11T02:45:11Z

MapReduceRunner: add Graph and its visitors.

commit b62238545c1ba95e9857710d91609431cd0a2f93
Author: Pei He 
Date:   2017-07-13T06:09:10Z

MapReduceRunner: add unit tests for GraphConverter and GraphPlanner.

commit 64548dc949d0251949efdd02df68eed6032a64f4
Author: Pei He 
Date:   2017-07-21T05:46:36Z

mr-runner: support BoundedSource with BeamInputFormat.

commit 3070fded4bc0dde8f08b63e53f94342d21d4bc53
Author: Pei He 
Date:   2017-07-24T12:15:37Z

mr-runner: add JobPrototype and translate it to a MR job.

commit 0e16c52463278c6c4f9db61253c6b8287c4718ff
Author: Pei He 
Date:   2017-07-25T13:44:34Z

mr-runner: add ParDoOperation and support ParDos chaining.

commit 72a50aa508726e34110475448e9bb52381711faf
Author: Pei He 
Date:   2017-07-26T13:19:30Z

mr-runner: add BeamReducer and support GroupByKey.

commit 1b449b0981ae2bb2e1b397113b48eec1df53a4b1
Author: Pei He 
Date:   2017-07-27T07:01:22Z

core-java: InMemoryTimerInternals expose getTimers() for timer firings in 
mr-runner.

commit 6d152a623550446b06bde91ad0c54df1f7e5c60b
Author: Pei He 
Date:   2017-07-27T02:52:32Z

mr-runner: support reduce side ParDos and WordCount.

commit 1ef0dec520ee301328007f99419c25b7a7b5b46f
Author: Pei He 
Date:   2017-07-27T07:05:06Z

mr-runner: add JarClassInstanceFactory to run ValidatesRunner tests.

commit 02c77375cc114a210f99079cf3efec3d2426941e
Author: Pei He 
Date:   2017-07-28T08:31:41Z

mr-runner: refactors and creates Graph data structures to handle general 
Beam pipelines.

commit bb3349e10c0cfacd81b610880ddfec030fedf34d
Author: Pei He 
Date:   2017-08-02T11:19:14Z

mr-runner: support graph visualization with dotfiles.

commit 0fd2f15847e1f9bdd42f4388f6de6e566f9b64ef
Author: Pei He 
Date:   2017-08-02T13:59:21Z

mr-runner: hack to get around that ViewAsXXX.expand() return wrong output 
PValue.

commit 5079322c2e2a092a85b9740d04a7ca9bd887460e
Author: Pei He 
Date:   2017-08-08T03:30:29Z

mr-runner: support PCollections materialization with multiple MR jobs.

commit ad4cd2d5ea2af795bba86319d6447e7f8c415bf2
Author: Pei He 
Date:   2017-08-08T07:49:04Z

mr-runner: support multiple SourceOperations by composing and partitioning.

commit de2859e1092bfc3fdd036c3becf9e79fbb8fc8fa
Author: Pei He 
Date:   2017-08-08T09:38:58Z

mr-runner: support side inputs by reading in all views contents.

commit 69ee0f92bf170f0628d788d5dabeb339e7f1ad0c
Author: Pei He 
Date:   2017-08-08T14:07:12Z

mr-runner: setup file paths for read and write sides of materialization.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at 

[1/2] beam git commit: [BEAM-2730] make BeamRecord an immutable type

[takidau]

Repository: beam
Updated Branches:
  refs/heads/DSL_SQL 926c70a34 -> 66286749f


[BEAM-2730] make BeamRecord an immutable type


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/fa36b128
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/fa36b128
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/fa36b128

Branch: refs/heads/DSL_SQL
Commit: fa36b1285a4d8fc967b6302456d0f7fcf7943894
Parents: 926c70a
Author: mingmxu 
Authored: Mon Aug 7 16:12:21 2017 -0700
Committer: Tyler Akidau 
Committed: Tue Aug 8 19:07:03 2017 -0700

--
 .../apache/beam/sdk/coders/BeamRecordCoder.java |  10 +-
 .../org/apache/beam/sdk/values/BeamRecord.java  |  24 ++---
 .../extensions/sql/example/BeamSqlExample.java  |   5 +-
 .../extensions/sql/impl/rel/BeamJoinRel.java|   7 +-
 .../extensions/sql/impl/rel/BeamValuesRel.java  |   7 +-
 .../transform/BeamAggregationTransforms.java|  26 +++--
 .../sql/impl/transform/BeamJoinTransforms.java  |  20 ++--
 .../sql/impl/transform/BeamSqlProjectFn.java|   9 +-
 .../sql/schema/BeamSqlRecordType.java   |   5 +-
 .../extensions/sql/schema/BeamTableUtils.java   |  41 +++
 .../sql/BeamSqlDslAggregationTest.java  | 107 ---
 .../beam/sdk/extensions/sql/BeamSqlDslBase.java |  56 +++---
 .../extensions/sql/BeamSqlDslProjectTest.java   |  48 -
 .../extensions/sql/BeamSqlDslUdfUdafTest.java   |   8 +-
 .../beam/sdk/extensions/sql/TestUtils.java  |   6 +-
 .../interpreter/BeamSqlFnExecutorTestBase.java  |   8 +-
 .../sql/schema/BeamSqlRowCoderTest.java |  16 +--
 .../sql/schema/kafka/BeamKafkaCSVTableTest.java |  12 +--
 18 files changed, 132 insertions(+), 283 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/fa36b128/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BeamRecordCoder.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BeamRecordCoder.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BeamRecordCoder.java
index a6200f6..4e24b82 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BeamRecordCoder.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BeamRecordCoder.java
@@ -20,6 +20,7 @@ package org.apache.beam.sdk.coders;
 import java.io.IOException;
 import java.io.InputStream;
 import java.io.OutputStream;
+import java.util.ArrayList;
 import java.util.BitSet;
 import java.util.List;
 import org.apache.beam.sdk.annotations.Experimental;
@@ -69,14 +70,15 @@ public class BeamRecordCoder extends 
CustomCoder {
   public BeamRecord decode(InputStream inStream) throws CoderException, 
IOException {
 BitSet nullFields = nullListCoder.decode(inStream);
 
-BeamRecord record = new BeamRecord(recordType);
+List fieldValues = new ArrayList<>(recordType.size());
 for (int idx = 0; idx < recordType.size(); ++idx) {
   if (nullFields.get(idx)) {
-continue;
+fieldValues.add(null);
+  } else {
+fieldValues.add(coderArray.get(idx).decode(inStream));
   }
-
-  record.addField(idx, coderArray.get(idx).decode(inStream));
 }
+BeamRecord record = new BeamRecord(recordType, fieldValues);
 
 return record;
   }

http://git-wip-us.apache.org/repos/asf/beam/blob/fa36b128/sdks/java/core/src/main/java/org/apache/beam/sdk/values/BeamRecord.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/BeamRecord.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/BeamRecord.java
index 35a96f6..6e4bd4c 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/BeamRecord.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/BeamRecord.java
@@ -20,6 +20,7 @@ package org.apache.beam.sdk.values;
 import java.io.Serializable;
 import java.math.BigDecimal;
 import java.util.ArrayList;
+import java.util.Arrays;
 import java.util.Date;
 import java.util.GregorianCalendar;
 import java.util.List;
@@ -32,29 +33,28 @@ import org.apache.beam.sdk.annotations.Experimental;
  */
 @Experimental
 public class BeamRecord implements Serializable {
+  //immutable list of field values.
   private List dataValues;
   private BeamRecordType dataType;
 
-  public BeamRecord(BeamRecordType dataType) {
+  public BeamRecord(BeamRecordType dataType, List rawdataValues) {
 this.dataType = dataType;
-this.dataValues = new ArrayList<>();
+this.dataValues = new ArrayList<>(dataType.size());
+
 for (int idx = 0; idx < dataType.size(); ++idx) {
   dataValues.add(null);
 }
-  }
 
-  public BeamRecord(BeamRecordType dataType, List dataValues) {
-this(dataType);
-

[2/2] beam git commit: [BEAM-2730] This closes #3692

[takidau]

[BEAM-2730] This closes #3692


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/66286749
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/66286749
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/66286749

Branch: refs/heads/DSL_SQL
Commit: 66286749f2d64eb3f25d99b4d9bf56c49d9f62b6
Parents: 926c70a fa36b12
Author: Tyler Akidau 
Authored: Tue Aug 8 19:07:56 2017 -0700
Committer: Tyler Akidau 
Committed: Tue Aug 8 19:07:56 2017 -0700

--
 .../apache/beam/sdk/coders/BeamRecordCoder.java |  10 +-
 .../org/apache/beam/sdk/values/BeamRecord.java  |  24 ++---
 .../extensions/sql/example/BeamSqlExample.java  |   5 +-
 .../extensions/sql/impl/rel/BeamJoinRel.java|   7 +-
 .../extensions/sql/impl/rel/BeamValuesRel.java  |   7 +-
 .../transform/BeamAggregationTransforms.java|  26 +++--
 .../sql/impl/transform/BeamJoinTransforms.java  |  20 ++--
 .../sql/impl/transform/BeamSqlProjectFn.java|   9 +-
 .../sql/schema/BeamSqlRecordType.java   |   5 +-
 .../extensions/sql/schema/BeamTableUtils.java   |  41 +++
 .../sql/BeamSqlDslAggregationTest.java  | 107 ---
 .../beam/sdk/extensions/sql/BeamSqlDslBase.java |  56 +++---
 .../extensions/sql/BeamSqlDslProjectTest.java   |  48 -
 .../extensions/sql/BeamSqlDslUdfUdafTest.java   |   8 +-
 .../beam/sdk/extensions/sql/TestUtils.java  |   6 +-
 .../interpreter/BeamSqlFnExecutorTestBase.java  |   8 +-
 .../sql/schema/BeamSqlRowCoderTest.java |  16 +--
 .../sql/schema/kafka/BeamKafkaCSVTableTest.java |  12 +--
 18 files changed, 132 insertions(+), 283 deletions(-)
--

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2792

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-2731) Add a generic Reshuffle transform to Python SDK

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-2731.
---
   Resolution: Duplicate
Fix Version/s: Not applicable

Duplicate of https://issues.apache.org/jira/browse/BEAM-1872

> Add a generic Reshuffle transform to Python SDK
> ---
>
> Key: BEAM-2731
> URL: https://issues.apache.org/jira/browse/BEAM-2731
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: Chamikara Jayalath
> Fix For: Not applicable
>
>
> Java SDK has a Reshuffle transform [1] but Python SDK does not have this.
> Following factors have to be considered when implementing this transform.
> * Transform in Java has been deprecated due to intended side effects not 
> being portable.
> * A lot of details related to portability were uncovered/discussed when we 
> tried to add a generic Redistribute transform [2][3].
> [1] 
> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Reshuffle.java
> [2] https://github.com/apache/beam/pull/1036
> [3] 
> https://lists.apache.org/thread.html/ac34c9ac665a8d9f67b0254015e44c59ea65ecc1360d4014b95d3b2e@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3733

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #4536

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3193: Use Bytes instead of Any in Runner API FunctionSpec

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3193


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3193

[tgroh]

This closes #3193


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9ed2cf41
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9ed2cf41
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9ed2cf41

Branch: refs/heads/master
Commit: 9ed2cf41f1f068b3b27e9a814d6f7ceed3406dbf
Parents: f5714f2 2b9b050
Author: Thomas Groh 
Authored: Tue Aug 8 18:19:00 2017 -0700
Committer: Thomas Groh 
Committed: Tue Aug 8 18:19:00 2017 -0700

--
 .../core/construction/CoderTranslation.java | 16 +---
 .../core/construction/CombineTranslation.java   | 28 ++
 .../CreatePCollectionViewTranslation.java   | 17 +---
 .../construction/PTransformTranslation.java | 12 +--
 .../core/construction/ParDoTranslation.java | 58 +---
 .../core/construction/PipelineTranslation.java  | 12 +--
 .../core/construction/ReadTranslation.java  | 56 +--
 .../construction/TestStreamTranslation.java |  5 +-
 .../construction/WindowIntoTranslation.java |  9 +-
 .../WindowingStrategyTranslation.java   | 97 
 .../construction/WriteFilesTranslation.java | 29 +++---
 .../CreatePCollectionViewTranslationTest.java   |  6 +-
 .../core/construction/ParDoTranslationTest.java |  3 +-
 .../construction/TestStreamTranslationTest.java |  4 +-
 .../src/main/proto/beam_runner_api.proto|  5 +-
 .../beam/fn/harness/BeamFnDataReadRunner.java   |  4 +-
 .../beam/fn/harness/BeamFnDataWriteRunner.java  |  4 +-
 .../beam/fn/harness/BoundedSourceRunner.java| 10 +-
 .../apache/beam/fn/harness/FnApiDoFnRunner.java | 10 +-
 .../fn/harness/BeamFnDataReadRunnerTest.java|  5 +-
 .../fn/harness/BeamFnDataWriteRunnerTest.java   |  5 +-
 .../fn/harness/BoundedSourceRunnerTest.java | 18 ++--
 .../beam/fn/harness/FnApiDoFnRunnerTest.java| 13 +--
 sdks/python/apache_beam/coders/coders.py| 11 +--
 .../runners/portability/fn_api_runner.py| 78 ++--
 .../runners/worker/bundle_processor.py  | 21 ++---
 sdks/python/apache_beam/transforms/core.py  | 10 +-
 .../python/apache_beam/transforms/ptransform.py |  6 +-
 sdks/python/apache_beam/utils/proto_utils.py| 11 +++
 sdks/python/apache_beam/utils/urns.py   |  6 +-
 30 files changed, 250 insertions(+), 319 deletions(-)
--

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2791

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-2755) Wordcount it test fail

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-2755:
-

 Summary: Wordcount it test fail
 Key: BEAM-2755
 URL: https://issues.apache.org/jira/browse/BEAM-2755
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Ahmet Altay
Assignee: Mark Liu


https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/2886/console
 failed with the error below. It might be a flake, or an error in test_utils:

Delete operation failed with exceptions {'*': IOError(OSError(2, 'No such file 
or directory'),), '-': IOError(OSError(2, 'No such file or directory'),), '/': 
IOError(OSError(13, 'Permission denied'),), '1': IOError(OSError(2, 'No such 
file or directory'),), '0': IOError(OSError(2, 'No such file or directory'),), 
'3': IOError(OSError(2, 'No such file or directory'),), '2': IOError(OSError(2, 
'No such file or directory'),), '5': IOError(OSError(2, 'No such file or 
directory'),), ':': IOError(OSError(2, 'No such file or directory'),), 'a': 
IOError(OSError(2, 'No such file or directory'),), 'c': IOError(OSError(2, 'No 
such file or directory'),), 'e': IOError(OSError(2, 'No such file or 
directory'),), 'd': IOError(OSError(2, 'No such file or directory'),), 'g': 
IOError(OSError(2, 'No such file or directory'),), 'f': IOError(OSError(2, 'No 
such file or directory'),), 'i': IOError(OSError(2, 'No such file or 
directory'),), 'm': IOError(OSError(2, 'No such file or directory'),), 'l': 
IOError(OSError(2, 'No such file or directory'),), 'o': IOError(OSError(2, 'No 
such file or directory'),), 'n': IOError(OSError(2, 'No such file or 
directory'),), 'p': IOError(OSError(2, 'No such file or directory'),), 's': 
IOError(OSError(2, 'No such file or directory'),), 'r': IOError(OSError(2, 'No 
such file or directory'),), 'u': IOError(OSError(2, 'No such file or 
directory'),), 't': IOError(OSError(2, 'No such file or directory'),), 'y': 
IOError(OSError(2, 'No such file or directory'),)}
 Traceback for above exception (most recent call last):
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/utils/retry.py",
 line 175, in wrapper
return fun(*args, **kwargs)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/testing/test_utils.py",
 line 91, in delete_files
FileSystems.delete(file_paths)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/io/filesystems.py",
 line 224, in delete
return filesystem.delete(paths)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/io/localfilesystem.py",
 line 267, in delete
raise BeamIOError("Delete operation failed", exceptions)

Mark, could you take a look at this?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2754) Simplify DefaultCoder

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119091#comment-16119091
 ] 

Kenneth Knowles commented on BEAM-2754:
---

We used to automate this via FooCoder.of(Class). This was lost when we added 
support for generics, when I think we switched to reflectively invoking 
getCoderProvider(). To this provider, we can pass both the type and subcoders, 
supporting strictly more use cases by merging the prior concepts of 
CoderFactory and CoderProvider as we had planned for some time.

I like the idea of adding back very simple cases. We should switch all of this 
to use an annotation-driven interface rather than magic methods.

> Simplify DefaultCoder
> -
>
> Key: BEAM-2754
> URL: https://issues.apache.org/jira/browse/BEAM-2754
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Minor
>
> DefaultCoder is too hard to use - it requires having a CoderProvider class 
> and a bunch of other boilerplate. It should have simpler usage modes, e.g. 
> @DefaultCoder(FooCoder.class) class Foo should simply work if FooCoder only 
> has a "static FooCoder of()" method, or "of(TypeDescriptor)" for more complex 
> use cases (eg generics).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : beam_PostCommit_Python_Verify #2887

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2754) Simplify DefaultCoder

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119049#comment-16119049
 ] 

Eugene Kirpichov commented on BEAM-2754:


cc: [~tgroh] [~kenn]

> Simplify DefaultCoder
> -
>
> Key: BEAM-2754
> URL: https://issues.apache.org/jira/browse/BEAM-2754
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Minor
>
> DefaultCoder is too hard to use - it requires having a CoderProvider class 
> and a bunch of other boilerplate. It should have simpler usage modes, e.g. 
> @DefaultCoder(FooCoder.class) class Foo should simply work if FooCoder only 
> has a "static FooCoder of()" method, or "of(TypeDescriptor)" for more complex 
> use cases (eg generics).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2754) Simplify DefaultCoder

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-2754:
--

 Summary: Simplify DefaultCoder
 Key: BEAM-2754
 URL: https://issues.apache.org/jira/browse/BEAM-2754
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Eugene Kirpichov
Priority: Minor


DefaultCoder is too hard to use - it requires having a CoderProvider class and 
a bunch of other boilerplate. It should have simpler usage modes, e.g. 
@DefaultCoder(FooCoder.class) class Foo should simply work if FooCoder only has 
a "static FooCoder of()" method, or "of(TypeDescriptor)" for more complex use 
cases (eg generics).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3732

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #2886

[Apache Jenkins Server]

See 


--
[...truncated 638.62 KB...]
  "name": "SideInput-s16", 
  "properties": {
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}
  ]
}, 
"output_name": "out", 
"user_name": 
"write/Write/WriteImpl/FinalizeWrite/AsIter(Extract.out.0).output"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s14"
}, 
"user_name": "write/Write/WriteImpl/FinalizeWrite/AsIter(Extract.out.0)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s17", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_finalize_write"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {
  "SideInput-s15": {
"@type": "OutputReference", 
"output_name": "out", 
"step_name": "SideInput-s15"
  }, 
  "SideInput-s16": {
"@type": "OutputReference", 
"output_name": "out", 
"step_name": "SideInput-s16"
  }
}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "write/Write/WriteImpl/FinalizeWrite.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s7"
}, 
"serialized_fn": "", 
"user_name": "write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2017-08-08_13_48_31-14349531698595810259]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2017-08-08_13_48_31-14349531698595810259?project=apache-beam-testing
root: INFO: Job 

[jira] [Created] (BEAM-2753) File DynamicDestinations side inputs don't work with sharding

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-2753:
--

 Summary: File DynamicDestinations side inputs don't work with 
sharding
 Key: BEAM-2753
 URL: https://issues.apache.org/jira/browse/BEAM-2753
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Reuven Lax


WriteWithShardingFactory uses PTransformReplacements.getSingletonMaininput 
https://github.com/apache/beam/blob/master/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WriteWithShardingFactory.java#L74

However if the dynamic destinations have a side input, then the transform has 
more than 1 input and the function fails:

Exception in thread "main" java.lang.IllegalArgumentException: Got multiple 
inputs that are not additional inputs for a singleton main input: Avro schema 
side input/ParMultiDo(Anonymous).out0 [PCollection] and Run read all/Execute 
queries/ParMultiDo(NaiveSpannerRead).out0 [PCollection]
at 
org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:383)
at 
org.apache.beam.runners.direct.repackaged.runners.core.construction.PTransformReplacements.getSingletonMainInput(PTransformReplacements.java:50)
at 
org.apache.beam.runners.direct.repackaged.runners.core.construction.PTransformReplacements.getSingletonMainInput(PTransformReplacements.java:41)
at 
org.apache.beam.runners.direct.WriteWithShardingFactory.getReplacementTransform(WriteWithShardingFactory.java:74)
at org.apache.beam.sdk.Pipeline.applyReplacement(Pipeline.java:540)
at org.apache.beam.sdk.Pipeline.replace(Pipeline.java:280)
at org.apache.beam.sdk.Pipeline.replaceAll(Pipeline.java:201)
at 
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:169)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:289)

This is not caught by unit tests because unit tests specify withoutSharding().
https://github.com/apache/beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java#L644

CC: [~mkhadikov]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2753) File DynamicDestinations side inputs don't work with sharding

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov updated BEAM-2753:
---
Fix Version/s: 2.2.0

> File DynamicDestinations side inputs don't work with sharding
> -
>
> Key: BEAM-2753
> URL: https://issues.apache.org/jira/browse/BEAM-2753
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Reuven Lax
> Fix For: 2.2.0
>
>
> WriteWithShardingFactory uses PTransformReplacements.getSingletonMaininput 
> https://github.com/apache/beam/blob/master/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WriteWithShardingFactory.java#L74
> However if the dynamic destinations have a side input, then the transform has 
> more than 1 input and the function fails:
> Exception in thread "main" java.lang.IllegalArgumentException: Got multiple 
> inputs that are not additional inputs for a singleton main input: Avro schema 
> side input/ParMultiDo(Anonymous).out0 [PCollection] and Run read all/Execute 
> queries/ParMultiDo(NaiveSpannerRead).out0 [PCollection]
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:383)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.PTransformReplacements.getSingletonMainInput(PTransformReplacements.java:50)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.PTransformReplacements.getSingletonMainInput(PTransformReplacements.java:41)
>   at 
> org.apache.beam.runners.direct.WriteWithShardingFactory.getReplacementTransform(WriteWithShardingFactory.java:74)
>   at org.apache.beam.sdk.Pipeline.applyReplacement(Pipeline.java:540)
>   at org.apache.beam.sdk.Pipeline.replace(Pipeline.java:280)
>   at org.apache.beam.sdk.Pipeline.replaceAll(Pipeline.java:201)
>   at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:169)
>   at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:289)
> This is not caught by unit tests because unit tests specify withoutSharding().
> https://github.com/apache/beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java#L644
> CC: [~mkhadikov]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3731

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2790

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2657) Add Solr IO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118862#comment-16118862
 ] 

ASF GitHub Bot commented on BEAM-2657:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3618


> Add Solr IO
> ---
>
> Key: BEAM-2657
> URL: https://issues.apache.org/jira/browse/BEAM-2657
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>
> I'm working on a new SolrIO ( this components borrow som design's idea from 
> ElasticsearchIO ) providing both bounded source and sink.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3618: [BEAM-2657] Create Solr IO

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3618


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3618: [BEAM-2657] Create Solr IO

[jkff]

This closes #3618: [BEAM-2657] Create Solr IO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f5714f22
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f5714f22
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f5714f22

Branch: refs/heads/master
Commit: f5714f2203974a0f97f778d62a82678025bd425e
Parents: 2fa4fde d00ff9e
Author: Eugene Kirpichov 
Authored: Tue Aug 8 11:54:25 2017 -0700
Committer: Eugene Kirpichov 
Committed: Tue Aug 8 11:54:25 2017 -0700

--
 pom.xml |   8 +-
 .../sdk/io/common/IOTestPipelineOptions.java|   6 +
 sdks/java/io/pom.xml|   1 +
 sdks/java/io/solr/pom.xml   | 147 
 .../beam/sdk/io/solr/AuthorizedSolrClient.java  |  91 +++
 .../beam/sdk/io/solr/JavaBinCodecCoder.java |  98 +++
 .../org/apache/beam/sdk/io/solr/SolrIO.java | 717 +++
 .../apache/beam/sdk/io/solr/package-info.java   |  20 +
 .../beam/sdk/io/solr/JavaBinCodecCoderTest.java |  81 +++
 .../org/apache/beam/sdk/io/solr/SolrIOTest.java | 269 +++
 .../beam/sdk/io/solr/SolrIOTestUtils.java   | 132 
 .../resources/cloud-minimal/conf/schema.xml |  29 +
 .../resources/cloud-minimal/conf/solrconfig.xml |  48 ++
 sdks/java/javadoc/pom.xml   |   5 +
 14 files changed, 1651 insertions(+), 1 deletion(-)
--

[1/2] beam git commit: [BEAM-2657] Create Solr IO

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master 2fa4fdecd -> f5714f220


[BEAM-2657] Create Solr IO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d00ff9e2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d00ff9e2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d00ff9e2

Branch: refs/heads/master
Commit: d00ff9e215092af2666459b742f62c6b0bb4bff9
Parents: 2fa4fde
Author: Cao Manh Dat 
Authored: Sat Jul 22 14:38:07 2017 +0700
Committer: Eugene Kirpichov 
Committed: Tue Aug 8 11:54:09 2017 -0700

--
 pom.xml |   8 +-
 .../sdk/io/common/IOTestPipelineOptions.java|   6 +
 sdks/java/io/pom.xml|   1 +
 sdks/java/io/solr/pom.xml   | 147 
 .../beam/sdk/io/solr/AuthorizedSolrClient.java  |  91 +++
 .../beam/sdk/io/solr/JavaBinCodecCoder.java |  98 +++
 .../org/apache/beam/sdk/io/solr/SolrIO.java | 717 +++
 .../apache/beam/sdk/io/solr/package-info.java   |  20 +
 .../beam/sdk/io/solr/JavaBinCodecCoderTest.java |  81 +++
 .../org/apache/beam/sdk/io/solr/SolrIOTest.java | 269 +++
 .../beam/sdk/io/solr/SolrIOTestUtils.java   | 132 
 .../resources/cloud-minimal/conf/schema.xml |  29 +
 .../resources/cloud-minimal/conf/solrconfig.xml |  48 ++
 sdks/java/javadoc/pom.xml   |   5 +
 14 files changed, 1651 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/d00ff9e2/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 80ab6e2..1bdaa97 100644
--- a/pom.xml
+++ b/pom.xml
@@ -524,7 +524,13 @@
 ${project.version}
   
 
- 
+  
+org.apache.beam
+beam-sdks-java-io-solr
+${project.version}
+  
+
+  
 org.apache.beam
 beam-sdks-java-io-hadoop-input-format
${project.version}

http://git-wip-us.apache.org/repos/asf/beam/blob/d00ff9e2/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
--
diff --git 
a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
 
b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
index 25ab929..256c94d 100644
--- 
a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
+++ 
b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
@@ -71,6 +71,12 @@ public interface IOTestPipelineOptions extends 
TestPipelineOptions {
   Integer getElasticsearchHttpPort();
   void setElasticsearchHttpPort(Integer value);
 
+  /* Solr */
+  @Description("Address of Zookeeper server for Solr")
+  @Default.String("zookeeper-server")
+  String getSolrZookeeperServer();
+  void setSolrZookeeperServer(String value);
+
   /* Cassandra */
   @Description("Host for Cassandra server (host name/ip address)")
   @Default.String("cassandra-host")

http://git-wip-us.apache.org/repos/asf/beam/blob/d00ff9e2/sdks/java/io/pom.xml
--
diff --git a/sdks/java/io/pom.xml b/sdks/java/io/pom.xml
index 4e02aa8..c291e5d 100644
--- a/sdks/java/io/pom.xml
+++ b/sdks/java/io/pom.xml
@@ -56,6 +56,7 @@
 kinesis
 mongodb
 mqtt
+solr
 xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/d00ff9e2/sdks/java/io/solr/pom.xml
--
diff --git a/sdks/java/io/solr/pom.xml b/sdks/java/io/solr/pom.xml
new file mode 100644
index 000..a757a57
--- /dev/null
+++ b/sdks/java/io/solr/pom.xml
@@ -0,0 +1,147 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+beam-sdks-java-io-parent
+org.apache.beam
+2.2.0-SNAPSHOT
+../pom.xml
+
+4.0.0
+
+beam-sdks-java-io-solr
+Apache Beam :: SDKs :: Java :: IO :: Solr
+IO to read and write from/to Solr.
+
+
+
+org.apache.beam
+beam-sdks-java-core
+
+
+
+com.google.guava
+guava
+
+
+
+org.apache.solr
+solr-solrj
+5.5.4
+
+
+
+com.google.code.findbugs
+jsr305
+
+
+
+org.apache.commons
+commons-compress
+
+
+
+
+com.google.auto.value
+auto-value
+provided

[beam] Git Push Summary

[jbonofre]

Repository: beam
Updated Tags:  refs/tags/v2.1.0-RC3 [created] da22fb904

beam git commit: [maven-release-plugin] rollback changes from release preparation of v2.1.0-RC3

[jbonofre]

Repository: beam
Updated Branches:
  refs/heads/release-2.1.0 ab5577f99 -> 656deff3e


[maven-release-plugin] rollback changes from release preparation of v2.1.0-RC3


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/656deff3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/656deff3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/656deff3

Branch: refs/heads/release-2.1.0
Commit: 656deff3ed9ee12999d35d3a38d20ddb851127f7
Parents: ab5577f
Author: Jean-Baptiste Onofré 
Authored: Tue Aug 8 20:45:27 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Tue Aug 8 20:45:27 2017 +0200

--
 examples/java/pom.xml   | 2 +-
 examples/java8/pom.xml  | 2 +-
 examples/pom.xml| 2 +-
 pom.xml | 4 ++--
 runners/apex/pom.xml| 2 +-
 runners/core-construction-java/pom.xml  | 2 +-
 runners/core-java/pom.xml   | 2 +-
 runners/direct-java/pom.xml | 2 +-
 runners/flink/pom.xml   | 2 +-
 runners/google-cloud-dataflow-java/pom.xml  | 2 +-
 runners/pom.xml | 2 +-
 runners/spark/pom.xml   | 2 +-
 sdks/common/fn-api/pom.xml  | 2 +-
 sdks/common/pom.xml | 2 +-
 sdks/common/runner-api/pom.xml  | 2 +-
 sdks/java/build-tools/pom.xml   | 2 +-
 sdks/java/core/pom.xml  | 2 +-
 sdks/java/extensions/google-cloud-platform-core/pom.xml | 2 +-
 sdks/java/extensions/jackson/pom.xml| 2 +-
 sdks/java/extensions/join-library/pom.xml   | 2 +-
 sdks/java/extensions/pom.xml| 2 +-
 sdks/java/extensions/protobuf/pom.xml   | 2 +-
 sdks/java/extensions/sorter/pom.xml | 2 +-
 sdks/java/harness/pom.xml   | 2 +-
 sdks/java/io/amqp/pom.xml   | 2 +-
 sdks/java/io/cassandra/pom.xml  | 2 +-
 sdks/java/io/common/pom.xml | 2 +-
 sdks/java/io/elasticsearch/pom.xml  | 2 +-
 sdks/java/io/google-cloud-platform/pom.xml  | 2 +-
 sdks/java/io/hadoop-common/pom.xml  | 2 +-
 sdks/java/io/hadoop-file-system/pom.xml | 2 +-
 sdks/java/io/hadoop/input-format/pom.xml| 2 +-
 sdks/java/io/hadoop/jdk1.8-tests/pom.xml| 2 +-
 sdks/java/io/hadoop/pom.xml | 2 +-
 sdks/java/io/hbase/pom.xml  | 2 +-
 sdks/java/io/hcatalog/pom.xml   | 2 +-
 sdks/java/io/jdbc/pom.xml   | 2 +-
 sdks/java/io/jms/pom.xml| 2 +-
 sdks/java/io/kafka/pom.xml  | 2 +-
 sdks/java/io/kinesis/pom.xml| 2 +-
 sdks/java/io/mongodb/pom.xml| 2 +-
 sdks/java/io/mqtt/pom.xml   | 2 +-
 sdks/java/io/pom.xml| 2 +-
 sdks/java/io/xml/pom.xml| 2 +-
 sdks/java/java8tests/pom.xml| 2 +-
 sdks/java/javadoc/pom.xml   | 2 +-
 sdks/java/maven-archetypes/examples-java8/pom.xml   | 2 +-
 sdks/java/maven-archetypes/examples/pom.xml | 2 +-
 sdks/java/maven-archetypes/pom.xml  | 2 +-
 sdks/java/maven-archetypes/starter/pom.xml  | 2 +-
 sdks/java/pom.xml   | 2 +-
 sdks/pom.xml| 2 +-
 sdks/python/pom.xml | 2 +-
 53 files changed, 54 insertions(+), 54 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/656deff3/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index c04932e..7ae4e6a 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-2.1.0
+2.1.0-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/656deff3/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index af18ac0..a0ce708 100644
--- a/examples/java8/pom.xml
+++ 

beam git commit: [maven-release-plugin] prepare release v2.1.0-RC3

[jbonofre]

Repository: beam
Updated Branches:
  refs/heads/release-2.1.0 bad996bb4 -> ab5577f99


[maven-release-plugin] prepare release v2.1.0-RC3


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ab5577f9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ab5577f9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ab5577f9

Branch: refs/heads/release-2.1.0
Commit: ab5577f9928555fee2412b3555e89328e37458b9
Parents: bad996b
Author: Jean-Baptiste Onofré 
Authored: Tue Aug 8 20:45:07 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Tue Aug 8 20:45:07 2017 +0200

--
 examples/java/pom.xml   | 2 +-
 examples/java8/pom.xml  | 2 +-
 examples/pom.xml| 2 +-
 pom.xml | 4 ++--
 runners/apex/pom.xml| 2 +-
 runners/core-construction-java/pom.xml  | 2 +-
 runners/core-java/pom.xml   | 2 +-
 runners/direct-java/pom.xml | 2 +-
 runners/flink/pom.xml   | 2 +-
 runners/google-cloud-dataflow-java/pom.xml  | 2 +-
 runners/pom.xml | 2 +-
 runners/spark/pom.xml   | 2 +-
 sdks/common/fn-api/pom.xml  | 2 +-
 sdks/common/pom.xml | 2 +-
 sdks/common/runner-api/pom.xml  | 2 +-
 sdks/java/build-tools/pom.xml   | 2 +-
 sdks/java/core/pom.xml  | 2 +-
 sdks/java/extensions/google-cloud-platform-core/pom.xml | 2 +-
 sdks/java/extensions/jackson/pom.xml| 2 +-
 sdks/java/extensions/join-library/pom.xml   | 2 +-
 sdks/java/extensions/pom.xml| 2 +-
 sdks/java/extensions/protobuf/pom.xml   | 2 +-
 sdks/java/extensions/sorter/pom.xml | 2 +-
 sdks/java/harness/pom.xml   | 2 +-
 sdks/java/io/amqp/pom.xml   | 2 +-
 sdks/java/io/cassandra/pom.xml  | 2 +-
 sdks/java/io/common/pom.xml | 2 +-
 sdks/java/io/elasticsearch/pom.xml  | 2 +-
 sdks/java/io/google-cloud-platform/pom.xml  | 2 +-
 sdks/java/io/hadoop-common/pom.xml  | 2 +-
 sdks/java/io/hadoop-file-system/pom.xml | 2 +-
 sdks/java/io/hadoop/input-format/pom.xml| 2 +-
 sdks/java/io/hadoop/jdk1.8-tests/pom.xml| 2 +-
 sdks/java/io/hadoop/pom.xml | 2 +-
 sdks/java/io/hbase/pom.xml  | 2 +-
 sdks/java/io/hcatalog/pom.xml   | 2 +-
 sdks/java/io/jdbc/pom.xml   | 2 +-
 sdks/java/io/jms/pom.xml| 2 +-
 sdks/java/io/kafka/pom.xml  | 2 +-
 sdks/java/io/kinesis/pom.xml| 2 +-
 sdks/java/io/mongodb/pom.xml| 2 +-
 sdks/java/io/mqtt/pom.xml   | 2 +-
 sdks/java/io/pom.xml| 2 +-
 sdks/java/io/xml/pom.xml| 2 +-
 sdks/java/java8tests/pom.xml| 2 +-
 sdks/java/javadoc/pom.xml   | 2 +-
 sdks/java/maven-archetypes/examples-java8/pom.xml   | 2 +-
 sdks/java/maven-archetypes/examples/pom.xml | 2 +-
 sdks/java/maven-archetypes/pom.xml  | 2 +-
 sdks/java/maven-archetypes/starter/pom.xml  | 2 +-
 sdks/java/pom.xml   | 2 +-
 sdks/pom.xml| 2 +-
 sdks/python/pom.xml | 2 +-
 53 files changed, 54 insertions(+), 54 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ab5577f9/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 7ae4e6a..c04932e 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-2.1.0-SNAPSHOT
+2.1.0
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/ab5577f9/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index a0ce708..af18ac0 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2789

[Apache Jenkins Server]

See 

[2/2] beam git commit: [BEAM-2742] This closes #3699

[takidau]

[BEAM-2742] This closes #3699


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/926c70a3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/926c70a3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/926c70a3

Branch: refs/heads/DSL_SQL
Commit: 926c70a345912b084fbb9366e9ecf2ecc4450739
Parents: d7120f0 990fedc
Author: Tyler Akidau 
Authored: Tue Aug 8 11:18:49 2017 -0700
Committer: Tyler Akidau 
Committed: Tue Aug 8 11:18:49 2017 -0700

--
 .../apache/beam/sdk/coders/BeamRecordCoder.java |  4 +-
 .../org/apache/beam/sdk/values/BeamRecord.java  | 39 
 .../extensions/sql/impl/rel/BeamSortRel.java|  8 ++--
 3 files changed, 23 insertions(+), 28 deletions(-)
--

[1/2] beam git commit: [BEAM-2742] change Field type from primitives to boxed types

[takidau]

Repository: beam
Updated Branches:
  refs/heads/DSL_SQL d7120f077 -> 926c70a34


[BEAM-2742] change Field type from primitives to boxed types


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/990fedc9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/990fedc9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/990fedc9

Branch: refs/heads/DSL_SQL
Commit: 990fedc934da4eee2ec34c6a9ed5bd5a2dacbded
Parents: d7120f0
Author: James Xu 
Authored: Tue Aug 8 14:26:09 2017 +0800
Committer: Tyler Akidau 
Committed: Tue Aug 8 11:17:46 2017 -0700

--
 .../apache/beam/sdk/coders/BeamRecordCoder.java |  4 +-
 .../org/apache/beam/sdk/values/BeamRecord.java  | 39 
 .../extensions/sql/impl/rel/BeamSortRel.java|  8 ++--
 3 files changed, 23 insertions(+), 28 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/990fedc9/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BeamRecordCoder.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BeamRecordCoder.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BeamRecordCoder.java
index 40b9f3f..a6200f6 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BeamRecordCoder.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BeamRecordCoder.java
@@ -57,7 +57,7 @@ public class BeamRecordCoder extends CustomCoder {
   throws CoderException, IOException {
 nullListCoder.encode(scanNullFields(value), outStream);
 for (int idx = 0; idx < value.size(); ++idx) {
-  if (value.isNull(idx)) {
+  if (value.getFieldValue(idx) == null) {
 continue;
   }
 
@@ -87,7 +87,7 @@ public class BeamRecordCoder extends CustomCoder {
   private BitSet scanNullFields(BeamRecord record){
 BitSet nullFields = new BitSet(record.size());
 for (int idx = 0; idx < record.size(); ++idx) {
-  if (record.isNull(idx)) {
+  if (record.getFieldValue(idx) == null) {
 nullFields.set(idx);
   }
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/990fedc9/sdks/java/core/src/main/java/org/apache/beam/sdk/values/BeamRecord.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/BeamRecord.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/BeamRecord.java
index 6cbd11b..35a96f6 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/BeamRecord.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/BeamRecord.java
@@ -63,27 +63,27 @@ public class BeamRecord implements Serializable {
 return getFieldValue(dataType.getFieldsName().indexOf(fieldName));
   }
 
-  public byte getByte(String fieldName) {
+  public Byte getByte(String fieldName) {
 return (Byte) getFieldValue(fieldName);
   }
 
-  public short getShort(String fieldName) {
+  public Short getShort(String fieldName) {
 return (Short) getFieldValue(fieldName);
   }
 
-  public int getInteger(String fieldName) {
+  public Integer getInteger(String fieldName) {
 return (Integer) getFieldValue(fieldName);
   }
 
-  public float getFloat(String fieldName) {
+  public Float getFloat(String fieldName) {
 return (Float) getFieldValue(fieldName);
   }
 
-  public double getDouble(String fieldName) {
+  public Double getDouble(String fieldName) {
 return (Double) getFieldValue(fieldName);
   }
 
-  public long getLong(String fieldName) {
+  public Long getLong(String fieldName) {
 return (Long) getFieldValue(fieldName);
   }
 
@@ -103,35 +103,35 @@ public class BeamRecord implements Serializable {
 return (BigDecimal) getFieldValue(fieldName);
   }
 
-  public boolean getBoolean(String fieldName) {
-return (boolean) getFieldValue(fieldName);
+  public Boolean getBoolean(String fieldName) {
+return (Boolean) getFieldValue(fieldName);
   }
 
   public Object getFieldValue(int fieldIdx) {
 return dataValues.get(fieldIdx);
   }
 
-  public byte getByte(int idx) {
+  public Byte getByte(int idx) {
 return (Byte) getFieldValue(idx);
   }
 
-  public short getShort(int idx) {
+  public Short getShort(int idx) {
 return (Short) getFieldValue(idx);
   }
 
-  public int getInteger(int idx) {
+  public Integer getInteger(int idx) {
 return (Integer) getFieldValue(idx);
   }
 
-  public float getFloat(int idx) {
+  public Float getFloat(int idx) {
 return (Float) getFieldValue(idx);
   }
 
-  public double getDouble(int idx) {
+  public Double getDouble(int idx) {
 return (Double) getFieldValue(idx);
   }
 
-  public long getLong(int idx) {
+  public Long getLong(int idx) {
 return (Long) getFieldValue(idx);

[2/2] beam git commit: [BEAM-2733] This closes #3704

[takidau]

[BEAM-2733] This closes #3704


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d7120f07
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d7120f07
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d7120f07

Branch: refs/heads/DSL_SQL
Commit: d7120f07795973b9c782d180a2a9cc3c1affe658
Parents: 880531a 2a1fde3
Author: Tyler Akidau 
Authored: Tue Aug 8 11:13:05 2017 -0700
Committer: Tyler Akidau 
Committed: Tue Aug 8 11:13:05 2017 -0700

--
 .../org/apache/beam/sdk/extensions/sql/BeamSql.java | 16 +---
 1 file changed, 13 insertions(+), 3 deletions(-)
--

[1/2] beam git commit: [BEAM-2733] update javadoc for BeamSql

[takidau]

Repository: beam
Updated Branches:
  refs/heads/DSL_SQL 880531aa2 -> d7120f077


[BEAM-2733] update javadoc for BeamSql


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2a1fde3a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2a1fde3a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2a1fde3a

Branch: refs/heads/DSL_SQL
Commit: 2a1fde3aaf2bc54783fca18138bd7effba8c667e
Parents: 880531a
Author: James Xu 
Authored: Tue Aug 8 19:52:53 2017 +0800
Committer: Tyler Akidau 
Committed: Tue Aug 8 11:12:49 2017 -0700

--
 .../org/apache/beam/sdk/extensions/sql/BeamSql.java | 16 +---
 1 file changed, 13 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2a1fde3a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSql.java
--
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSql.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSql.java
index 86e4d8d..d0a6360 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSql.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSql.java
@@ -71,6 +71,7 @@ 
outputTableB.apply(...).apply(TextIO.write().to("/my/output/path"));
 p.run().waitUntilFinish();
  * }
  * 
+ *
  */
 @Experimental
 public class BeamSql {
@@ -82,8 +83,14 @@ public class BeamSql {
* table. The {@link PCollectionTuple} contains the mapping from {@code 
table names} to
* {@code PCollection}, each representing an input table.
*
-   * It is an error to apply a {@link PCollectionTuple} missing any {@code 
table names}
-   * referenced within the query.
+   * 
+   * If the sql query only uses a subset of tables from the upstream 
{@link PCollectionTuple},
+   * this is valid;
+   * If the sql query references a table not included in the upstream 
{@link PCollectionTuple},
+   * an {@code IllegalStateException} is thrown during query 
validation;
+   * Always, tables from the upstream {@link PCollectionTuple} are only 
valid in the scope
+   * of the current query call.
+   * 
*/
   public static QueryTransform query(String sqlQuery) {
 return QueryTransform.builder()
@@ -100,7 +107,7 @@ public class BeamSql {
*
* Make sure to query it from a static table name PCOLLECTION.
*/
-  public static SimpleQueryTransform simpleQuery(String sqlQuery) throws 
Exception {
+  public static SimpleQueryTransform simpleQuery(String sqlQuery) {
 return SimpleQueryTransform.builder()
 .setSqlEnv(new BeamSqlEnv())
 .setSqlQuery(sqlQuery)
@@ -109,6 +116,9 @@ public class BeamSql {
 
   /**
* A {@link PTransform} representing an execution plan for a SQL query.
+   *
+   * The table names in the input {@code PCollectionTuple} are only valid 
during the current
+   * query.
*/
   @AutoValue
   public abstract static class QueryTransform extends

[beam] Git Push Summary

[kenn]

Repository: beam
Updated Branches:
  refs/heads/mr-runner [created] 2fa4fdecd

[beam] Git Push Summary

[kenn]

Repository: beam
Updated Branches:
  refs/heads/tez-runner [created] 2fa4fdecd

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3730

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #4533

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2724) MSEC counters should support Structured Names in Dataflow

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118736#comment-16118736
 ] 

ASF GitHub Bot commented on BEAM-2724:
--

GitHub user pabloem reopened a pull request:

https://github.com/apache/beam/pull/3684

[BEAM-2724] Preparing support for Structured Names in Dataflow counters

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
---

* Adding a CounterName class to help support structured names for counters 
in the future.
* Also remove function calls from the `apiclient.py` file that are no 
longer used.
* And also removing function calls used for aggregators, which have been 
dead code for a while.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam add-countername

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3684.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3684






> MSEC counters should support Structured Names in Dataflow
> -
>
> Key: BEAM-2724
> URL: https://issues.apache.org/jira/browse/BEAM-2724
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2724) MSEC counters should support Structured Names in Dataflow

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118735#comment-16118735
 ] 

ASF GitHub Bot commented on BEAM-2724:
--

Github user pabloem closed the pull request at:

https://github.com/apache/beam/pull/3684


> MSEC counters should support Structured Names in Dataflow
> -
>
> Key: BEAM-2724
> URL: https://issues.apache.org/jira/browse/BEAM-2724
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3684: [BEAM-2724] Preparing support for Structured Names ...

[pabloem]

Github user pabloem closed the pull request at:

https://github.com/apache/beam/pull/3684


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3684: [BEAM-2724] Preparing support for Structured Names ...

[pabloem]

GitHub user pabloem reopened a pull request:

https://github.com/apache/beam/pull/3684

[BEAM-2724] Preparing support for Structured Names in Dataflow counters

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
---

* Adding a CounterName class to help support structured names for counters 
in the future.
* Also remove function calls from the `apiclient.py` file that are no 
longer used.
* And also removing function calls used for aggregators, which have been 
dead code for a while.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam add-countername

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3684.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3684






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2788

[Apache Jenkins Server]

See 

beam git commit: Fix compile break

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 89236e3b5 -> 2fa4fdecd


Fix compile break


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2fa4fdec
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2fa4fdec
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2fa4fdec

Branch: refs/heads/master
Commit: 2fa4fdecd8ef06534e369e527b15ae8193823c8b
Parents: 89236e3
Author: Kenneth Knowles 
Authored: Tue Aug 8 09:12:29 2017 -0700
Committer: Kenneth Knowles 
Committed: Tue Aug 8 09:12:29 2017 -0700

--
 .../runners/gearpump/translators/CreateStreamingGearpumpView.java | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2fa4fdec/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/CreateStreamingGearpumpView.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/CreateStreamingGearpumpView.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/CreateStreamingGearpumpView.java
index 60577b3..3ebe5c8 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/CreateStreamingGearpumpView.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/CreateStreamingGearpumpView.java
@@ -124,8 +124,7 @@ class CreateStreamingGearpumpView
 @Override
 public PCollection expand(PCollection input) {
   return PCollection.createPrimitiveOutputInternal(
-  input.getPipeline(), input.getWindowingStrategy(), 
input.isBounded())
-  .setCoder(input.getCoder());
+  input.getPipeline(), input.getWindowingStrategy(), 
input.isBounded(), input.getCoder());
 }
 
 public PCollectionView getView() {

[jira] [Assigned] (BEAM-2748) accept SerializableFunction as UDF

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reassigned BEAM-2748:


Assignee: Xu Mingmin

> accept SerializableFunction as UDF
> --
>
> Key: BEAM-2748
> URL: https://issues.apache.org/jira/browse/BEAM-2748
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge, dsl_sql_review
>
> Refer to 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhgo



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-2741) update JavaDoc for BeamRecord

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reassigned BEAM-2741:


Assignee: Xu Mingmin

> update JavaDoc for BeamRecord
> -
>
> Key: BEAM-2741
> URL: https://issues.apache.org/jira/browse/BEAM-2741
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge, dsl_sql_review
>
> Refer to 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhfw



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-2747) accept CombineFn as UDAF

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reassigned BEAM-2747:


Assignee: Xu Mingmin

> accept CombineFn as UDAF
> 
>
> Key: BEAM-2747
> URL: https://issues.apache.org/jira/browse/BEAM-2747
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge, dsl_sql_review
>
> refer to 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhgo



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2752) Job fails to checkpoint with kinesis stream as an input for Flink job

[Pawel Bartoszek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pawel Bartoszek updated BEAM-2752:
--
Description: 
Our job is reading from kinesis stream as a job input. Quiet often when the job 
is checkpointing for the first time the exception is thrown:

The scenario the produces the exception:

# Upload a new jar file with job logic
# Start new job
# Stop the job with savepoint that is written to s3
# Upload a new jar file with job logic(in this case the jar contains the same 
code - but our pipeline generates new jar file name for every build)
# Start a new job from savepoint
# The first checkpoint fails causing the job to be cancelled

If the job is started without passing savepoint the checkpointing works fine.

Other information:

Flink version 1.2.1
Beam 2.0.0
Flink Parallelism - 20 slots 
Number of task managers - 4
Number of kinesis shards - 8



{code:java}
java.lang.Exception: Error while triggering checkpoint 59 for Source: 
Read(KinesisSource) -> Flat Map -> ParMultiDo(KinesisExtractor) -> Flat Map -> 
ParMultiDo(StringToRecord) -> Flat Map -> ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ToRRecord) -> Flat Map -> ParMultiDo(AddTimestamps) -> Flat Map -> 
..GroupByOneMinuteWindow GROUP RDOTRECORDS BY ONE MINUTE 
WINDOWS/Window.Assign.out -> (ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ToSomeKey) -> Flat Map -> ToKeyedWorkItem, 
ParMultiDo(ToCompositeKey) -> Flat Map -> ParMultiDo(Anonymous) -> Flat Map -> 
ToKeyedWorkItem, ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ApplyShardingKey) -> Flat Map -> ToKeyedWorkItem) (1/20)
at org.apache.flink.runtime.taskmanager.Task$3.run(Task.java:1136)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception: Could not perform checkpoint 59 for operator 
Source: Read(KinesisSource) -> Flat Map -> ParMultiDo(KinesisExtractor) -> Flat 
Map -> ParMultiDo(StringToRecord) -> Flat Map -> ParMultiDo(Anonymous) -> Flat 
Map -> ParMultiDo(ToRRecord) -> Flat Map -> ParMultiDo(AddTimestamps) -> Flat 
Map -> ..GroupByOneMinuteWindow GROUP RDOTRECORDS BY ONE MINUTE 
WINDOWS/Window.Assign.out -> (ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ToSomeKey) -> Flat Map -> ToKeyedWorkItem, 
ParMultiDo(ToCompositeKey) -> Flat Map -> ParMultiDo(Anonymous) -> Flat Map -> 
ToKeyedWorkItem, ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ApplyShardingKey) -> Flat Map -> ToKeyedWorkItem) (1/20).
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:524)
at org.apache.flink.runtime.taskmanager.Task$3.run(Task.java:1125)
... 5 more
Caused by: java.lang.Exception: Could not complete snapshot 59 for operator 
Source: Read(KinesisSource) -> Flat Map -> ParMultiDo(KinesisExtractor) -> Flat 
Map -> ParMultiDo(StringToRecord) -> Flat Map -> ParMultiDo(Anonymous) -> Flat 
Map -> ParMultiDo(ToRRecord) -> Flat Map -> ParMultiDo(AddTimestamps) -> Flat 
Map -> ..GroupByOneMinuteWindow GROUP RDOTRECORDS BY ONE MINUTE 
WINDOWS/Window.Assign.out -> (ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ToSomeKey) -> Flat Map -> ToKeyedWorkItem, 
ParMultiDo(ToCompositeKey) -> Flat Map -> ParMultiDo(Anonymous) -> Flat Map -> 
ToKeyedWorkItem, ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ApplyShardingKey) -> Flat Map -> ToKeyedWorkItem) (1/20).
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:379)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.checkpointStreamOperator(StreamTask.java:1157)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing(StreamTask.java:1090)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(StreamTask.java:630)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:575)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:518)
... 6 more
Caused by: java.util.ConcurrentModificationException
at java.util.ArrayDeque$DeqIterator.next(ArrayDeque.java:643)
at 
org.apache.beam.sdks.java.io.kinesis.repackaged.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
at 
org.apache.beam.sdks.java.io.kinesis.repackaged.com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:409)
at 

[jira] [Commented] (BEAM-2671) CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark runner

[Aviem Zur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118451#comment-16118451
 ] 

Aviem Zur commented on BEAM-2671:
-

[~jkff] we've been running Spark runner in streaming mode using 2.0.0 
We've encountered the issue with the watermarks not advancing which was a no-go 
for us, since results were not emitted and eventually apps failed on out of 
memory exceptions. This was the reason for the fix in 
https://github.com/apache/beam/pull/3343

After integrating this fix (on our local fork) we were able to run applications 
successfully.
Regarding running the current state of master, we are not, so hard to comment 
on if the failing tests cause real applications to fail, this will have to be 
assessed. What we do know is the tests are failing.

Regarding ValidatesRunner tests, while these do pass, IIRC the creation of the 
streaming tests for Spark runner were in fact due to insufficient tests for 
Spark runner's streaming mode. So while they are not perfect and flaky, they do 
provide value. Perhaps [~amitsela] can elaborate on these tests.

> CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark 
> runner
> 
>
> Key: BEAM-2671
> URL: https://issues.apache.org/jira/browse/BEAM-2671
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Etienne Chauchot
>Assignee: Jean-Baptiste Onofré
> Fix For: 2.2.0
>
>
> Error message:
> Flatten.Iterables/FlattenIterables/FlatMap/ParMultiDo(Anonymous).out0: 
> Expected: iterable over [] in any order
>  but: Not matched: "late"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2752) Job fails to checkpoint with kinesis stream as an input for Flink job

[Pawel Bartoszek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pawel Bartoszek updated BEAM-2752:
--
Description: 
Our job is reading from kinesis stream as a job input. Quiet often when the job 
is checkpointing for the first time the exception is thrown:

The scenario the produces the exception:

# Upload a new jar file with job logic
# Start new job
# Stop the job with savepoint that is written to s3
# Upload a new jar file with job logic(in this case the jar contains the same 
code - but our pipeline generates new jar file name for every build)
# Start a new job from savepoint
# The first checkpoint fails causing the job to be cancelled

If the job is started without passing savepoint the checkpointing works fine.

Other information:

Flink version 1.2.1
Beam 2.0.0
Flink Parallelism - 20 slots 
Number of task managers - 4



{code:java}
java.lang.Exception: Error while triggering checkpoint 59 for Source: 
Read(KinesisSource) -> Flat Map -> ParMultiDo(KinesisExtractor) -> Flat Map -> 
ParMultiDo(StringToRecord) -> Flat Map -> ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ToRRecord) -> Flat Map -> ParMultiDo(AddTimestamps) -> Flat Map -> 
..GroupByOneMinuteWindow GROUP RDOTRECORDS BY ONE MINUTE 
WINDOWS/Window.Assign.out -> (ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ToSomeKey) -> Flat Map -> ToKeyedWorkItem, 
ParMultiDo(ToCompositeKey) -> Flat Map -> ParMultiDo(Anonymous) -> Flat Map -> 
ToKeyedWorkItem, ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ApplyShardingKey) -> Flat Map -> ToKeyedWorkItem) (1/20)
at org.apache.flink.runtime.taskmanager.Task$3.run(Task.java:1136)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception: Could not perform checkpoint 59 for operator 
Source: Read(KinesisSource) -> Flat Map -> ParMultiDo(KinesisExtractor) -> Flat 
Map -> ParMultiDo(StringToRecord) -> Flat Map -> ParMultiDo(Anonymous) -> Flat 
Map -> ParMultiDo(ToRRecord) -> Flat Map -> ParMultiDo(AddTimestamps) -> Flat 
Map -> ..GroupByOneMinuteWindow GROUP RDOTRECORDS BY ONE MINUTE 
WINDOWS/Window.Assign.out -> (ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ToSomeKey) -> Flat Map -> ToKeyedWorkItem, 
ParMultiDo(ToCompositeKey) -> Flat Map -> ParMultiDo(Anonymous) -> Flat Map -> 
ToKeyedWorkItem, ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ApplyShardingKey) -> Flat Map -> ToKeyedWorkItem) (1/20).
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:524)
at org.apache.flink.runtime.taskmanager.Task$3.run(Task.java:1125)
... 5 more
Caused by: java.lang.Exception: Could not complete snapshot 59 for operator 
Source: Read(KinesisSource) -> Flat Map -> ParMultiDo(KinesisExtractor) -> Flat 
Map -> ParMultiDo(StringToRecord) -> Flat Map -> ParMultiDo(Anonymous) -> Flat 
Map -> ParMultiDo(ToRRecord) -> Flat Map -> ParMultiDo(AddTimestamps) -> Flat 
Map -> ..GroupByOneMinuteWindow GROUP RDOTRECORDS BY ONE MINUTE 
WINDOWS/Window.Assign.out -> (ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ToSomeKey) -> Flat Map -> ToKeyedWorkItem, 
ParMultiDo(ToCompositeKey) -> Flat Map -> ParMultiDo(Anonymous) -> Flat Map -> 
ToKeyedWorkItem, ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ApplyShardingKey) -> Flat Map -> ToKeyedWorkItem) (1/20).
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:379)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.checkpointStreamOperator(StreamTask.java:1157)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing(StreamTask.java:1090)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(StreamTask.java:630)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:575)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:518)
... 6 more
Caused by: java.util.ConcurrentModificationException
at java.util.ArrayDeque$DeqIterator.next(ArrayDeque.java:643)
at 
org.apache.beam.sdks.java.io.kinesis.repackaged.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
at 
org.apache.beam.sdks.java.io.kinesis.repackaged.com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:409)
at 

[jira] [Created] (BEAM-2752) Job fails to checkpoint with kinesis stream as an input for Flink job

[Pawel Bartoszek (JIRA)]

Pawel Bartoszek created BEAM-2752:
-

 Summary: Job fails to checkpoint with kinesis stream as an input 
for Flink job
 Key: BEAM-2752
 URL: https://issues.apache.org/jira/browse/BEAM-2752
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Affects Versions: 2.0.0
Reporter: Pawel Bartoszek
Assignee: Davor Bonaci
Priority: Minor


Our job is reading from kinesis stream as a job input. Quiet often when the job 
is checkpointing for the first time the exception is thrown:


{code:java}
Our job is reading from kinesis stream as a job input. Quiet often when the job 
is checkpointing for the first time the exception is thrown.

The scenario the produces the exception:

# Upload a new jar file with job logic
# Start new job
# Stop the job with savepoint that is written to s3
# Upload a new jar file with job logic(in this case the jar contains the same 
code - but our pipeline generates new jar file name for every build)
# Start a new job from savepoint
# The first checkpoint fails causing the job to be cancelled

If the job is started without passing savepoint the checkpointing works fine.

Other information:

Flink version 1.2.1
Beam 2.0.0
Flink Parallelism - 20 slots 
Number of task managers - 4



{code:java}
java.lang.Exception: Error while triggering checkpoint 59 for Source: 
Read(KinesisSource) -> Flat Map -> ParMultiDo(KinesisExtractor) -> Flat Map -> 
ParMultiDo(StringToRecord) -> Flat Map -> ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ToRRecord) -> Flat Map -> ParMultiDo(AddTimestamps) -> Flat Map -> 
..GroupByOneMinuteWindow GROUP RDOTRECORDS BY ONE MINUTE 
WINDOWS/Window.Assign.out -> (ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ToSomeKey) -> Flat Map -> ToKeyedWorkItem, 
ParMultiDo(ToCompositeKey) -> Flat Map -> ParMultiDo(Anonymous) -> Flat Map -> 
ToKeyedWorkItem, ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ApplyShardingKey) -> Flat Map -> ToKeyedWorkItem) (1/20)
at org.apache.flink.runtime.taskmanager.Task$3.run(Task.java:1136)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception: Could not perform checkpoint 59 for operator 
Source: Read(KinesisSource) -> Flat Map -> ParMultiDo(KinesisExtractor) -> Flat 
Map -> ParMultiDo(StringToRecord) -> Flat Map -> ParMultiDo(Anonymous) -> Flat 
Map -> ParMultiDo(ToRRecord) -> Flat Map -> ParMultiDo(AddTimestamps) -> Flat 
Map -> ..GroupByOneMinuteWindow GROUP RDOTRECORDS BY ONE MINUTE 
WINDOWS/Window.Assign.out -> (ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ToSomeKey) -> Flat Map -> ToKeyedWorkItem, 
ParMultiDo(ToCompositeKey) -> Flat Map -> ParMultiDo(Anonymous) -> Flat Map -> 
ToKeyedWorkItem, ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ApplyShardingKey) -> Flat Map -> ToKeyedWorkItem) (1/20).
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:524)
at org.apache.flink.runtime.taskmanager.Task$3.run(Task.java:1125)
... 5 more
Caused by: java.lang.Exception: Could not complete snapshot 59 for operator 
Source: Read(KinesisSource) -> Flat Map -> ParMultiDo(KinesisExtractor) -> Flat 
Map -> ParMultiDo(StringToRecord) -> Flat Map -> ParMultiDo(Anonymous) -> Flat 
Map -> ParMultiDo(ToRRecord) -> Flat Map -> ParMultiDo(AddTimestamps) -> Flat 
Map -> ..GroupByOneMinuteWindow GROUP RDOTRECORDS BY ONE MINUTE 
WINDOWS/Window.Assign.out -> (ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ToSomeKey) -> Flat Map -> ToKeyedWorkItem, 
ParMultiDo(ToCompositeKey) -> Flat Map -> ParMultiDo(Anonymous) -> Flat Map -> 
ToKeyedWorkItem, ParMultiDo(Anonymous) -> Flat Map -> 
ParMultiDo(ApplyShardingKey) -> Flat Map -> ToKeyedWorkItem) (1/20).
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:379)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.checkpointStreamOperator(StreamTask.java:1157)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing(StreamTask.java:1090)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(StreamTask.java:630)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:575)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:518)
... 6 more
Caused by: java.util.ConcurrentModificationException
at 

[jira] [Commented] (BEAM-2671) CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark runner

[Etienne Chauchot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118355#comment-16118355
 ] 

Etienne Chauchot commented on BEAM-2671:


I think Nexmark (https://github.com/apache/beam/pull/3114) can be of great help 
here. It has a very comprehensive test suite in both batch and streaming modes. 
I'll run the Nexmark test suite for Spark streaming against v2.1.0-RC2 and keep 
you posted.

> CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark 
> runner
> 
>
> Key: BEAM-2671
> URL: https://issues.apache.org/jira/browse/BEAM-2671
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Etienne Chauchot
>Assignee: Jean-Baptiste Onofré
> Fix For: 2.2.0
>
>
> Error message:
> Flatten.Iterables/FlattenIterables/FlatMap/ParMultiDo(Anonymous).out0: 
> Expected: iterable over [] in any order
>  but: Not matched: "late"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3729

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2787

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2733) update JavaDoc of BeamSql

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118233#comment-16118233
 ] 

ASF GitHub Bot commented on BEAM-2733:
--

GitHub user xumingming opened a pull request:

https://github.com/apache/beam/pull/3704

[BEAM-2733] update javadoc for BeamSql

Summary:

* `Exception` declaration are removed from `BeamSql.query ()`
* Added javadoc to explain when the sql query use a non-exist table、a 
subset of tables from upstream, what happens.
* Added javadoc for BeamSql.query and QueryTransform to describe the 
lifetime of tables in PCollectionTuple.
* SimpleQueryTransform and QueryTransform are actually user-facing (they 
are return types of query() and simpleQuery()), so they are kept `public`.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xumingming/beam BEAM-2733-BeamSql

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3704.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3704


commit df4d24f2104b61b65bbfc027d005415d4282c992
Author: James Xu 
Date:   2017-08-08T11:52:53Z

[BEAM-2733] update javadoc for BeamSql




> update JavaDoc of BeamSql
> -
>
> Key: BEAM-2733
> URL: https://issues.apache.org/jira/browse/BEAM-2733
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>  Labels: dsl_sql_merge, dsl_sql_review
>
> refer to discussions:
> 1. 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmFcik
> 2. 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhfc
> 3. 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhfU
> 4. 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhfY
> 5. 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhfQ



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3704: [BEAM-2733] update javadoc for BeamSql

[xumingming]

GitHub user xumingming opened a pull request:

https://github.com/apache/beam/pull/3704

[BEAM-2733] update javadoc for BeamSql

Summary:

* `Exception` declaration are removed from `BeamSql.query ()`
* Added javadoc to explain when the sql query use a non-exist table、a 
subset of tables from upstream, what happens.
* Added javadoc for BeamSql.query and QueryTransform to describe the 
lifetime of tables in PCollectionTuple.
* SimpleQueryTransform and QueryTransform are actually user-facing (they 
are return types of query() and simpleQuery()), so they are kept `public`.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xumingming/beam BEAM-2733-BeamSql

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3704.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3704


commit df4d24f2104b61b65bbfc027d005415d4282c992
Author: James Xu 
Date:   2017-08-08T11:52:53Z

[BEAM-2733] update javadoc for BeamSql




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-2733) update JavaDoc of BeamSql

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu reassigned BEAM-2733:
--

Assignee: James Xu

> update JavaDoc of BeamSql
> -
>
> Key: BEAM-2733
> URL: https://issues.apache.org/jira/browse/BEAM-2733
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>  Labels: dsl_sql_merge, dsl_sql_review
>
> refer to discussions:
> 1. 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmFcik
> 2. 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhfc
> 3. 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhfU
> 4. 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhfY
> 5. 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhfQ



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-1637) Create Elasticsearch IO compatible with ES 5.x

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118125#comment-16118125
 ] 

ASF GitHub Bot commented on BEAM-1637:
--

GitHub user echauchot opened a pull request:

https://github.com/apache/beam/pull/3703

[BEAM-1637] Create Elasticsearch IO compatible with ES 5.x

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [X] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [X] Each commit in the pull request should have a meaningful subject 
line and body.
 - [X] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [X] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [X] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [X] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---
R: @jkff 
CC: @jbonofre 

Some comments about this pull request:

1. As discussed in the ML, the architecture is with a common module and 
modules per version (which differ in features but also in UTests). Modules per 
version use the same package name for backward compatibility (exactly same 
pipeline code). Classes in common package shall not be used directly by users. 
In a previous design, the common module had the same java package name than 
version modules to allow putting common classes package private. But I 
abandoned this design because of javadoc generation problems (no public classes 
in common module and no package exclusion possible otherwise no ES javadoc at 
all). So in the end, I just put common classes in a common package with public 
visibility and a javadoc warning stating that they shall not be used by 
pipeline authors. If you have a better suggestion, I'm all ear :)

2. I could not use inheritance because of statics so I used composition. If 
you have a better design, feel free to comment.

3. There is a very hacky thing in the JarHell class. The problem was that 
surefire dependencies entailed a duplicate class in the classpath which caused 
the jarHell detection to fail the build. Please read the javadoc of this class. 
If you have any other suggestion to avoid jarHell problem, feel free to comment.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/echauchot/beam BEAM-1637-ELASTICSEARCH-5

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3703.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3703


commit e623e8a529ab687b267685ef040e6059f61caa08
Author: Etienne Chauchot 
Date:   2017-06-26T08:58:21Z

[BEAM-1637] Create Elasticsearch IO compatible with ES 5.x




> Create Elasticsearch IO compatible with ES 5.x
> --
>
> Key: BEAM-1637
> URL: https://issues.apache.org/jira/browse/BEAM-1637
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Minor
>
> The current Elasticsearch IO (see 
> https://issues.apache.org/jira/browse/BEAM-425) is only compatible with 
> Elasticsearch v 2.x. The aim is to have an IO compatible with ES v 5.x. 
> Beyond being able to address v5.x elasticsearch instances, we could also 
> leverage the use of the Elasticsearch pipeline API and also better split the 
> dataset (be as close as possible of desiredBundleSize) thanks to the new ES 
> split API that allows ES shards splitting.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3703: [BEAM-1637] Create Elasticsearch IO compatible with...

[echauchot]

GitHub user echauchot opened a pull request:

https://github.com/apache/beam/pull/3703

[BEAM-1637] Create Elasticsearch IO compatible with ES 5.x

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [X] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [X] Each commit in the pull request should have a meaningful subject 
line and body.
 - [X] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [X] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [X] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [X] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---
R: @jkff 
CC: @jbonofre 

Some comments about this pull request:

1. As discussed in the ML, the architecture is with a common module and 
modules per version (which differ in features but also in UTests). Modules per 
version use the same package name for backward compatibility (exactly same 
pipeline code). Classes in common package shall not be used directly by users. 
In a previous design, the common module had the same java package name than 
version modules to allow putting common classes package private. But I 
abandoned this design because of javadoc generation problems (no public classes 
in common module and no package exclusion possible otherwise no ES javadoc at 
all). So in the end, I just put common classes in a common package with public 
visibility and a javadoc warning stating that they shall not be used by 
pipeline authors. If you have a better suggestion, I'm all ear :)

2. I could not use inheritance because of statics so I used composition. If 
you have a better design, feel free to comment.

3. There is a very hacky thing in the JarHell class. The problem was that 
surefire dependencies entailed a duplicate class in the classpath which caused 
the jarHell detection to fail the build. Please read the javadoc of this class. 
If you have any other suggestion to avoid jarHell problem, feel free to comment.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/echauchot/beam BEAM-1637-ELASTICSEARCH-5

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3703.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3703


commit e623e8a529ab687b267685ef040e6059f61caa08
Author: Etienne Chauchot 
Date:   2017-06-26T08:58:21Z

[BEAM-1637] Create Elasticsearch IO compatible with ES 5.x




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3728

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2743) rename size() in BeamRecord

[James Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117997#comment-16117997
 ] 

James Xu commented on BEAM-2743:


already covered by BEAM-2744

> rename size() in BeamRecord
> ---
>
> Key: BEAM-2743
> URL: https://issues.apache.org/jira/browse/BEAM-2743
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>  Labels: dsl_sql_merge, dsl_sql_review
>
> refer to 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTrrtwc



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2744) rename size() in BeamRecordType

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117995#comment-16117995
 ] 

ASF GitHub Bot commented on BEAM-2744:
--

GitHub user xumingming opened a pull request:

https://github.com/apache/beam/pull/3702

[BEAM-2744] rename BeamRecordType#size()

Summary:

* BeamRecord.size() -> BeamRecord.getFieldCount()
* BeamRecordType.size() -> BeamRecordType.getFieldCount()
* remove BeamRecord.valueInString()

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xumingming/beam 
BEAM-2744-rename-BeamRecordType-size

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3702.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3702


commit 6466a9fcfa0dcc4c73d05a98ffd8727584d190b4
Author: James Xu 
Date:   2017-08-08T07:15:59Z

[BEAM-2744] rename BeamRecordType#size()

commit ff10e1a5a64e9b7c607b965a9b5bc318465f1995
Author: James Xu 
Date:   2017-08-08T07:27:48Z

remove BeamRecord#valueInString()




> rename size() in BeamRecordType
> ---
>
> Key: BEAM-2744
> URL: https://issues.apache.org/jira/browse/BEAM-2744
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>  Labels: dsl_sql_merge, dsl_sql_review
>
> refer to 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTrrtwc



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3702: [BEAM-2744] rename BeamRecordType#size()

[xumingming]

GitHub user xumingming opened a pull request:

https://github.com/apache/beam/pull/3702

[BEAM-2744] rename BeamRecordType#size()

Summary:

* BeamRecord.size() -> BeamRecord.getFieldCount()
* BeamRecordType.size() -> BeamRecordType.getFieldCount()
* remove BeamRecord.valueInString()

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xumingming/beam 
BEAM-2744-rename-BeamRecordType-size

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3702.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3702


commit 6466a9fcfa0dcc4c73d05a98ffd8727584d190b4
Author: James Xu 
Date:   2017-08-08T07:15:59Z

[BEAM-2744] rename BeamRecordType#size()

commit ff10e1a5a64e9b7c607b965a9b5bc318465f1995
Author: James Xu 
Date:   2017-08-08T07:27:48Z

remove BeamRecord#valueInString()




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3701: Updates BEAM_CONTAINER_VERSION to 2.2.0.

[chamikaramj]

GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/3701

Updates BEAM_CONTAINER_VERSION to 2.2.0.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/beam update_container_version_2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3701.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3701


commit cc7b4da2f88c0e5fdfc27c0588d0cc66a489a928
Author: chamik...@google.com 
Date:   2017-08-08T06:47:57Z

Updates BEAM_CONTAINER_VERSION to 2.2.0.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2745) add getFieldTypeByIndex(index) in BeamRecordSqlType

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117936#comment-16117936
 ] 

ASF GitHub Bot commented on BEAM-2745:
--

GitHub user xumingming opened a pull request:

https://github.com/apache/beam/pull/3700

[BEAM-2745] add BeamRecordSqlType.getFieldTypeByIndex()

Summary:

* BeamSqlRecordType -> BeamRecordSqlType
* fieldsName -> fieldNames
* fieldsType -> fieldTypes
* add getFieldTypeByIndex()

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xumingming/beam 
BEAM-2745-add-getFieldTypeByIndex

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3700.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3700


commit cec323b1ddde3185082794dd420904eb89374bdb
Author: James Xu 
Date:   2017-08-08T06:53:18Z

[BEAM-2745] add BeamRecordSqlType.getFieldTypeByIndex()




> add getFieldTypeByIndex(index) in BeamRecordSqlType
> ---
>
> Key: BEAM-2745
> URL: https://issues.apache.org/jira/browse/BEAM-2745
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>  Labels: dsl_sql_merge, dsl_sql_review
>
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhgQ



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3700: [BEAM-2745] add BeamRecordSqlType.getFieldTypeByInd...

[xumingming]

GitHub user xumingming opened a pull request:

https://github.com/apache/beam/pull/3700

[BEAM-2745] add BeamRecordSqlType.getFieldTypeByIndex()

Summary:

* BeamSqlRecordType -> BeamRecordSqlType
* fieldsName -> fieldNames
* fieldsType -> fieldTypes
* add getFieldTypeByIndex()

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xumingming/beam 
BEAM-2745-add-getFieldTypeByIndex

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3700.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3700


commit cec323b1ddde3185082794dd420904eb89374bdb
Author: James Xu 
Date:   2017-08-08T06:53:18Z

[BEAM-2745] add BeamRecordSqlType.getFieldTypeByIndex()




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2786

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4531

[Apache Jenkins Server]

See 


--
[...truncated 1.21 MB...]
2017-08-08T06:28:40.471 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hive/hive-service-rpc/2.1.0/hive-service-rpc-2.1.0.jar
 (1503 KB at 584.6 KB/sec)
2017-08-08T06:28:40.471 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/slider/slider-core/0.90.2-incubating/slider-core-0.90.2-incubating.jar
2017-08-08T06:28:40.480 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hive/hive-llap-server/2.1.0/hive-llap-server-2.1.0.jar
 (545 KB at 211.2 KB/sec)
2017-08-08T06:28:40.480 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/beust/jcommander/1.30/jcommander-1.30.jar
2017-08-08T06:28:40.512 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/beust/jcommander/1.30/jcommander-1.30.jar
 (59 KB at 22.6 KB/sec)
2017-08-08T06:28:40.512 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-registry/2.7.1/hadoop-yarn-registry-2.7.1.jar
2017-08-08T06:28:40.532 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-hadoop2-compat/1.1.1/hbase-hadoop2-compat-1.1.1.jar
2017-08-08T06:28:40.557 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-registry/2.7.1/hadoop-yarn-registry-2.7.1.jar
 (96 KB at 36.1 KB/sec)
2017-08-08T06:28:40.557 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-server/1.1.1/hbase-server-1.1.1.jar
2017-08-08T06:28:40.597 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/parquet/parquet-hadoop-bundle/1.8.1/parquet-hadoop-bundle-1.8.1.jar
 (2835 KB at 1051.3 KB/sec)
2017-08-08T06:28:40.597 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-procedure/1.1.1/hbase-procedure-1.1.1.jar
2017-08-08T06:28:40.612 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-hadoop2-compat/1.1.1/hbase-hadoop2-compat-1.1.1.jar
 (80 KB at 29.3 KB/sec)
2017-08-08T06:28:40.612 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-common/1.1.1/hbase-common-1.1.1-tests.jar
2017-08-08T06:28:40.658 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-procedure/1.1.1/hbase-procedure-1.1.1.jar
 (100 KB at 36.1 KB/sec)
2017-08-08T06:28:40.658 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-prefix-tree/1.1.1/hbase-prefix-tree-1.1.1.jar
2017-08-08T06:28:40.668 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-common/1.1.1/hbase-common-1.1.1-tests.jar
 (206 KB at 74.2 KB/sec)
2017-08-08T06:28:40.668 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-hadoop-compat/1.1.1/hbase-hadoop-compat-1.1.1.jar
2017-08-08T06:28:40.712 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-hadoop-compat/1.1.1/hbase-hadoop-compat-1.1.1.jar
 (36 KB at 12.5 KB/sec)
2017-08-08T06:28:40.712 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/javax/servlet/jsp-api/2.0/jsp-api-2.0.jar
2017-08-08T06:28:40.714 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-prefix-tree/1.1.1/hbase-prefix-tree-1.1.1.jar
 (100 KB at 35.4 KB/sec)
2017-08-08T06:28:40.714 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jamon/jamon-runtime/2.3.1/jamon-runtime-2.3.1.jar
2017-08-08T06:28:40.719 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/slider/slider-core/0.90.2-incubating/slider-core-0.90.2-incubating.jar
 (1744 KB at 618.6 KB/sec)
2017-08-08T06:28:40.719 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/thrift/libthrift/0.9.3/libthrift-0.9.3.jar
2017-08-08T06:28:40.746 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/jamon/jamon-runtime/2.3.1/jamon-runtime-2.3.1.jar
 (21 KB at 7.1 KB/sec)
2017-08-08T06:28:40.747 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/javax/servlet/jsp-api/2.0/jsp-api-2.0.jar 
(50 KB at 17.3 KB/sec)
2017-08-08T06:28:40.775 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/thrift/libthrift/0.9.3/libthrift-0.9.3.jar
 (229 KB at 79.6 KB/sec)
2017-08-08T06:28:40.972 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-server/1.1.1/hbase-server-1.1.1.jar
 (3878 KB at 1262.6 KB/sec)
2017-08-08T06:28:41.226 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/it/unimi/dsi/fastutil/6.5.6/fastutil-6.5.6.jar
 (16502 KB at 4963.0 KB/sec)
2017-08-08T06:28:41.245 [INFO] Downloading: 
http://www.datanucleus.org/downloads/maven2/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
2017-08-08T06:28:41.545 [INFO] Downloading: 
http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar

[jira] [Commented] (BEAM-2742) Switch from primitives to Java types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117923#comment-16117923
 ] 

ASF GitHub Bot commented on BEAM-2742:
--

GitHub user xumingming opened a pull request:

https://github.com/apache/beam/pull/3699

[BEAM-2742] change Field type from primitives to boxed types

change Field type of `BeamRecord` from primitive types to boxed types, and 
removed `isNull` method.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xumingming/beam 
BEAM-2742-switch-to-boxed-types

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3699.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3699


commit 189959a65f5fc1f7835dcd2a99a76c7717860e07
Author: James Xu 
Date:   2017-08-08T06:26:09Z

[BEAM-2742] change Field type from primitives to boxed types




> Switch from primitives to Java types
> 
>
> Key: BEAM-2742
> URL: https://issues.apache.org/jira/browse/BEAM-2742
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>  Labels: dsl_sql_merge, dsl_sql_review
>
> Refer to 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTrrtwc



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3699: [BEAM-2742] change Field type from primitives to bo...

[xumingming]

GitHub user xumingming opened a pull request:

https://github.com/apache/beam/pull/3699

[BEAM-2742] change Field type from primitives to boxed types

change Field type of `BeamRecord` from primitive types to boxed types, and 
removed `isNull` method.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xumingming/beam 
BEAM-2742-switch-to-boxed-types

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3699.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3699


commit 189959a65f5fc1f7835dcd2a99a76c7717860e07
Author: James Xu 
Date:   2017-08-08T06:26:09Z

[BEAM-2742] change Field type from primitives to boxed types




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-2746) change visibility of BeamSqlTable

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu reassigned BEAM-2746:
--

Assignee: James Xu

> change visibility of BeamSqlTable
> -
>
> Key: BEAM-2746
> URL: https://issues.apache.org/jira/browse/BEAM-2746
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>  Labels: dsl_sql_merge, dsl_sql_review
>
> refer to 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhgQ



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-2740) update exposed methods in BeamSqlEnv

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu reassigned BEAM-2740:
--

Assignee: James Xu

> update exposed methods in BeamSqlEnv
> 
>
> Key: BEAM-2740
> URL: https://issues.apache.org/jira/browse/BEAM-2740
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>  Labels: dsl_sql_merge, dsl_sql_review
>
> Refer to
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmFcis
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmFciw



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-2745) add getFieldTypeByIndex(index) in BeamRecordSqlType

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu reassigned BEAM-2745:
--

Assignee: James Xu

> add getFieldTypeByIndex(index) in BeamRecordSqlType
> ---
>
> Key: BEAM-2745
> URL: https://issues.apache.org/jira/browse/BEAM-2745
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>  Labels: dsl_sql_merge, dsl_sql_review
>
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhgQ



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-2744) rename size() in BeamRecordType

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu reassigned BEAM-2744:
--

Assignee: James Xu

> rename size() in BeamRecordType
> ---
>
> Key: BEAM-2744
> URL: https://issues.apache.org/jira/browse/BEAM-2744
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>  Labels: dsl_sql_merge, dsl_sql_review
>
> refer to 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTrrtwc



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-2743) rename size() in BeamRecord

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu reassigned BEAM-2743:
--

Assignee: James Xu

> rename size() in BeamRecord
> ---
>
> Key: BEAM-2743
> URL: https://issues.apache.org/jira/browse/BEAM-2743
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>  Labels: dsl_sql_merge, dsl_sql_review
>
> refer to 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTrrtwc



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PerformanceTests_JDBC #190

[Apache Jenkins Server]

See 


--
[...truncated 25.37 KB...]
[INFO] Replacing 

 with 

[INFO] Replacing original test artifact with shaded test artifact.
[INFO] Replacing 

 with 

[INFO] Dependency-reduced POM written at: 

[INFO] 
[INFO] --- maven-failsafe-plugin:2.20:integration-test (integration-test) @ 
beam-runners-google-cloud-dataflow-java ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-dependency-plugin:3.0.1:analyze-only (default) @ 
beam-runners-google-cloud-dataflow-java ---
[INFO] No dependency problems found
[INFO] 
[INFO] --- maven-failsafe-plugin:2.20:verify (integration-test) @ 
beam-runners-google-cloud-dataflow-java ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ 
beam-runners-google-cloud-dataflow-java ---
[INFO] Installing 

 to 
/home/jenkins/.m2/repository/org/apache/beam/beam-runners-google-cloud-dataflow-java/2.2.0-SNAPSHOT/beam-runners-google-cloud-dataflow-java-2.2.0-SNAPSHOT.jar
[INFO] Installing 

 to 
/home/jenkins/.m2/repository/org/apache/beam/beam-runners-google-cloud-dataflow-java/2.2.0-SNAPSHOT/beam-runners-google-cloud-dataflow-java-2.2.0-SNAPSHOT.pom
[INFO] Installing 

 to 
/home/jenkins/.m2/repository/org/apache/beam/beam-runners-google-cloud-dataflow-java/2.2.0-SNAPSHOT/beam-runners-google-cloud-dataflow-java-2.2.0-SNAPSHOT-tests.jar
[INFO] Installing 

 to 
/home/jenkins/.m2/repository/org/apache/beam/beam-runners-google-cloud-dataflow-java/2.2.0-SNAPSHOT/beam-runners-google-cloud-dataflow-java-2.2.0-SNAPSHOT-tests.jar
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 24.829 s
[INFO] Finished at: 2017-08-08T06:00:44Z
[INFO] Final Memory: 90M/1411M
[INFO] 
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins6586181238128069560.sh
+ /home/jenkins/tools/maven/latest/bin/mvn -B -e verify -pl sdks/java/io/jdbc 
-Dio-it-suite 
-DpkbLocation=
 -DmvnBinary=/home/jenkins/tools/maven/latest/bin/mvn 
'-DintegrationTestPipelineOptions=[ "--project=apache-beam-testing", 
"--tempRoot=gs://temp-storage-for-end-to-end-tests" ]'
[INFO] Error stacktraces are turned on.
[INFO] Scanning for projects...
[INFO] 
[INFO] Detecting the operating system and CPU architecture
[INFO] 
[INFO] os.detected.name: linux
[INFO] os.detected.arch: x86_64
[INFO] os.detected.version: 3.19
[INFO] os.detected.version.major: 3
[INFO] os.detected.version.minor: 19
[INFO] os.detected.release: ubuntu
[INFO] os.detected.release.version: 14.04
[INFO] os.detected.release.like.ubuntu: true
[INFO] os.detected.release.like.debian: true
[INFO] os.detected.classifier: linux-x86_64
[INFO] 
[INFO] 
[INFO] Building Apache Beam :: SDKs :: Java :: IO :: JDBC 2.2.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce) @ 
beam-sdks-java-io-jdbc ---
[INFO] 
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-banned-dependencies) 
@