[jira] [Commented] (BEAM-2203) Arithmetic operators: support DATETIME & DATETIME_INTERVAL

[James Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011762#comment-16011762
 ] 

James Xu commented on BEAM-2203:


`+`, `-` has not been implemented for Date: 
https://calcite.apache.org/docs/reference.html#datetime-functions

> Arithmetic operators: support DATETIME & DATETIME_INTERVAL
> --
>
> Key: BEAM-2203
> URL: https://issues.apache.org/jira/browse/BEAM-2203
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: James Xu
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-2247) support date/time functions

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu reassigned BEAM-2247:
--

Assignee: James Xu

> support date/time functions
> ---
>
> Key: BEAM-2247
> URL: https://issues.apache.org/jira/browse/BEAM-2247
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
>
> https://calcite.apache.org/docs/reference.html#datetime-functions



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2247) support date/time functions

[James Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011759#comment-16011759
 ] 

James Xu commented on BEAM-2247:


I will take this one.

> support date/time functions
> ---
>
> Key: BEAM-2247
> URL: https://issues.apache.org/jira/browse/BEAM-2247
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
>
> https://calcite.apache.org/docs/reference.html#datetime-functions



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2297) Set Opearator: UNION

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011752#comment-16011752
 ] 

ASF GitHub Bot commented on BEAM-2297:
--

GitHub user xumingming opened a pull request:

https://github.com/apache/beam/pull/3162

[BEAM-2297] support UNION

support UNION

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xumingming/beam BEAM-2297-union

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3162.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3162


commit d540d78359f28a8e7a38af22d69f1136c778bd74
Author: James Xu 
Date:   2017-05-16T05:10:56Z

support UNION




> Set Opearator: UNION
> 
>
> Key: BEAM-2297
> URL: https://issues.apache.org/jira/browse/BEAM-2297
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3162: [BEAM-2297] support UNION

[xumingming]

GitHub user xumingming opened a pull request:

https://github.com/apache/beam/pull/3162

[BEAM-2297] support UNION

support UNION

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xumingming/beam BEAM-2297-union

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3162.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3162


commit d540d78359f28a8e7a38af22d69f1136c778bd74
Author: James Xu 
Date:   2017-05-16T05:10:56Z

support UNION




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

svn commit: r19682 - in /dev/beam: 2.0.0-RC1/ 2.0.0-RC2/ 2.0.0-RC3/ 2.0.0-RC4/

[dhalperi]

Author: dhalperi
Date: Tue May 16 05:02:25 2017
New Revision: 19682

Log:
Remove all release candidates for Apache Beam, version 2.0.0

Following the release of RC4 as version 2.0.0

Removed:
dev/beam/2.0.0-RC1/
dev/beam/2.0.0-RC2/
dev/beam/2.0.0-RC3/
dev/beam/2.0.0-RC4/

svn commit: r19681 - /release/beam/2.0.0/

[dhalperi]

Author: dhalperi
Date: Tue May 16 05:01:41 2017
New Revision: 19681

Log:
Release Apache Beam, version 2.0.0


Added:
release/beam/2.0.0/
release/beam/2.0.0/apache-beam-2.0.0-python.zip   (with props)
release/beam/2.0.0/apache-beam-2.0.0-python.zip.asc
release/beam/2.0.0/apache-beam-2.0.0-python.zip.md5
release/beam/2.0.0/apache-beam-2.0.0-python.zip.sha1
release/beam/2.0.0/apache-beam-2.0.0-source-release.zip   (with props)
release/beam/2.0.0/apache-beam-2.0.0-source-release.zip.asc
release/beam/2.0.0/apache-beam-2.0.0-source-release.zip.md5
release/beam/2.0.0/apache-beam-2.0.0-source-release.zip.sha1

Added: release/beam/2.0.0/apache-beam-2.0.0-python.zip
==
Binary file - no diff available.

Propchange: release/beam/2.0.0/apache-beam-2.0.0-python.zip
--
svn:mime-type = application/octet-stream

Added: release/beam/2.0.0/apache-beam-2.0.0-python.zip.asc
==
--- release/beam/2.0.0/apache-beam-2.0.0-python.zip.asc (added)
+++ release/beam/2.0.0/apache-beam-2.0.0-python.zip.asc Tue May 16 05:01:41 2017
@@ -0,0 +1,11 @@
+-BEGIN PGP SIGNATURE-
+Version: GnuPG v1
+
+iQEcBAABAgAGBQJZFo3pAAoJEMkEN+GPDTNPIL8H/Aw98I6RFpLcIEITnTh1EZeG
+ZQZR5OPWF2uZJiseYrcJKeh/V33Kphnlj0w2CH4emHDgMLTQhuAC50pFif82OVEm
+Ogq1BykjZsvZAbSN3me83TD5jqZaKfbzcQOP1Q3QHI1j92OVRhGBohgU7BUEnA4I
+ekHeWkFxpudo6HstE+o+Xfhps5Fh6Ucm1USVrN94apGAcdmgPHKxcjeV9zG0EMYG
+i5z0FmXhG13jTfWEpd3qkPqlGeHXuDCopEH0e++HOMo7EELnvQsLhsdxko7WA4sM
+6MCSMugUGc0EmKI3anxu8kfyzIERZl3CVucwIgxECVYlHCUG8TsSVJOZ/Y6d5Ak=
+=wtZK
+-END PGP SIGNATURE-

Added: release/beam/2.0.0/apache-beam-2.0.0-python.zip.md5
==
--- release/beam/2.0.0/apache-beam-2.0.0-python.zip.md5 (added)
+++ release/beam/2.0.0/apache-beam-2.0.0-python.zip.md5 Tue May 16 05:01:41 2017
@@ -0,0 +1 @@
+ec2eee1939284afa3993ea4c55cea8f3  apache-beam-2.0.0-python.zip

Added: release/beam/2.0.0/apache-beam-2.0.0-python.zip.sha1
==
--- release/beam/2.0.0/apache-beam-2.0.0-python.zip.sha1 (added)
+++ release/beam/2.0.0/apache-beam-2.0.0-python.zip.sha1 Tue May 16 05:01:41 
2017
@@ -0,0 +1 @@
+eae09f8bd00264ab297f93c28929fb381daad5d4  apache-beam-2.0.0-python.zip

Added: release/beam/2.0.0/apache-beam-2.0.0-source-release.zip
==
Binary file - no diff available.

Propchange: release/beam/2.0.0/apache-beam-2.0.0-source-release.zip
--
svn:mime-type = application/octet-stream

Added: release/beam/2.0.0/apache-beam-2.0.0-source-release.zip.asc
==
--- release/beam/2.0.0/apache-beam-2.0.0-source-release.zip.asc (added)
+++ release/beam/2.0.0/apache-beam-2.0.0-source-release.zip.asc Tue May 16 
05:01:41 2017
@@ -0,0 +1,11 @@
+-BEGIN PGP SIGNATURE-
+Version: GnuPG v1
+
+iQEcBAABAgAGBQJZFo4CAAoJEMkEN+GPDTNPWYEH/0U1cAyS8tMQDHSIBKws2eHx
++HestkX3b5Skl8s6PanJXGxnnGvGRCaVkDgIyhK56OKtcMikhAPRyvbuCr8dvrog
+lme8fjHoEqPgfKdkyMJyMuva25vhDAa30aX33UkC3hU/+9Mmk+nIJZ8as+/H1zan
++vK0YTnPeivwe+csnygxKtAj2WvQgohHbXC4nm6Hrx1G1pawojVZKECrPdE4EPu3
+lv44btdkcLN1XcWi/Mt7SJhQQJUC4nCuLAMR1sTVXr6992EagjYx7lIAh3/gGnX4
+iDCXR/ZNEfdYZ5kTpZ+GSVdALxjv4/+07rsdIvkBLLUw8dCvksk6cLueUO0VyWk=
+=PjzK
+-END PGP SIGNATURE-

Added: release/beam/2.0.0/apache-beam-2.0.0-source-release.zip.md5
==
--- release/beam/2.0.0/apache-beam-2.0.0-source-release.zip.md5 (added)
+++ release/beam/2.0.0/apache-beam-2.0.0-source-release.zip.md5 Tue May 16 
05:01:41 2017
@@ -0,0 +1 @@
+6931d1ad79aed5d25330da27af0c792e  apache-beam-2.0.0-source-release.zip

Added: release/beam/2.0.0/apache-beam-2.0.0-source-release.zip.sha1
==
--- release/beam/2.0.0/apache-beam-2.0.0-source-release.zip.sha1 (added)
+++ release/beam/2.0.0/apache-beam-2.0.0-source-release.zip.sha1 Tue May 16 
05:01:41 2017
@@ -0,0 +1 @@
+4713e8c44021cd9ec8b06d93aa6b4d86a704c3c6  apache-beam-2.0.0-source-release.zip

[beam] Git Push Summary

[dhalperi]

Repository: beam
Updated Tags:  refs/tags/v2.0.0 [created] 2392ce8b1

[GitHub] beam pull request #3161: [BEAM-2302] Add spilling code to WriteFiles.

[reuvenlax]

GitHub user reuvenlax opened a pull request:

https://github.com/apache/beam/pull/3161

[BEAM-2302] Add spilling code to WriteFiles.

This is similar to the fix for BEAM-2154.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reuvenlax/incubator-beam 
windowed_file_scalability

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3161.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3161


commit 675b4fa342e11a437c648f48a890e06b9b42e9bb
Author: Reuven Lax 
Date:   2017-05-13T19:53:08Z

Add spilling code to WriteFiles.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2302) WriteFiles with runner-determined sharding and large numbers of windows causes OOM errors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011733#comment-16011733
 ] 

ASF GitHub Bot commented on BEAM-2302:
--

GitHub user reuvenlax opened a pull request:

https://github.com/apache/beam/pull/3161

[BEAM-2302] Add spilling code to WriteFiles.

This is similar to the fix for BEAM-2154.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reuvenlax/incubator-beam 
windowed_file_scalability

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3161.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3161


commit 675b4fa342e11a437c648f48a890e06b9b42e9bb
Author: Reuven Lax 
Date:   2017-05-13T19:53:08Z

Add spilling code to WriteFiles.




> WriteFiles with runner-determined sharding and large numbers of windows 
> causes OOM errors
> -
>
> Key: BEAM-2302
> URL: https://issues.apache.org/jira/browse/BEAM-2302
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Davor Bonaci
>
> This is because the WriteWindowedBundles transform will create many file 
> writers, and the sheer number of file buffers (which defaults to 64mb per 
> writer) uses up all memory. The fix is the same as was done in BigQueryIO - 
> if too many writers are opened, spill into a shuffle, and write the files 
> after the shuffle



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2838

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-93) Support Compute Engine Subnetworks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011731#comment-16011731
 ] 

ASF GitHub Bot commented on BEAM-93:


Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3154


> Support Compute Engine Subnetworks
> --
>
> Key: BEAM-93
> URL: https://issues.apache.org/jira/browse/BEAM-93
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Sam McVeety
>Assignee: Sam McVeety
>Priority: Minor
> Fix For: 0.1.0-incubating
>
>
> The Dataflow runner for Beam should support Compute Engine subnetworks for 
> the workers: https://cloud.google.com/compute/docs/subnetworks



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: Adding support for subnetwork in Python Pipelineoptions

[altay]

Repository: beam
Updated Branches:
  refs/heads/master aad85553b -> 529f251b8


Adding support for subnetwork in Python Pipelineoptions


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/902f27a3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/902f27a3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/902f27a3

Branch: refs/heads/master
Commit: 902f27a34cf65923823d10c0edf11ba27e883c30
Parents: aad8555
Author: Pablo 
Authored: Mon May 15 15:47:18 2017 -0700
Committer: Ahmet Altay 
Committed: Mon May 15 21:32:46 2017 -0700

--
 .../apache_beam/options/pipeline_options.py | 11 +-
 .../runners/dataflow/internal/apiclient.py  |  2 ++
 .../runners/dataflow/internal/apiclient_test.py | 21 
 3 files changed, 33 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/902f27a3/sdks/python/apache_beam/options/pipeline_options.py
--
diff --git a/sdks/python/apache_beam/options/pipeline_options.py 
b/sdks/python/apache_beam/options/pipeline_options.py
index 983d128..777926a 100644
--- a/sdks/python/apache_beam/options/pipeline_options.py
+++ b/sdks/python/apache_beam/options/pipeline_options.py
@@ -169,7 +169,7 @@ class PipelineOptions(HasDisplayData):
 """Returns a PipelineOptions from a dictionary of arguments.
 
 Args:
-  options: Dictinary of argument value pairs.
+  options: Dictionary of argument value pairs.
 
 Returns:
   A PipelineOptions object representing the given arguments.
@@ -455,6 +455,15 @@ class WorkerOptions(PipelineOptions):
 'GCE network for launching workers. Default is up to the Dataflow '
 'service.'))
 parser.add_argument(
+'--subnetwork',
+default=None,
+help=(
+'GCE subnetwork for launching workers. Default is up to the '
+'Dataflow service. Expected format is '
+'regions/REGION/subnetworks/SUBNETWORK or the fully qualified '
+'subnetwork name. For more information, see '
+'https://cloud.google.com/compute/docs/vpc/'))
+parser.add_argument(
 '--worker_harness_container_image',
 default=None,
 help=('Docker registry location of container image to use for the '

http://git-wip-us.apache.org/repos/asf/beam/blob/902f27a3/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py 
b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
index ea49593..bfdd5e4 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
@@ -199,6 +199,8 @@ class Environment(object):
   pool.zone = self.worker_options.zone
 if self.worker_options.network:
   pool.network = self.worker_options.network
+if self.worker_options.subnetwork:
+  pool.subnetwork = self.worker_options.subnetwork
 if self.worker_options.worker_harness_container_image:
   pool.workerHarnessContainerImage = (
   self.worker_options.worker_harness_container_image)

http://git-wip-us.apache.org/repos/asf/beam/blob/902f27a3/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py
--
diff --git 
a/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py 
b/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py
index 6ed1fb4..67cf77f 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py
@@ -44,6 +44,27 @@ class UtilTest(unittest.TestCase):
 pipeline_options,
 DataflowRunner.BATCH_ENVIRONMENT_MAJOR_VERSION)
 
+  def test_set_network(self):
+pipeline_options = PipelineOptions(
+['--network', 'anetworkname',
+ '--temp_location', 'gs://any-location/temp'])
+env = apiclient.Environment([], #packages
+pipeline_options,
+'2.0.0') #any environment version
+self.assertEqual(env.proto.workerPools[0].network,
+ 'anetworkname')
+
+  def test_set_subnetwork(self):
+pipeline_options = PipelineOptions(
+['--subnetwork', '/regions/MY/subnetworks/SUBNETWORK',
+ '--temp_location', 'gs://any-location/temp'])
+
+env = apiclient.Environment([], #packages
+pipeline_options,
+'2.0.0') #any environment version
+

[jira] [Created] (BEAM-2302) WriteFiles with runner-determined sharding and large numbers of windows causes OOM errors

[Reuven Lax (JIRA)]

Reuven Lax created BEAM-2302:


 Summary: WriteFiles with runner-determined sharding and large 
numbers of windows causes OOM errors
 Key: BEAM-2302
 URL: https://issues.apache.org/jira/browse/BEAM-2302
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Reuven Lax
Assignee: Davor Bonaci


This is because the WriteWindowedBundles transform will create many file 
writers, and the sheer number of file buffers (which defaults to 64mb per 
writer) uses up all memory. The fix is the same as was done in BigQueryIO - if 
too many writers are opened, spill into a shuffle, and write the files after 
the shuffle



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #245: Website updates for version 2.0.0: downloads, J...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam-site/pull/245

Website updates for version 2.0.0: downloads, Javadoc, Pydoc

This takes over from #231 with manual fixes for RC4.

This closes #231.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam-site takeover-231

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/245.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #245


commit 31e6d57379c680486ea970d2c00ad9e91cb285c5
Author: Davor Bonaci 
Date:   2017-05-07T03:19:06Z

Add Javadoc for 2.0.0 (release candidate 1)

commit 13ccbb04a432d3a715ef213f4b7745c8cb3920eb
Author: Davor Bonaci 
Date:   2017-05-07T03:22:40Z

Add Pydoc for version 2.0.0 (release candidate 1)

commit caaaf61f3b84484c028f61fd042897ec62f126a1
Author: Davor Bonaci 
Date:   2017-05-07T03:23:53Z

Update latest version metadata information

commit bdd871af79170fc4ab25427735029f673be6a508
Author: Davor Bonaci 
Date:   2017-05-07T03:28:31Z

Add version 2.0.0 to Downloads page

commit 0dd2d923499e465de4e826a18afe6b2d6c107279
Author: Davor Bonaci 
Date:   2017-05-12T07:52:28Z

Update Javadoc for version 2.0.0 (release candidate 3)

commit 45080dcbbcf12529f1674e86a9ad44b1c7c9c692
Author: Davor Bonaci 
Date:   2017-05-12T07:55:21Z

Update Pydoc for version 2.0.0 (release candidate 3)

commit e9658a5c396a87e1a6040a4f32f20ce7baee29c0
Author: Davor Bonaci 
Date:   2017-05-13T04:45:17Z

Updating Javadoc for Apache Beam, version 2.0.0, release candidate 4

commit de93142292a1a2aef5d06ff40c7107f12c056bf8
Author: Davor Bonaci 
Date:   2017-05-13T04:47:24Z

Update Pydoc for Apache Beam, version 2.0.0, release candidate #4

commit d4285b5613fe989aeebc0df406100a2385b620f8
Author: Dan Halperin 
Date:   2017-05-16T04:15:13Z

Javadoc overview: manually add 2.0.0 header

commit 26be316693fc3f1d887cfafbedd6ad121721d54d
Author: Dan Halperin 
Date:   2017-05-16T04:27:31Z

Python: fix links to PipelineOptions after move from utils to options module




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3160: [TRIVIAL] runners-core: delete placeholder

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/3160

[TRIVIAL] runners-core: delete placeholder

This was used to enable checkin of an empty directory, but the directory
is no longer empty.

R: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam fixup-placeholder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3160.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3160


commit 5def8881e55fc75ba8d0256a8c95c2bcd7979060
Author: Dan Halperin 
Date:   2017-05-16T04:00:05Z

[TRIVIAL] runners-core: delete placeholder

This was used to enable checkin of an empty directory, but the directory
is no longer empty.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2837

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #3822

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3155: Use built-in cmp python function in comparing Datas...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3155


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3155

[altay]

This closes #3155


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/aad85553
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/aad85553
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/aad85553

Branch: refs/heads/master
Commit: aad85553bf7e60c430790fd20745fcc968c90a02
Parents: 754d5d8 0a17c3d
Author: Ahmet Altay 
Authored: Mon May 15 18:42:05 2017 -0700
Committer: Ahmet Altay 
Committed: Mon May 15 18:42:05 2017 -0700

--
 .../io/gcp/datastore/v1/fake_datastore.py   |  8 +++--
 .../apache_beam/io/gcp/datastore/v1/helper.py   | 14 ++--
 .../io/gcp/datastore/v1/query_splitter_test.py  | 37 +++-
 3 files changed, 30 insertions(+), 29 deletions(-)
--

[1/2] beam git commit: Use built-in cmp python function in comparing datastore paths

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 754d5d833 -> aad85553b


Use built-in cmp python function in comparing datastore paths


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0a17c3da
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0a17c3da
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0a17c3da

Branch: refs/heads/master
Commit: 0a17c3da5a249dbe4408c5dce7e1825897865e1e
Parents: 754d5d8
Author: Vikas Kedigehalli 
Authored: Mon May 15 15:50:25 2017 -0700
Committer: Ahmet Altay 
Committed: Mon May 15 18:42:00 2017 -0700

--
 .../io/gcp/datastore/v1/fake_datastore.py   |  8 +++--
 .../apache_beam/io/gcp/datastore/v1/helper.py   | 14 ++--
 .../io/gcp/datastore/v1/query_splitter_test.py  | 37 +++-
 3 files changed, 30 insertions(+), 29 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/0a17c3da/sdks/python/apache_beam/io/gcp/datastore/v1/fake_datastore.py
--
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1/fake_datastore.py 
b/sdks/python/apache_beam/io/gcp/datastore/v1/fake_datastore.py
index 2332579..aa37805 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1/fake_datastore.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1/fake_datastore.py
@@ -90,13 +90,17 @@ def create_response(entities, end_cursor, finish):
   return resp
 
 
-def create_entities(count):
+def create_entities(count, id_or_name=False):
   """Creates a list of entities with random keys."""
   entities = []
 
   for _ in range(count):
 entity_result = query_pb2.EntityResult()
-entity_result.entity.key.path.add().name = str(uuid.uuid4())
+if id_or_name:
+  entity_result.entity.key.path.add().id = (
+  uuid.uuid4().int & ((1 << 63) - 1))
+else:
+  entity_result.entity.key.path.add().name = str(uuid.uuid4())
 entities.append(entity_result)
 
   return entities

http://git-wip-us.apache.org/repos/asf/beam/blob/0a17c3da/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py
--
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py 
b/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py
index 9e2c053..f977536 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py
@@ -80,7 +80,7 @@ def compare_path(p1, p2):
   3. If no `id` is defined for both paths, then their `names` are compared.
   """
 
-  result = str_compare(p1.kind, p2.kind)
+  result = cmp(p1.kind, p2.kind)
   if result != 0:
 return result
 
@@ -88,20 +88,12 @@ def compare_path(p1, p2):
 if not p2.HasField('id'):
   return -1
 
-return p1.id - p2.id
+return cmp(p1.id, p2.id)
 
   if p2.HasField('id'):
 return 1
 
-  return str_compare(p1.name, p2.name)
-
-
-def str_compare(s1, s2):
-  if s1 == s2:
-return 0
-  elif s1 < s2:
-return -1
-  return 1
+  return cmp(p1.name, p2.name)
 
 
 def get_datastore(project):

http://git-wip-us.apache.org/repos/asf/beam/blob/0a17c3da/sdks/python/apache_beam/io/gcp/datastore/v1/query_splitter_test.py
--
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1/query_splitter_test.py 
b/sdks/python/apache_beam/io/gcp/datastore/v1/query_splitter_test.py
index b7b054f..52f25fa 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1/query_splitter_test.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1/query_splitter_test.py
@@ -157,27 +157,32 @@ class QuerySplitterTest(unittest.TestCase):
   batch_size: the number of entities returned by fake datastore in one req.
 """
 
-entities = fake_datastore.create_entities(num_entities)
-mock_datastore = MagicMock()
-# Assign a fake run_query method as a side_effect to the mock.
-mock_datastore.run_query.side_effect = \
-fake_datastore.create_run_query(entities, batch_size)
+# Test for both random long ids and string ids.
+id_or_name = [True, False]
 
-split_queries = query_splitter.get_splits(mock_datastore, query, 
num_splits)
+for id_type in id_or_name:
+  entities = fake_datastore.create_entities(num_entities, id_type)
+  mock_datastore = MagicMock()
+  # Assign a fake run_query method as a side_effect to the mock.
+  mock_datastore.run_query.side_effect = \
+  fake_datastore.create_run_query(entities, batch_size)
 
-# if request num_splits is greater than num_entities, the best it can
-# do is one entity per split.
-expected_num_splits = min(num_splits, num_entities + 1)
-self.assertEqual(len(split_queries), expected_num_splits)
+  split_queries = 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2836

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2252) maven-shade-plugin should be defined in pluginManagement instead of plugins so that the plugin execution order defined by the root pom.xml is used

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011559#comment-16011559
 ] 

ASF GitHub Bot commented on BEAM-2252:
--

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/3159

[BEAM-2252] Use resource transformer plugin to use versions defined in root 
pom.xml within maven archetype poms

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`.
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam beam2253

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3159.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3159


commit d90fe13c0da4f34860f3574ab1d586a4fb418417
Author: Luke Cwik 
Date:   2017-05-15T22:43:00Z

[BEAM-2253] Use resource transformer plugin to use versions defined in root 
pom.xml within maven archetype poms




> maven-shade-plugin should be defined in pluginManagement instead of plugins 
> so that the plugin execution order defined by the root pom.xml is used
> --
>
> Key: BEAM-2252
> URL: https://issues.apache.org/jira/browse/BEAM-2252
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow, runner-direct, sdk-java-core, 
> sdk-java-gcp
>Affects Versions: 2.1.0
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
>  Labels: newbie, starter
>
> Several of our child pom.xml declare maven-shade-plugin within build/plugins 
> instead of build/pluginManagement which affects the plugin execution order.
> See BEAM-2242 for when we surgically fixed how shading was performed and this 
> issue is about revisiting that change and improving upon it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3159: [BEAM-2252] Use resource transformer plugin to use ...

[lukecwik]

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/3159

[BEAM-2252] Use resource transformer plugin to use versions defined in root 
pom.xml within maven archetype poms

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`.
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam beam2253

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3159.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3159


commit d90fe13c0da4f34860f3574ab1d586a4fb418417
Author: Luke Cwik 
Date:   2017-05-15T22:43:00Z

[BEAM-2253] Use resource transformer plugin to use versions defined in root 
pom.xml within maven archetype poms




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #2835

[Apache Jenkins Server]

See 

[GitHub] beam pull request #2855: [BEAM-1888] ApexRunner SDF support

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2855


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1888) Support Splittable DoFn in Apex Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011553#comment-16011553
 ] 

ASF GitHub Bot commented on BEAM-1888:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2855


> Support Splittable DoFn in Apex Runner
> --
>
> Key: BEAM-1888
> URL: https://issues.apache.org/jira/browse/BEAM-1888
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-apex
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

beam git commit: This closes #2855

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master f5efca029 -> 754d5d833


This closes #2855


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/754d5d83
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/754d5d83
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/754d5d83

Branch: refs/heads/master
Commit: 754d5d833f15d3095257cc129a061e1ea3267502
Parents: 69522fe f5efca0
Author: Eugene Kirpichov 
Authored: Mon May 15 17:36:16 2017 -0700
Committer: Eugene Kirpichov 
Committed: Mon May 15 17:36:16 2017 -0700

--
 .../apache/beam/runners/apex/ApexRunner.java|  34 +
 .../translation/ApexPipelineTranslator.java |  23 
 .../apex/translation/ParDoTranslator.java   |  68 +-
 .../operators/ApexGroupByKeyOperator.java   |   4 +-
 .../operators/ApexParDoOperator.java| 124 ++-
 .../operators/ApexProcessFnOperator.java|   8 +-
 .../operators/ApexTimerInternals.java   |  21 +++-
 .../translation/utils/ApexStateInternals.java   |   4 +
 .../translation/utils/StateInternalsProxy.java  |   7 +-
 .../operators/ApexTimerInternalsTest.java   |   8 +-
 10 files changed, 286 insertions(+), 15 deletions(-)
--

beam git commit: ApexRunner SDF support

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master 69522fe66 -> f5efca029


ApexRunner SDF support


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f5efca02
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f5efca02
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f5efca02

Branch: refs/heads/master
Commit: f5efca0292eb9ace2d6b4895bc08e94344854336
Parents: 69522fe
Author: Thomas Weise 
Authored: Sat Apr 8 13:58:48 2017 -0700
Committer: Eugene Kirpichov 
Committed: Mon May 15 17:04:20 2017 -0700

--
 .../apache/beam/runners/apex/ApexRunner.java|  34 +
 .../translation/ApexPipelineTranslator.java |  23 
 .../apex/translation/ParDoTranslator.java   |  68 +-
 .../operators/ApexGroupByKeyOperator.java   |   4 +-
 .../operators/ApexParDoOperator.java| 124 ++-
 .../operators/ApexProcessFnOperator.java|   8 +-
 .../operators/ApexTimerInternals.java   |  21 +++-
 .../translation/utils/ApexStateInternals.java   |   4 +
 .../translation/utils/StateInternalsProxy.java  |   7 +-
 .../operators/ApexTimerInternalsTest.java   |   8 +-
 10 files changed, 286 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f5efca02/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
index 366308e..2fd0b22 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
@@ -31,6 +31,7 @@ import java.net.URISyntaxException;
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.List;
+import java.util.Map;
 import java.util.Properties;
 import java.util.concurrent.atomic.AtomicReference;
 import org.apache.apex.api.EmbeddedAppLauncher;
@@ -38,9 +39,11 @@ import org.apache.apex.api.Launcher;
 import org.apache.apex.api.Launcher.AppHandle;
 import org.apache.apex.api.Launcher.LaunchMode;
 import org.apache.beam.runners.apex.translation.ApexPipelineTranslator;
+import org.apache.beam.runners.core.SplittableParDo;
 import org.apache.beam.runners.core.construction.PTransformMatchers;
 import org.apache.beam.runners.core.construction.PTransformReplacements;
 import org.apache.beam.runners.core.construction.PrimitiveCreate;
+import org.apache.beam.runners.core.construction.ReplacementOutputs;
 import 
org.apache.beam.runners.core.construction.SingleInputOutputOverrideFactory;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.PipelineRunner;
@@ -51,18 +54,23 @@ import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsValidator;
 import org.apache.beam.sdk.runners.AppliedPTransform;
 import org.apache.beam.sdk.runners.PTransformOverride;
+import org.apache.beam.sdk.runners.PTransformOverrideFactory;
 import org.apache.beam.sdk.transforms.Combine;
 import org.apache.beam.sdk.transforms.Combine.GloballyAsSingletonView;
 import org.apache.beam.sdk.transforms.Create;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.ParDo.MultiOutput;
 import org.apache.beam.sdk.transforms.View;
 import org.apache.beam.sdk.transforms.View.AsIterable;
 import org.apache.beam.sdk.transforms.View.AsSingleton;
 import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.PCollectionViews;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.TupleTag;
 import org.apache.hadoop.conf.Configuration;
 
 /**
@@ -112,6 +120,10 @@ public class ApexRunner extends 
PipelineRunner {
 PTransformOverride.of(
 
PTransformMatchers.classEqualTo(Combine.GloballyAsSingletonView.class),
 new StreamingCombineGloballyAsSingletonView.Factory()))
+.add(
+PTransformOverride.of(
+PTransformMatchers.splittableParDoMulti(),
+new SplittableParDoOverrideFactory<>()))
 .build();
   }
 
@@ -424,4 +436,26 @@ public class ApexRunner extends 
PipelineRunner {
 }
   }
 
+  /**
+   * A {@link PTransformOverrideFactory} that overrides a
+   * https://s.apache.org/splittable-do-fn;>Splittable DoFn with
+   * {@link SplittableParDo}.
+   */
+  static class SplittableParDoOverrideFactory implements 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #15

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-2299] Run maven install on Windows machine for build/test coverage

[sourabhbajaj] Fix documentation for the shard_template_name

--
[...truncated 1.99 MB...]
2017-05-16T00:21:33.631 [INFO] Excluding 
io.dropwizard.metrics:metrics-core:jar:3.1.0 from the shaded jar.
2017-05-16T00:21:33.631 [INFO] Excluding 
io.dropwizard.metrics:metrics-jvm:jar:3.1.0 from the shaded jar.
2017-05-16T00:21:33.631 [INFO] Excluding 
com.clearspring.analytics:stream:jar:2.5.2 from the shaded jar.
2017-05-16T00:21:33.631 [INFO] Excluding it.unimi.dsi:fastutil:jar:6.5.7 from 
the shaded jar.
2017-05-16T00:21:33.631 [INFO] Excluding org.apache.thrift:libthrift:jar:0.9.2 
from the shaded jar.
2017-05-16T00:21:33.631 [INFO] Excluding io.netty:netty-all:jar:4.0.39.Final 
from the shaded jar.
2017-05-16T00:21:33.631 [INFO] Excluding org.ow2.asm:asm:jar:5.0.3 from the 
shaded jar.
2017-05-16T00:21:33.631 [INFO] Excluding commons-io:commons-io:jar:2.4 from the 
shaded jar.
2017-05-16T00:21:33.631 [INFO] Excluding 
org.apache.beam:beam-sdks-java-core:jar:2.1.0-SNAPSHOT from the shaded jar.
2017-05-16T00:21:33.631 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-05-16T00:21:33.631 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-05-16T00:21:33.631 [INFO] Excluding net.bytebuddy:byte-buddy:jar:1.6.8 
from the shaded jar.
2017-05-16T00:21:33.631 [INFO] Excluding 
org.apache.logging.log4j:log4j-api:jar:2.6.2 from the shaded jar.
2017-05-16T00:21:36.235 [INFO] Replacing original artifact with shaded artifact.
2017-05-16T00:21:36.235 [INFO] Replacing 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT.jar
 with 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-shaded.jar
2017-05-16T00:21:36.235 [INFO] Replacing original test artifact with shaded 
test artifact.
2017-05-16T00:21:36.235 [INFO] Replacing 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-tests.jar
 with 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-shaded-tests.jar
2017-05-16T00:21:36.421 [INFO] 
2017-05-16T00:21:36.421 [INFO] --- maven-javadoc-plugin:2.10.4:jar 
(attach-javadocs) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-16T00:21:36.606 [INFO] 
2017-05-16T00:21:36.607 [INFO] --- maven-source-plugin:3.0.1:jar-no-fork 
(attach-sources) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-16T00:21:36.614 [INFO] Building jar: 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-sources.jar
2017-05-16T00:21:36.804 [INFO] 
2017-05-16T00:21:36.804 [INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork 
(attach-test-sources) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-16T00:21:36.811 [INFO] Building jar: 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-test-sources.jar
2017-05-16T00:21:37.010 [INFO] 
2017-05-16T00:21:37.010 [INFO] --- maven-dependency-plugin:3.0.0:analyze-only 
(default) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-16T00:21:37.526 [INFO] No dependency problems found
2017-05-16T00:21:37.707 [INFO] 
2017-05-16T00:21:37.707 [INFO] --- apache-rat-plugin:0.12:check (default) @ 
beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-16T00:21:37.725 [INFO] Enabled default license matchers.
2017-05-16T00:21:37.725 [INFO] Will parse SCM ignores for exclusions...
2017-05-16T00:21:37.726 [INFO] Finished adding exclusions from SCM ignore files.
2017-05-16T00:21:37.726 [INFO] 61 implicit excludes (use -debug for more 
details).
2017-05-16T00:21:37.726 [INFO] Exclude: **/target/**/*
2017-05-16T00:21:37.726 [INFO] Exclude: **/dependency-reduced-pom.xml
2017-05-16T00:21:37.726 [INFO] Exclude: **/hs_err_pid*.log
2017-05-16T00:21:37.726 [INFO] Exclude: .github/**/*
2017-05-16T00:21:37.726 [INFO] Exclude: **/*.iml
2017-05-16T00:21:37.726 [INFO] Exclude: **/.idea/**/*
2017-05-16T00:21:37.726 [INFO] Exclude: **/*.egg-info/**/*
2017-05-16T00:21:37.726 [INFO] Exclude: **/package-list
2017-05-16T00:21:37.726 [INFO] Exclude: **/user.avsc
2017-05-16T00:21:37.726 [INFO] Exclude: **/test/resources/**/*.txt
2017-05-16T00:21:37.726 [INFO] Exclude: **/test/**/.placeholder
2017-05-16T00:21:37.726 

[jira] [Commented] (BEAM-1158) Create HiveIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011536#comment-16011536
 ] 

ASF GitHub Bot commented on BEAM-1158:
--

GitHub user seshadri-cr opened a pull request:

https://github.com/apache/beam/pull/3158

[BEAM-1158]Hive integration test for HadoopInputFormatIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/seshadri-cr/beam hiveintegrationtest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3158.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3158


commit 3276fd8572c4beff4857fbd41abb5e88fe3c004c
Author: Seshadri Chakkravarthy 
Date:   2017-05-16T00:15:42Z

Hive integration test for HadoopInputFormatIO




> Create HiveIO
> -
>
> Key: BEAM-1158
> URL: https://issues.apache.org/jira/browse/BEAM-1158
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Affects Versions: Not applicable
>Reporter: Ismaël Mejía
>Assignee: Seshadri Raghunathan
>Priority: Minor
>
> Support for reading and writing to Hive. The HiveIO will directly use the 
> native API and HiveQL



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3158: [BEAM-1158]Hive integration test for HadoopInputFor...

[seshadri-cr]

GitHub user seshadri-cr opened a pull request:

https://github.com/apache/beam/pull/3158

[BEAM-1158]Hive integration test for HadoopInputFormatIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/seshadri-cr/beam hiveintegrationtest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3158.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3158


commit 3276fd8572c4beff4857fbd41abb5e88fe3c004c
Author: Seshadri Chakkravarthy 
Date:   2017-05-16T00:15:42Z

Hive integration test for HadoopInputFormatIO




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2834

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1265) Add streaming support to Python DirectRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011527#comment-16011527
 ] 

ASF GitHub Bot commented on BEAM-1265:
--

GitHub user charlesccychen opened a pull request:

https://github.com/apache/beam/pull/3157

[BEAM-1265] Do not merge: streaming scratch work

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/charlesccychen/beam streaming-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3157.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3157


commit f78e47ca47e19ac2684c13bf13dbb66e65f04048
Author: Charles Chen 
Date:   2017-04-19T01:20:07Z

streaming test

commit dd3966a572b73ffa9421a176cc03f096707cdb0d
Author: Charles Chen 
Date:   2017-04-20T05:54:02Z

Working TestStream evaluator

commit 68b5a84595f4ca9adc131fe2fcee38f91078a047
Author: Charles Chen 
Date:   2017-04-20T08:59:05Z

Slightly better watermark propagation, debugging print cleanup.

commit 48d4f1ef68829e98e964070f60ce9470528fcb0e
Author: Charles Chen 
Date:   2017-04-20T09:44:19Z

Factor out GroupAlsoByWindows so that it can be overridden by the 
DirectRunner.

commit b5e7073b66c02f634a4bdec8f787b38bb923d52a
Author: Charles Chen 
Date:   2017-04-21T05:51:27Z

Don't buffer elements in GroupByKeyOnly.

commit f81c2d0b536e2715fe4cd278359eed5c6a855dc5
Author: Charles Chen 
Date:   2017-04-21T17:33:03Z

Skeleton implementation of GroupAlsoByKeyEvaluator.

commit 05bd9c677c89d3d3413c205a2965ec0cc6827786
Author: Charles Chen 
Date:   2017-04-22T08:32:25Z

Mostly working streaming GroupAlsoByWindow.

commit f9197ff807d27e61eef337b261f826d5f4b43908
Author: Charles Chen 
Date:   2017-04-24T08:58:58Z

Introduce keyed bundles, KeyedWorkItems.

commit c4b6faf9a4b8231616ec35ec2fd766b863e7a76e
Author: Charles Chen 
Date:   2017-04-24T15:16:05Z

Rename old non-keyed state to legacy_state, introduce KeyedWorkItem, 
prepare to expose DirectUnmergedState API.




> Add streaming support to Python DirectRunner
> 
>
> Key: BEAM-1265
> URL: https://issues.apache.org/jira/browse/BEAM-1265
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Charles Chen
>
> Continue the work started in https://issues.apache.org/jira/browse/BEAM-428



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3157: [BEAM-1265] Do not merge: streaming scratch work

[charlesccychen]

GitHub user charlesccychen opened a pull request:

https://github.com/apache/beam/pull/3157

[BEAM-1265] Do not merge: streaming scratch work

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/charlesccychen/beam streaming-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3157.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3157


commit f78e47ca47e19ac2684c13bf13dbb66e65f04048
Author: Charles Chen 
Date:   2017-04-19T01:20:07Z

streaming test

commit dd3966a572b73ffa9421a176cc03f096707cdb0d
Author: Charles Chen 
Date:   2017-04-20T05:54:02Z

Working TestStream evaluator

commit 68b5a84595f4ca9adc131fe2fcee38f91078a047
Author: Charles Chen 
Date:   2017-04-20T08:59:05Z

Slightly better watermark propagation, debugging print cleanup.

commit 48d4f1ef68829e98e964070f60ce9470528fcb0e
Author: Charles Chen 
Date:   2017-04-20T09:44:19Z

Factor out GroupAlsoByWindows so that it can be overridden by the 
DirectRunner.

commit b5e7073b66c02f634a4bdec8f787b38bb923d52a
Author: Charles Chen 
Date:   2017-04-21T05:51:27Z

Don't buffer elements in GroupByKeyOnly.

commit f81c2d0b536e2715fe4cd278359eed5c6a855dc5
Author: Charles Chen 
Date:   2017-04-21T17:33:03Z

Skeleton implementation of GroupAlsoByKeyEvaluator.

commit 05bd9c677c89d3d3413c205a2965ec0cc6827786
Author: Charles Chen 
Date:   2017-04-22T08:32:25Z

Mostly working streaming GroupAlsoByWindow.

commit f9197ff807d27e61eef337b261f826d5f4b43908
Author: Charles Chen 
Date:   2017-04-24T08:58:58Z

Introduce keyed bundles, KeyedWorkItems.

commit c4b6faf9a4b8231616ec35ec2fd766b863e7a76e
Author: Charles Chen 
Date:   2017-04-24T15:16:05Z

Rename old non-keyed state to legacy_state, introduce KeyedWorkItem, 
prepare to expose DirectUnmergedState API.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3821

[Apache Jenkins Server]

See 


--
[...truncated 432.69 KB...]
at 
org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:785)
at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at 
org.jvnet.hudson.maven3.launcher.Maven33Launcher.main(Maven33Launcher.java:129)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
at jenkins.maven3.agent.Maven33Main.launch(Maven33Main.java:176)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at hudson.maven.Maven3Builder.call(Maven3Builder.java:139)
at hudson.maven.Maven3Builder.call(Maven3Builder.java:70)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:336)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

[WARNING] Cannot calculate digest of mojo class, because mojo wasn't loaded 
from a jar, but from: 

[JENKINS] Recording test results
2017-05-16T00:07:49.087 [INFO] 
2017-05-16T00:07:49.087 [INFO] --- jacoco-maven-plugin:0.7.8:report (report) @ 
beam-sdks-java-core ---
2017-05-16T00:07:49.188 [INFO] Skipping JaCoCo execution due to missing 
execution data file.
[WARNING] Cannot calculate digest of mojo class, because mojo wasn't loaded 
from a jar, but from: 

2017-05-16T00:07:49.278 [INFO] 
2017-05-16T00:07:49.278 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-sdks-java-core ---
[WARNING] Cannot calculate digest of mojo class, because mojo wasn't loaded 
from a jar, but from: 

2017-05-16T00:07:49.379 [INFO] 
2017-05-16T00:07:49.380 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-sdks-java-core ---
2017-05-16T00:07:49.552 [INFO] Building jar: 

[WARNING] Cannot calculate digest of mojo class, because mojo wasn't loaded 
from a jar, but from: 

2017-05-16T00:07:49.736 [INFO] 
2017-05-16T00:07:49.736 [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ beam-sdks-java-core ---
[WARNING] 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2833

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3153: Fix documentation for the shard_template_name

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3153


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Fix documentation for the shard_template_name

[chamikara]

Repository: beam
Updated Branches:
  refs/heads/master 13e1be2f0 -> 69522fe66


Fix documentation for the shard_template_name


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b3f74044
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b3f74044
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b3f74044

Branch: refs/heads/master
Commit: b3f74044436f4b4a92f3135411b5d5093437ec3e
Parents: 13e1be2
Author: Sourabh Bajaj 
Authored: Mon May 15 14:34:45 2017 -0700
Committer: Sourabh Bajaj 
Committed: Mon May 15 14:34:45 2017 -0700

--
 sdks/python/apache_beam/io/avroio.py | 13 ++---
 sdks/python/apache_beam/io/textio.py | 13 ++---
 sdks/python/apache_beam/io/tfrecordio.py | 15 +++
 3 files changed, 19 insertions(+), 22 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b3f74044/sdks/python/apache_beam/io/avroio.py
--
diff --git a/sdks/python/apache_beam/io/avroio.py 
b/sdks/python/apache_beam/io/avroio.py
index e02e1f7..7df9983 100644
--- a/sdks/python/apache_beam/io/avroio.py
+++ b/sdks/python/apache_beam/io/avroio.py
@@ -312,13 +312,12 @@ class WriteToAvro(beam.transforms.PTransform):
 the performance of a pipeline.  Setting this value is not recommended
 unless you require a specific number of output files.
   shard_name_template: A template string containing placeholders for
-the shard number and shard count. Currently only '' and
-'-S-of-N' are patterns accepted by the service.
-When constructing a filename for a particular shard number, the
-upper-case letters 'S' and 'N' are replaced with the 0-padded shard
-number and shard count respectively.  This argument can be '' in which
-case it behaves as if num_shards was set to 1 and only one file will be
-generated. The default pattern used is '-S-of-N'.
+the shard number and shard count. When constructing a filename for a
+particular shard number, the upper-case letters 'S' and 'N' are
+replaced with the 0-padded shard number and shard count respectively.
+This argument can be '' in which case it behaves as if num_shards was
+set to 1 and only one file will be generated. The default pattern used
+is '-S-of-N' if None is passed as the shard_name_template.
   mime_type: The MIME type to use for the produced files, if the filesystem
 supports specifying MIME types.
 

http://git-wip-us.apache.org/repos/asf/beam/blob/b3f74044/sdks/python/apache_beam/io/textio.py
--
diff --git a/sdks/python/apache_beam/io/textio.py 
b/sdks/python/apache_beam/io/textio.py
index eeefaf6..60e1512 100644
--- a/sdks/python/apache_beam/io/textio.py
+++ b/sdks/python/apache_beam/io/textio.py
@@ -291,13 +291,12 @@ class _TextSink(filebasedsink.FileBasedSink):
 the performance of a pipeline.  Setting this value is not recommended
 unless you require a specific number of output files.
   shard_name_template: A template string containing placeholders for
-the shard number and shard count. Currently only '' and
-'-S-of-N' are patterns accepted by the service.
-When constructing a filename for a particular shard number, the
-upper-case letters 'S' and 'N' are replaced with the 0-padded shard
-number and shard count respectively.  This argument can be '' in which
-case it behaves as if num_shards was set to 1 and only one file will be
-generated. The default pattern used is '-S-of-N'.
+the shard number and shard count. When constructing a filename for a
+particular shard number, the upper-case letters 'S' and 'N' are
+replaced with the 0-padded shard number and shard count respectively.
+This argument can be '' in which case it behaves as if num_shards was
+set to 1 and only one file will be generated. The default pattern used
+is '-S-of-N' if None is passed as the shard_name_template.
   coder: Coder used to encode each line.
   compression_type: Used to handle compressed output files. Typical value
 is CompressionTypes.AUTO, in which case the final file path's

http://git-wip-us.apache.org/repos/asf/beam/blob/b3f74044/sdks/python/apache_beam/io/tfrecordio.py
--
diff --git a/sdks/python/apache_beam/io/tfrecordio.py 
b/sdks/python/apache_beam/io/tfrecordio.py
index a8cd1ce..d7eb932 100644
--- a/sdks/python/apache_beam/io/tfrecordio.py
+++ 

[2/2] beam git commit: This closes #3153

[chamikara]

This closes #3153


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/69522fe6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/69522fe6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/69522fe6

Branch: refs/heads/master
Commit: 69522fe66e45ad64ee7aa0419edc71c055cfadce
Parents: 13e1be2 b3f7404
Author: chamik...@google.com 
Authored: Mon May 15 16:23:27 2017 -0700
Committer: chamik...@google.com 
Committed: Mon May 15 16:23:27 2017 -0700

--
 sdks/python/apache_beam/io/avroio.py | 13 ++---
 sdks/python/apache_beam/io/textio.py | 13 ++---
 sdks/python/apache_beam/io/tfrecordio.py | 15 +++
 3 files changed, 19 insertions(+), 22 deletions(-)
--

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #3138

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2301) Standard expansion of SDF should be in runners-core-construction

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011483#comment-16011483
 ] 

ASF GitHub Bot commented on BEAM-2301:
--

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/3156

[BEAM-2301] Splits SplittableParDo into a core-construction part and a 
runners-core part

SplittableParDo itself goes into core-construction, and expands into a 
slightly different transform.

This change is almost completely simply moving code around.

Before:
```
elements: InputT
| pair with restriction -> ElementAndRestriction
| split restriction -> same
| explode windows -> same
| assign unique key -> KV>
| GBKIntoKeyedWorkItems -> KeyedWorkItem>
| ProcessElements -> PCollection
```

After:
```
elements: InputT
| ...
| assign unique key -> KV>
| SplittableProcessKeyed -> PCollection
```

Most runners (except Dataflow) will still want to go through KeyedWorkItem. 
That part is encapsulated in `SplittableParDoViaKeyedWorkItems`, which has an 
`OverrideFactory` for `SplittableProcessKeyed` expanding it into the good old 
`GBKIntoKeyedWorkItems` and `ProcessElements`. So runner changes are very minor.

Dataflow, however, can not use runners-core during expansion, so it will 
translate `SplittableProcessKeyed` directly and perform its expansion 
service-side, and will instantiate `ProcessFn` worker-side.

R: @tgroh 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam sdf-expansion

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3156.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3156


commit be95bdd679fba755785a8e35a87eb1ec6c440882
Author: Eugene Kirpichov 
Date:   2017-05-15T22:54:03Z

Splits SplittableParDo into a core-construction part and a KWI-related part




> Standard expansion of SDF should be in runners-core-construction
> 
>
> Key: BEAM-2301
> URL: https://issues.apache.org/jira/browse/BEAM-2301
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>
> As should standard expansions of everything else.
> Since SplittableParDo (the standard expansion of SDF) uses KeyedWorkItem and 
> other things in runners-core that are not available in 
> runners-core-construction, it needs to be refactored somewhat.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3156: [BEAM-2301] Splits SplittableParDo into a core-cons...

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/3156

[BEAM-2301] Splits SplittableParDo into a core-construction part and a 
runners-core part

SplittableParDo itself goes into core-construction, and expands into a 
slightly different transform.

This change is almost completely simply moving code around.

Before:
```
elements: InputT
| pair with restriction -> ElementAndRestriction
| split restriction -> same
| explode windows -> same
| assign unique key -> KV>
| GBKIntoKeyedWorkItems -> KeyedWorkItem>
| ProcessElements -> PCollection
```

After:
```
elements: InputT
| ...
| assign unique key -> KV>
| SplittableProcessKeyed -> PCollection
```

Most runners (except Dataflow) will still want to go through KeyedWorkItem. 
That part is encapsulated in `SplittableParDoViaKeyedWorkItems`, which has an 
`OverrideFactory` for `SplittableProcessKeyed` expanding it into the good old 
`GBKIntoKeyedWorkItems` and `ProcessElements`. So runner changes are very minor.

Dataflow, however, can not use runners-core during expansion, so it will 
translate `SplittableProcessKeyed` directly and perform its expansion 
service-side, and will instantiate `ProcessFn` worker-side.

R: @tgroh 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam sdf-expansion

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3156.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3156


commit be95bdd679fba755785a8e35a87eb1ec6c440882
Author: Eugene Kirpichov 
Date:   2017-05-15T22:54:03Z

Splits SplittableParDo into a core-construction part and a KWI-related part




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2301) Standard expansion of SDF should be in runners-core-construction

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-2301:
--

 Summary: Standard expansion of SDF should be in 
runners-core-construction
 Key: BEAM-2301
 URL: https://issues.apache.org/jira/browse/BEAM-2301
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Eugene Kirpichov


As should standard expansions of everything else.

Since SplittableParDo (the standard expansion of SDF) uses KeyedWorkItem and 
other things in runners-core that are not available in 
runners-core-construction, it needs to be refactored somewhat.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3155: Use built-in cmp python function in comparing Datas...

[vikkyrk]

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/3155

Use built-in cmp python function in comparing Datastore paths

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam ds_py

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3155.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3155


commit 2ede86de42b005d7d393ac24dd6a6f3ed9093235
Author: Vikas Kedigehalli 
Date:   2017-05-15T22:50:25Z

Use built-in cmp python function in comparing datastore paths




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-93) Support Compute Engine Subnetworks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011460#comment-16011460
 ] 

ASF GitHub Bot commented on BEAM-93:


GitHub user pabloem opened a pull request:

https://github.com/apache/beam/pull/3154

[BEAM-93] Adding support for subnetwork in Python Pipelineoptions

r: @aaltay 
also adding simple unittests for network and subnetwork options.
I have not yet submitted a job to dtaflow using these options. will do in a 
moment.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam beam-93

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3154.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3154


commit 256e25600400737c9ce4ea23953dd37247916840
Author: Pablo 
Date:   2017-05-15T22:47:18Z

Adding support for subnetwork in Python Pipelineoptions




> Support Compute Engine Subnetworks
> --
>
> Key: BEAM-93
> URL: https://issues.apache.org/jira/browse/BEAM-93
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Sam McVeety
>Assignee: Sam McVeety
>Priority: Minor
> Fix For: 0.1.0-incubating
>
>
> The Dataflow runner for Beam should support Compute Engine subnetworks for 
> the workers: https://cloud.google.com/compute/docs/subnetworks



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3154: [BEAM-93] Adding support for subnetwork in Python P...

[pabloem]

GitHub user pabloem opened a pull request:

https://github.com/apache/beam/pull/3154

[BEAM-93] Adding support for subnetwork in Python Pipelineoptions

r: @aaltay 
also adding simple unittests for network and subnetwork options.
I have not yet submitted a job to dtaflow using these options. will do in a 
moment.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam beam-93

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3154.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3154


commit 256e25600400737c9ce4ea23953dd37247916840
Author: Pablo 
Date:   2017-05-15T22:47:18Z

Adding support for subnetwork in Python Pipelineoptions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-2253) maven archetype poms should have versions controlled automatically based upon root pom.xml versions

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-2253:
---

Assignee: Luke Cwik

> maven archetype poms should have versions controlled automatically based upon 
> root pom.xml versions
> ---
>
> Key: BEAM-2253
> URL: https://issues.apache.org/jira/browse/BEAM-2253
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Affects Versions: 2.1.0
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
>
> Use a tool which sets the versions of plugins/dependencies during the build 
> process within:
> https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
> https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
> https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
> https://github.com/apache/beam/blob/master/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
> from:
> https://github.com/apache/beam/blob/master/pom.xml



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-93) Support Compute Engine Subnetworks

[Pablo Estrada (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011429#comment-16011429
 ] 

Pablo Estrada commented on BEAM-93:
---

It seems that the Pyhton SDK does not support this. I'm working on a PR for 
that.

> Support Compute Engine Subnetworks
> --
>
> Key: BEAM-93
> URL: https://issues.apache.org/jira/browse/BEAM-93
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Sam McVeety
>Assignee: Sam McVeety
>Priority: Minor
> Fix For: 0.1.0-incubating
>
>
> The Dataflow runner for Beam should support Compute Engine subnetworks for 
> the workers: https://cloud.google.com/compute/docs/subnetworks



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1663) Generated package names should be deterministic and consistent across runs.

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011398#comment-16011398
 ] 

Kenneth Knowles commented on BEAM-1663:
---

Just an update here - the funky issue blocking this has been root caused. We 
should have a solution shortly.

> Generated package names should be deterministic and consistent across runs.
> ---
>
> Key: BEAM-1663
> URL: https://issues.apache.org/jira/browse/BEAM-1663
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core, sdk-java-core
>Reporter: Alex Amato
>Assignee: Kenneth Knowles
>
> This makes it difficult for error/stack trace processing systems to identify 
> that these two errors are the same.
> These two errors contain generated package names with random strings in them, 
> would it be possible to make the generated package names consistent across 
> each job run? From what I understand the random string is to prevent two 
> package names from colliding. Perhaps we could have the name be of the 
> generated packages be based off the functions they are wrapping/referring to?
> YZjse2Tt
> 1TK4aKgJ
> src.test.java.org.apache.beam.examples.cookbook.LoggedException: 
> LoggedException
> at 
> src.test.java.org.apache.beam.examples.cookbook.CloudErrorReporting$WriteLogs$1.processElement
>  (CloudErrorReporting.java:94)
> at 
> src.test.java.org.apache.beam.examples.cookbook.CloudErrorReporting$WriteLogs$1$auxiliary$1TK4aKgJ.invokeProcessElement
>  (Unknown Source)
> ---
> src.test.java.org.apache.beam.examples.cookbook.LoggedException: 
> LoggedException
> at 
> src.test.java.org.apache.beam.examples.cookbook.CloudErrorReporting$WriteLogs$1.processElement
>  (CloudErrorReporting.java:94)
> at 
> src.test.java.org.apache.beam.examples.cookbook.CloudErrorReporting$WriteLogs$1$auxiliary$YZjse2Tt.invokeProcessElement
>  (Unknown Source)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3137

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #2832

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-2252) maven-shade-plugin should be defined in pluginManagement instead of plugins so that the plugin execution order defined by the root pom.xml is used

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-2252:
---

Assignee: Luke Cwik

> maven-shade-plugin should be defined in pluginManagement instead of plugins 
> so that the plugin execution order defined by the root pom.xml is used
> --
>
> Key: BEAM-2252
> URL: https://issues.apache.org/jira/browse/BEAM-2252
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow, runner-direct, sdk-java-core, 
> sdk-java-gcp
>Affects Versions: 2.1.0
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
>  Labels: newbie, starter
>
> Several of our child pom.xml declare maven-shade-plugin within build/plugins 
> instead of build/pluginManagement which affects the plugin execution order.
> See BEAM-2242 for when we surgically fixed how shading was performed and this 
> issue is about revisiting that change and improving upon it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3153: Fix documentation for the shard_template_name

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/3153

Fix documentation for the shard_template_name

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @chamikaramj PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam 
BEAM-fix-documentation-shard-template

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3153.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3153


commit b3f74044436f4b4a92f3135411b5d5093437ec3e
Author: Sourabh Bajaj 
Date:   2017-05-15T21:34:45Z

Fix documentation for the shard_template_name




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2252) maven-shade-plugin should be defined in pluginManagement instead of plugins so that the plugin execution order defined by the root pom.xml is used

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011368#comment-16011368
 ] 

ASF GitHub Bot commented on BEAM-2252:
--

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/3152

[BEAM-2252] Use pluginManagement instead of redefining maven-shade-plugin 
to inherit build order from root pom.xml

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam beam2252

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3152.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3152


commit ca62f3311d6a622ce3895a716da97187bbd0709a
Author: Luke Cwik 
Date:   2017-05-15T20:39:12Z

[BEAM-2252] Use pluginManagement instead of redefining maven-shade-plugin 
to inherit build order from root pom.xml




> maven-shade-plugin should be defined in pluginManagement instead of plugins 
> so that the plugin execution order defined by the root pom.xml is used
> --
>
> Key: BEAM-2252
> URL: https://issues.apache.org/jira/browse/BEAM-2252
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow, runner-direct, sdk-java-core, 
> sdk-java-gcp
>Affects Versions: 2.1.0
>Reporter: Luke Cwik
>Priority: Minor
>  Labels: newbie, starter
>
> Several of our child pom.xml declare maven-shade-plugin within build/plugins 
> instead of build/pluginManagement which affects the plugin execution order.
> See BEAM-2242 for when we surgically fixed how shading was performed and this 
> issue is about revisiting that change and improving upon it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3152: [BEAM-2252] Use pluginManagement instead of redefin...

[lukecwik]

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/3152

[BEAM-2252] Use pluginManagement instead of redefining maven-shade-plugin 
to inherit build order from root pom.xml

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam beam2252

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3152.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3152


commit ca62f3311d6a622ce3895a716da97187bbd0709a
Author: Luke Cwik 
Date:   2017-05-15T20:39:12Z

[BEAM-2252] Use pluginManagement instead of redefining maven-shade-plugin 
to inherit build order from root pom.xml




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2299) Beam repo build failes in Windows OS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011362#comment-16011362
 ] 

ASF GitHub Bot commented on BEAM-2299:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3143


> Beam repo build failes in Windows OS
> 
>
> Key: BEAM-2299
> URL: https://issues.apache.org/jira/browse/BEAM-2299
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
>Assignee: Jason Kuster
> Fix For: 2.1.0
>
>
> Need to run unit tests in Windows OS.
> Currently, many unit tests fail when doing "mvn clean install" in Windows OS.
> [ERROR] Errors:
> [ERROR]   AvroSourceTest.testCreationWithSchema:403 ? IllegalState Unable to 
> fin
> d regist...
> [ERROR]   AvroSourceTest.testGetCurrentFromUnstartedReader:305 ? IllegalState 
> Un
> able to ...
> [ERROR]   AvroSourceTest.testGetProgressFromUnstartedReader:204 ? 
> IllegalState U
> nable to...
> [ERROR]   AvroSourceTest.testMultipleFiles:390 ? IllegalState Unable to find 
> reg
> istrar f...
> [ERROR]   AvroSourceTest.testProgress:225 ? IllegalState Unable to find 
> registra
> r for c
> [ERROR]   AvroSourceTest.testProgressEmptySource:278 ? IllegalState Unable to 
> fi
> nd regis...
> [ERROR]   AvroSourceTest.testReadMetadataWithCodecs:676 ? IllegalState Unable 
> to
>  find re...
> [ERROR]   AvroSourceTest.testReadSchemaString:688 ? IllegalState Unable to 
> find
> registra...
> [ERROR]   AvroSourceTest.testReadWithDifferentCodecs:158 ? IllegalState 
> Unable t
> o find r...
> [ERROR]   AvroSourceTest.testSchemaIsInterned:460 ? IllegalState Unable to 
> find
> registra...
> [ERROR]   AvroSourceTest.testSchemaStringIsInterned:441 ? IllegalState Unable 
> to
>  find re...
> [ERROR]   AvroSourceTest.testSchemaUpdate:425 ? IllegalState Unable to find 
> regi
> strar fo...
> [ERROR]   AvroSourceTest.testSplitAtFraction:176 ? IllegalState Unable to 
> find r
> egistrar...
> [ERROR]   AvroSourceTest.testSplitAtFractionExhaustive:322 ? IllegalState 
> Unable
>  to find...
> [ERROR]   AvroSourceTest.testSplitsWithSmallBlocks:341 ? IllegalState Unable 
> to
> find reg...
> [ERROR]   CompressedSourceTest.testEmptyGzipProgress:646 ? IllegalState 
> Unable t
> o find r...
> [ERROR]   CompressedSourceTest.testGzipProgress:673 ? IllegalState Unable to 
> fin
> d regist...
> [ERROR]   CompressedSourceTest.testSplittableProgress:739 ? IllegalState 
> Unable
> to find ...
> [ERROR]   CompressedSourceTest.testUncompressedFileIsSplittable:333 ? 
> IllegalSta
> te Unabl...
> [ERROR]   CompressedSourceTest.testUnsplittable:715 ? IllegalState Unable to 
> fin
> d regist...
> [ERROR]   FileBasedSinkTest.testCopyToOutputFiles:301 ? IllegalState Unable 
> to f
> ind regi...
> [ERROR]   
> FileBasedSinkTest.testFinalize:154->generateTemporaryFilesForFinalize:
> 189 ? IO
> [ERROR]   
> FileBasedSinkTest.testFinalizeMultipleCalls:161->generateTemporaryFile
> sForFinalize:189 ? IO
> [ERROR]   
> FileBasedSinkTest.testFinalizeWithIntermediateState:171->generateTempo
> raryFilesForFinalize:189 ? IO
> [ERROR]   FileBasedSourceTest.testCloseUnstartedFilePatternReader:390 ? 
> IllegalS
> tate Una...
> [ERROR]   FileBasedSourceTest.testEstimatedSizeOfFile:746 ? IllegalState 
> Unable
> to find ...
> [ERROR]   FileBasedSourceTest.testEstimatedSizeOfFilePattern:772 ? 
> IllegalState
> Unable t...
> [ERROR]   FileBasedSourceTest.testFractionConsumedWhenReadingFilepattern:422 
> ? I
> llegalState
> [ERROR]   FileBasedSourceTest.testFullyReadFilePattern:370 ? IllegalState 
> Unable
>  to find...
> [ERROR]   FileBasedSourceTest.testFullyReadFilePatternFirstRecordEmpty:461 ? 
> Ill
> egalState
> [ERROR]   FileBasedSourceTest.testFullyReadSingleFile:346 ? IllegalState 
> Unable
> to find ...
> [ERROR]   FileBasedSourceTest.testReadAllSplitsOfFilePattern:792 ? 
> IllegalState
> Unable t...
> [ERROR]   FileBasedSourceTest.testReadAllSplitsOfSingleFile:681 ? 
> IllegalState U
> nable to...
> [ERROR]   FileBasedSourceTest.testReadEverythingFromFileWithSplits:502 ? 
> Illegal
> State Un...
> [ERROR]   FileBasedSourceTest.testReadFileWithSplitsWithEmptyRange:578 ? 
> Illegal
> State Un...
> [ERROR]   FileBasedSourceTest.testReadRangeAtEnd:659 ? IllegalState Unable to 
> fi
> nd regis...
> [ERROR]   FileBasedSourceTest.testReadRangeAtMiddle:637 ? IllegalState Unable 
> to
>  find re...
> [ERROR]   FileBasedSourceTest.testReadRangeAtStart:472 ? IllegalState Unable 
> to
> find reg...
> [ERROR]   FileBasedSourceTest.testReadRangeFromFileWithSplitsFromMiddle:546 ? 
> Il
> legalState
> [ERROR]   
> FileBasedSourceTest.testReadRangeFromFileWithSplitsFromMiddleOfHeader:
> 615 ? IllegalState
> [ERROR]   FileBasedSourceTest.testReadRangeFromFileWithSplitsFromStart:517 ? 
> Ill
> egalState
> [ERROR]   

[1/2] beam git commit: [BEAM-2299] Run maven install on Windows machine for build/test coverage on Windows

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master cd91eb298 -> 13e1be2f0


[BEAM-2299] Run maven install on Windows machine for build/test coverage on 
Windows


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/044d1832
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/044d1832
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/044d1832

Branch: refs/heads/master
Commit: 044d1832f44394f2dc61172d4a2737157a4e2210
Parents: cd91eb2
Author: Luke Cwik 
Authored: Mon May 15 09:40:49 2017 -0700
Committer: Luke Cwik 
Committed: Mon May 15 14:29:56 2017 -0700

--
 .../jenkins/common_job_properties.groovy| 29 +
 ..._PostCommit_Java_MavenInstall_Windows.groovy | 45 
 sdks/common/runner-api/pom.xml  |  3 --
 3 files changed, 65 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/044d1832/.test-infra/jenkins/common_job_properties.groovy
--
diff --git a/.test-infra/jenkins/common_job_properties.groovy 
b/.test-infra/jenkins/common_job_properties.groovy
index 56eb0de..f47ab28 100644
--- a/.test-infra/jenkins/common_job_properties.groovy
+++ b/.test-infra/jenkins/common_job_properties.groovy
@@ -24,14 +24,25 @@ class common_job_properties {
 
   // Sets common top-level job properties for website repository jobs.
   static void setTopLevelWebsiteJobProperties(context) {
-setTopLevelJobProperties(context, 'beam-site', 'asf-site', 30)
+setTopLevelJobProperties(
+context,
+'beam-site',
+'asf-site',
+'beam',
+30)
   }
 
   // Sets common top-level job properties for main repository jobs.
   static void setTopLevelMainJobProperties(context,
-   String defaultBranch = 'master',
-   int defaultTimeout = 100) {
-setTopLevelJobProperties(context, 'beam', defaultBranch, defaultTimeout)
+   String branch = 'master',
+   int timeout = 100,
+   String jenkinsExecutorLabel = 
'beam') {
+setTopLevelJobProperties(
+context,
+'beam',
+branch,
+jenkinsExecutorLabel,
+timeout)
   }
 
   // Sets common top-level job properties. Accessed through one of the above
@@ -39,6 +50,7 @@ class common_job_properties {
   private static void setTopLevelJobProperties(context,
String repositoryName,
String defaultBranch,
+   String jenkinsExecutorLabel,
int defaultTimeout) {
 
 // GitHub project.
@@ -49,9 +61,8 @@ class common_job_properties {
 // Set JDK version.
 context.jdk('JDK 1.8 (latest)')
 
-// Restrict this project to run only on Jenkins executors dedicated to the
-// Apache Beam project.
-context.label('beam')
+// Restrict this project to run only on Jenkins executors as specified
+context.label(jenkinsExecutorLabel)
 
 // Discard old builds. Build records are only kept up to this number of 
days.
 context.logRotator {
@@ -163,8 +174,8 @@ class common_job_properties {
   }
 
   // Sets common config for Maven jobs.
-  static void setMavenConfig(context) {
-context.mavenInstallation('Maven 3.3.3')
+  static void setMavenConfig(context, mavenInstallation='Maven 3.3.3') {
+context.mavenInstallation(mavenInstallation)
 context.mavenOpts('-Dorg.slf4j.simpleLogger.showDateTime=true')
 
context.mavenOpts('-Dorg.slf4j.simpleLogger.dateTimeFormat=-MM-dd\\\'T\\\'HH:mm:ss.SSS')
 // The -XX:+TieredCompilation -XX:TieredStopAtLevel=1 JVM options enable

http://git-wip-us.apache.org/repos/asf/beam/blob/044d1832/.test-infra/jenkins/job_beam_PostCommit_Java_MavenInstall_Windows.groovy
--
diff --git 
a/.test-infra/jenkins/job_beam_PostCommit_Java_MavenInstall_Windows.groovy 
b/.test-infra/jenkins/job_beam_PostCommit_Java_MavenInstall_Windows.groovy
new file mode 100644
index 000..f781b4e
--- /dev/null
+++ b/.test-infra/jenkins/job_beam_PostCommit_Java_MavenInstall_Windows.groovy
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * 

[GitHub] beam pull request #3143: [BEAM-2299] Run maven install on Windows machine fo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3143


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #3817

[Apache Jenkins Server]

See 

[2/2] beam git commit: [BEAM-2299] Run maven install on Windows machine for build/test coverage on Windows.

[lcwik]

[BEAM-2299] Run maven install on Windows machine for build/test coverage on 
Windows.

This closes #3143


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/13e1be2f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/13e1be2f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/13e1be2f

Branch: refs/heads/master
Commit: 13e1be2f0a280fc704616b9b89551281a561c374
Parents: cd91eb2 044d183
Author: Luke Cwik 
Authored: Mon May 15 14:30:44 2017 -0700
Committer: Luke Cwik 
Committed: Mon May 15 14:30:44 2017 -0700

--
 .../jenkins/common_job_properties.groovy| 29 +
 ..._PostCommit_Java_MavenInstall_Windows.groovy | 45 
 sdks/common/runner-api/pom.xml  |  3 --
 3 files changed, 65 insertions(+), 12 deletions(-)
--

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3816

[Apache Jenkins Server]

See 

[jira] [Closed] (BEAM-2142) add missing composite transforms Python snippets to programming guide

[Melissa Pashniak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Melissa Pashniak closed BEAM-2142.
--

> add missing composite transforms Python snippets to programming guide
> -
>
> Key: BEAM-2142
> URL: https://issues.apache.org/jira/browse/BEAM-2142
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>Assignee: Melissa Pashniak
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #14

[Apache Jenkins Server]

See 


Changes:

[altay] Remove "Dataflow" from apache_beam __init__.py file

--
[...truncated 1.99 MB...]
2017-05-15T21:15:39.368 [INFO] Excluding org.slf4j:log4j-over-slf4j:jar:1.7.7 
from the shaded jar.
2017-05-15T21:15:39.368 [INFO] Excluding org.slf4j:jcl-over-slf4j:jar:1.7.7 
from the shaded jar.
2017-05-15T21:15:39.368 [INFO] Excluding 
io.dropwizard.metrics:metrics-core:jar:3.1.0 from the shaded jar.
2017-05-15T21:15:39.368 [INFO] Excluding 
io.dropwizard.metrics:metrics-jvm:jar:3.1.0 from the shaded jar.
2017-05-15T21:15:39.368 [INFO] Excluding 
com.clearspring.analytics:stream:jar:2.5.2 from the shaded jar.
2017-05-15T21:15:39.368 [INFO] Excluding it.unimi.dsi:fastutil:jar:6.5.7 from 
the shaded jar.
2017-05-15T21:15:39.368 [INFO] Excluding org.apache.thrift:libthrift:jar:0.9.2 
from the shaded jar.
2017-05-15T21:15:39.368 [INFO] Excluding io.netty:netty-all:jar:4.0.39.Final 
from the shaded jar.
2017-05-15T21:15:39.368 [INFO] Excluding org.ow2.asm:asm:jar:5.0.3 from the 
shaded jar.
2017-05-15T21:15:39.368 [INFO] Excluding commons-io:commons-io:jar:2.4 from the 
shaded jar.
2017-05-15T21:15:39.368 [INFO] Excluding 
org.apache.beam:beam-sdks-java-core:jar:2.1.0-SNAPSHOT from the shaded jar.
2017-05-15T21:15:39.368 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-05-15T21:15:39.368 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-05-15T21:15:39.368 [INFO] Excluding net.bytebuddy:byte-buddy:jar:1.6.8 
from the shaded jar.
2017-05-15T21:15:39.368 [INFO] Excluding 
org.apache.logging.log4j:log4j-api:jar:2.6.2 from the shaded jar.
2017-05-15T21:15:41.931 [INFO] Replacing original artifact with shaded artifact.
2017-05-15T21:15:41.931 [INFO] Replacing 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT.jar
 with 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-shaded.jar
2017-05-15T21:15:41.932 [INFO] Replacing original test artifact with shaded 
test artifact.
2017-05-15T21:15:41.932 [INFO] Replacing 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-tests.jar
 with 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-shaded-tests.jar
2017-05-15T21:15:42.109 [INFO] 
2017-05-15T21:15:42.109 [INFO] --- maven-javadoc-plugin:2.10.4:jar 
(attach-javadocs) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-15T21:15:42.292 [INFO] 
2017-05-15T21:15:42.293 [INFO] --- maven-source-plugin:3.0.1:jar-no-fork 
(attach-sources) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-15T21:15:42.299 [INFO] Building jar: 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-sources.jar
2017-05-15T21:15:42.488 [INFO] 
2017-05-15T21:15:42.488 [INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork 
(attach-test-sources) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-15T21:15:42.494 [INFO] Building jar: 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-test-sources.jar
2017-05-15T21:15:42.694 [INFO] 
2017-05-15T21:15:42.694 [INFO] --- maven-dependency-plugin:3.0.0:analyze-only 
(default) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-15T21:15:43.188 [INFO] No dependency problems found
2017-05-15T21:15:43.377 [INFO] 
2017-05-15T21:15:43.377 [INFO] --- apache-rat-plugin:0.12:check (default) @ 
beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-15T21:15:43.394 [INFO] Enabled default license matchers.
2017-05-15T21:15:43.394 [INFO] Will parse SCM ignores for exclusions...
2017-05-15T21:15:43.394 [INFO] Finished adding exclusions from SCM ignore files.
2017-05-15T21:15:43.394 [INFO] 61 implicit excludes (use -debug for more 
details).
2017-05-15T21:15:43.394 [INFO] Exclude: **/target/**/*
2017-05-15T21:15:43.394 [INFO] Exclude: **/dependency-reduced-pom.xml
2017-05-15T21:15:43.394 [INFO] Exclude: **/hs_err_pid*.log
2017-05-15T21:15:43.394 [INFO] Exclude: .github/**/*
2017-05-15T21:15:43.394 [INFO] Exclude: **/*.iml
2017-05-15T21:15:43.394 [INFO] Exclude: **/.idea/**/*
2017-05-15T21:15:43.394 [INFO] Exclude: **/*.egg-info/**/*
2017-05-15T21:15:43.394 [INFO] Exclude: **/package-list
2017-05-15T21:15:43.395 [INFO] Exclude: **/user.avsc
2017-05-15T21:15:43.395 [INFO] Exclude: 

[jira] [Resolved] (BEAM-2142) add missing composite transforms Python snippets to programming guide

[Melissa Pashniak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Melissa Pashniak resolved BEAM-2142.

   Resolution: Fixed
Fix Version/s: Not applicable

> add missing composite transforms Python snippets to programming guide
> -
>
> Key: BEAM-2142
> URL: https://issues.apache.org/jira/browse/BEAM-2142
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>Assignee: Melissa Pashniak
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2831

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #13

[Apache Jenkins Server]

See 


Changes:

[altay] Moving the data file for trigger tests to testing/data

--
[...truncated 1.99 MB...]
2017-05-15T20:58:31.876 [INFO] Excluding org.slf4j:jcl-over-slf4j:jar:1.7.7 
from the shaded jar.
2017-05-15T20:58:31.876 [INFO] Excluding 
io.dropwizard.metrics:metrics-core:jar:3.1.0 from the shaded jar.
2017-05-15T20:58:31.876 [INFO] Excluding 
io.dropwizard.metrics:metrics-jvm:jar:3.1.0 from the shaded jar.
2017-05-15T20:58:31.876 [INFO] Excluding 
com.clearspring.analytics:stream:jar:2.5.2 from the shaded jar.
2017-05-15T20:58:31.876 [INFO] Excluding it.unimi.dsi:fastutil:jar:6.5.7 from 
the shaded jar.
2017-05-15T20:58:31.876 [INFO] Excluding org.apache.thrift:libthrift:jar:0.9.2 
from the shaded jar.
2017-05-15T20:58:31.876 [INFO] Excluding io.netty:netty-all:jar:4.0.39.Final 
from the shaded jar.
2017-05-15T20:58:31.876 [INFO] Excluding org.ow2.asm:asm:jar:5.0.3 from the 
shaded jar.
2017-05-15T20:58:31.876 [INFO] Excluding commons-io:commons-io:jar:2.4 from the 
shaded jar.
2017-05-15T20:58:31.876 [INFO] Excluding 
org.apache.beam:beam-sdks-java-core:jar:2.1.0-SNAPSHOT from the shaded jar.
2017-05-15T20:58:31.876 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-05-15T20:58:31.876 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-05-15T20:58:31.876 [INFO] Excluding net.bytebuddy:byte-buddy:jar:1.6.8 
from the shaded jar.
2017-05-15T20:58:31.876 [INFO] Excluding 
org.apache.logging.log4j:log4j-api:jar:2.6.2 from the shaded jar.
2017-05-15T20:58:34.363 [INFO] Replacing original artifact with shaded artifact.
2017-05-15T20:58:34.363 [INFO] Replacing 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT.jar
 with 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-shaded.jar
2017-05-15T20:58:34.363 [INFO] Replacing original test artifact with shaded 
test artifact.
2017-05-15T20:58:34.363 [INFO] Replacing 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-tests.jar
 with 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-shaded-tests.jar
2017-05-15T20:58:34.598 [INFO] 
2017-05-15T20:58:34.598 [INFO] --- maven-javadoc-plugin:2.10.4:jar 
(attach-javadocs) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-15T20:58:36.859 [INFO] 
2017-05-15T20:58:36.859 [INFO] --- maven-source-plugin:3.0.1:jar-no-fork 
(attach-sources) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-15T20:58:36.866 [INFO] Building jar: 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-sources.jar
2017-05-15T20:58:38.888 [INFO] 
2017-05-15T20:58:38.888 [INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork 
(attach-test-sources) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-15T20:58:38.895 [INFO] Building jar: 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-test-sources.jar
2017-05-15T20:58:39.636 [INFO] 
2017-05-15T20:58:39.636 [INFO] --- maven-dependency-plugin:3.0.0:analyze-only 
(default) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-15T20:58:40.195 [INFO] No dependency problems found
2017-05-15T20:58:40.674 [INFO] 
2017-05-15T20:58:40.674 [INFO] --- apache-rat-plugin:0.12:check (default) @ 
beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-15T20:58:40.693 [INFO] Enabled default license matchers.
2017-05-15T20:58:40.695 [INFO] Will parse SCM ignores for exclusions...
2017-05-15T20:58:40.695 [INFO] Finished adding exclusions from SCM ignore files.
2017-05-15T20:58:40.695 [INFO] 61 implicit excludes (use -debug for more 
details).
2017-05-15T20:58:40.695 [INFO] Exclude: **/target/**/*
2017-05-15T20:58:40.695 [INFO] Exclude: **/dependency-reduced-pom.xml
2017-05-15T20:58:40.695 [INFO] Exclude: **/hs_err_pid*.log
2017-05-15T20:58:40.695 [INFO] Exclude: .github/**/*
2017-05-15T20:58:40.695 [INFO] Exclude: **/*.iml
2017-05-15T20:58:40.695 [INFO] Exclude: **/.idea/**/*
2017-05-15T20:58:40.695 [INFO] Exclude: **/*.egg-info/**/*
2017-05-15T20:58:40.695 [INFO] Exclude: **/package-list
2017-05-15T20:58:40.695 [INFO] Exclude: **/user.avsc
2017-05-15T20:58:40.695 [INFO] Exclude: **/test/resources/**/*.txt
2017-05-15T20:58:40.695 [INFO] Exclude: **/test/**/.placeholder

[GitHub] beam pull request #3148: Remove "Dataflow" from apache_beam __init__.py file

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3148


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3148

[altay]

This closes #3148


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/cd91eb29
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/cd91eb29
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/cd91eb29

Branch: refs/heads/master
Commit: cd91eb298d48ba4ba51241eb7ba1c3d7c6efdbfa
Parents: 9b5a8e6 96906ae
Author: Ahmet Altay 
Authored: Mon May 15 13:51:39 2017 -0700
Committer: Ahmet Altay 
Committed: Mon May 15 13:51:39 2017 -0700

--
 sdks/python/apache_beam/__init__.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[1/2] beam git commit: Remove "Dataflow" from apache_beam __init__.py file

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 9b5a8e62c -> cd91eb298


Remove "Dataflow" from apache_beam __init__.py file


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/96906aee
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/96906aee
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/96906aee

Branch: refs/heads/master
Commit: 96906aee9d0cdbb9fae67dd5bd2af0710a1b67bf
Parents: 9b5a8e6
Author: Charles Chen 
Authored: Mon May 15 11:26:59 2017 -0700
Committer: Ahmet Altay 
Committed: Mon May 15 13:51:33 2017 -0700

--
 sdks/python/apache_beam/__init__.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/96906aee/sdks/python/apache_beam/__init__.py
--
diff --git a/sdks/python/apache_beam/__init__.py 
b/sdks/python/apache_beam/__init__.py
index 9921498..8b772c9 100644
--- a/sdks/python/apache_beam/__init__.py
+++ b/sdks/python/apache_beam/__init__.py
@@ -68,7 +68,7 @@ import sys
 
 if not (sys.version_info[0] == 2 and sys.version_info[1] == 7):
   raise RuntimeError(
-  'Dataflow SDK for Python is supported only on Python 2.7. '
+  'The Apache Beam SDK for Python is supported only on Python 2.7. '
   'It is not supported on Python ['+ str(sys.version_info) + '].')
 
 # pylint: disable=wrong-import-position

[2/2] beam git commit: This closes #3144

[altay]

This closes #3144


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9b5a8e62
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9b5a8e62
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9b5a8e62

Branch: refs/heads/master
Commit: 9b5a8e62ca695b0d0def27844084237102f0055a
Parents: 926157d 911128b
Author: Ahmet Altay 
Authored: Mon May 15 13:34:09 2017 -0700
Committer: Ahmet Altay 
Committed: Mon May 15 13:34:09 2017 -0700

--
 .../testing/data/trigger_transcripts.yaml   | 224 +++
 .../apache_beam/transforms/trigger_test.py  |   5 +-
 .../transforms/trigger_transcripts.yaml | 224 ---
 3 files changed, 227 insertions(+), 226 deletions(-)
--

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #2829

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #3815

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #12

[Apache Jenkins Server]

See 


Changes:

[lcwik] Fix GcsResourceIdTest in postcommits

--
[...truncated 355.53 KB...]
2017-05-15T20:20:02.116 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/3.2.0/protobuf-java-3.2.0.pom
2017-05-15T20:20:02.147 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/3.2.0/protobuf-java-3.2.0.pom
 (5 KB at 147.1 KB/sec)
2017-05-15T20:20:02.147 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-parent/3.2.0/protobuf-parent-3.2.0.pom
2017-05-15T20:20:02.178 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-parent/3.2.0/protobuf-parent-3.2.0.pom
 (7 KB at 215.6 KB/sec)
2017-05-15T20:20:02.178 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-core/1.2.0/grpc-core-1.2.0.pom
2017-05-15T20:20:02.209 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-core/1.2.0/grpc-core-1.2.0.pom
 (3 KB at 83.9 KB/sec)
2017-05-15T20:20:02.209 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.pom
2017-05-15T20:20:02.241 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.pom
 (2 KB at 49.9 KB/sec)
2017-05-15T20:20:02.241 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_parent/2.0.15/error_prone_parent-2.0.15.pom
2017-05-15T20:20:02.272 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_parent/2.0.15/error_prone_parent-2.0.15.pom
 (6 KB at 161.5 KB/sec)
2017-05-15T20:20:02.272 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.pom
2017-05-15T20:20:02.303 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.pom
 (5 KB at 135.0 KB/sec)
2017-05-15T20:20:02.303 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.pom
2017-05-15T20:20:02.334 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.pom
 (2 KB at 55.1 KB/sec)
2017-05-15T20:20:02.334 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/instrumentation/instrumentation-api/0.3.0/instrumentation-api-0.3.0.pom
2017-05-15T20:20:02.366 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/instrumentation/instrumentation-api/0.3.0/instrumentation-api-0.3.0.pom
 (2 KB at 45.7 KB/sec)
2017-05-15T20:20:02.366 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf/1.2.0/grpc-protobuf-1.2.0.pom
2017-05-15T20:20:02.397 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf/1.2.0/grpc-protobuf-1.2.0.pom
 (3 KB at 83.6 KB/sec)
2017-05-15T20:20:02.397 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java-util/3.2.0/protobuf-java-util-3.2.0.pom
2017-05-15T20:20:02.428 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java-util/3.2.0/protobuf-java-util-3.2.0.pom
 (5 KB at 133.9 KB/sec)
2017-05-15T20:20:02.428 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.7/gson-2.7.pom
2017-05-15T20:20:02.459 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.7/gson-2.7.pom 
(2 KB at 45.5 KB/sec)
2017-05-15T20:20:02.459 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson-parent/2.7/gson-parent-2.7.pom
2017-05-15T20:20:02.491 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson-parent/2.7/gson-parent-2.7.pom
 (4 KB at 109.3 KB/sec)
2017-05-15T20:20:02.491 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf-lite/1.2.0/grpc-protobuf-lite-1.2.0.pom
2017-05-15T20:20:02.522 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf-lite/1.2.0/grpc-protobuf-lite-1.2.0.pom
 (3 KB at 66.9 KB/sec)
2017-05-15T20:20:02.522 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-stub/1.2.0/grpc-stub-1.2.0.pom
2017-05-15T20:20:02.553 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-stub/1.2.0/grpc-stub-1.2.0.pom
 (3 KB at 65.5 KB/sec)
2017-05-15T20:20:02.553 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/3.2.0/protobuf-java-3.2.0.jar
2017-05-15T20:20:02.553 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-core/1.2.0/grpc-core-1.2.0.jar
2017-05-15T20:20:02.553 [INFO] Downloading: 

[GitHub] beam pull request #3147: Fix GcsResourceIdTest in postcommits

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3147


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (BEAM-1013) Recheck all existing programming guide code snippets for correctness

[Melissa Pashniak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Melissa Pashniak closed BEAM-1013.
--

> Recheck all existing programming guide code snippets for correctness
> 
>
> Key: BEAM-1013
> URL: https://issues.apache.org/jira/browse/BEAM-1013
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>Assignee: Melissa Pashniak
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: Fix GcsResourceIdTest in postcommits

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master 88ec00860 -> 926157d42


Fix GcsResourceIdTest in postcommits

This test is not a @ValidatesRunner test, so it should not use 
TestPipelineOptions.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/15ec2cd8
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/15ec2cd8
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/15ec2cd8

Branch: refs/heads/master
Commit: 15ec2cd83d4291957a75d9f2ac838ff867dcbe0c
Parents: 88ec008
Author: Dan Halperin 
Authored: Mon May 15 10:57:24 2017 -0700
Committer: Luke Cwik 
Committed: Mon May 15 13:11:48 2017 -0700

--
 .../beam/sdk/extensions/gcp/storage/GcsResourceIdTest.java   | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/15ec2cd8/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/storage/GcsResourceIdTest.java
--
diff --git 
a/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/storage/GcsResourceIdTest.java
 
b/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/storage/GcsResourceIdTest.java
index c1e214e..ae7d40c 100644
--- 
a/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/storage/GcsResourceIdTest.java
+++ 
b/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/storage/GcsResourceIdTest.java
@@ -26,7 +26,7 @@ import org.apache.beam.sdk.io.FileSystems;
 import org.apache.beam.sdk.io.fs.ResolveOptions.StandardResolveOptions;
 import org.apache.beam.sdk.io.fs.ResourceId;
 import org.apache.beam.sdk.io.fs.ResourceIdTester;
-import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.util.gcsfs.GcsPath;
 import org.junit.Rule;
 import org.junit.Test;
@@ -168,7 +168,7 @@ public class GcsResourceIdTest {
 
   @Test
   public void testResourceIdTester() throws Exception {
-
FileSystems.setDefaultPipelineOptions(TestPipeline.testingPipelineOptions());
+FileSystems.setDefaultPipelineOptions(PipelineOptionsFactory.create());
 
ResourceIdTester.runResourceIdBattery(toResourceIdentifier("gs://bucket/foo/"));
   }
 

[2/2] beam git commit: Fix GcsResourceIdTest in postcommits

[lcwik]

Fix GcsResourceIdTest in postcommits

This closes #3147


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/926157d4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/926157d4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/926157d4

Branch: refs/heads/master
Commit: 926157d42db572dacb9142665d5c922dcad589f0
Parents: 88ec008 15ec2cd
Author: Luke Cwik 
Authored: Mon May 15 13:16:05 2017 -0700
Committer: Luke Cwik 
Committed: Mon May 15 13:16:05 2017 -0700

--
 .../beam/sdk/extensions/gcp/storage/GcsResourceIdTest.java   | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[jira] [Commented] (BEAM-1013) Recheck all existing programming guide code snippets for correctness

[Melissa Pashniak (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011249#comment-16011249
 ] 

Melissa Pashniak commented on BEAM-1013:


I think this is complete for now, and if we find issues in the future we can 
open JIRAs on the specifics.


> Recheck all existing programming guide code snippets for correctness
> 
>
> Key: BEAM-1013
> URL: https://issues.apache.org/jira/browse/BEAM-1013
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>Assignee: Melissa Pashniak
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #11

[Apache Jenkins Server]

See 


Changes:

[github] readAvros should't have proto Message upper bound

[lcwik] [BEAM-2299] Run maven install on Windows machine for build/test coverage

--
[...truncated 1.99 MB...]
2017-05-15T20:10:38.208 [INFO] Excluding org.slf4j:jcl-over-slf4j:jar:1.7.7 
from the shaded jar.
2017-05-15T20:10:38.208 [INFO] Excluding 
io.dropwizard.metrics:metrics-core:jar:3.1.0 from the shaded jar.
2017-05-15T20:10:38.208 [INFO] Excluding 
io.dropwizard.metrics:metrics-jvm:jar:3.1.0 from the shaded jar.
2017-05-15T20:10:38.208 [INFO] Excluding 
com.clearspring.analytics:stream:jar:2.5.2 from the shaded jar.
2017-05-15T20:10:38.209 [INFO] Excluding it.unimi.dsi:fastutil:jar:6.5.7 from 
the shaded jar.
2017-05-15T20:10:38.209 [INFO] Excluding org.apache.thrift:libthrift:jar:0.9.2 
from the shaded jar.
2017-05-15T20:10:38.209 [INFO] Excluding io.netty:netty-all:jar:4.0.39.Final 
from the shaded jar.
2017-05-15T20:10:38.209 [INFO] Excluding org.ow2.asm:asm:jar:5.0.3 from the 
shaded jar.
2017-05-15T20:10:38.209 [INFO] Excluding commons-io:commons-io:jar:2.4 from the 
shaded jar.
2017-05-15T20:10:38.209 [INFO] Excluding 
org.apache.beam:beam-sdks-java-core:jar:2.1.0-SNAPSHOT from the shaded jar.
2017-05-15T20:10:38.209 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-05-15T20:10:38.209 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-05-15T20:10:38.209 [INFO] Excluding net.bytebuddy:byte-buddy:jar:1.6.8 
from the shaded jar.
2017-05-15T20:10:38.209 [INFO] Excluding 
org.apache.logging.log4j:log4j-api:jar:2.6.2 from the shaded jar.
2017-05-15T20:10:40.638 [INFO] Replacing original artifact with shaded artifact.
2017-05-15T20:10:40.638 [INFO] Replacing 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT.jar
 with 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-shaded.jar
2017-05-15T20:10:40.639 [INFO] Replacing original test artifact with shaded 
test artifact.
2017-05-15T20:10:40.639 [INFO] Replacing 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-tests.jar
 with 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-shaded-tests.jar
2017-05-15T20:10:40.816 [INFO] 
2017-05-15T20:10:40.816 [INFO] --- maven-javadoc-plugin:2.10.4:jar 
(attach-javadocs) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-15T20:10:41.001 [INFO] 
2017-05-15T20:10:41.001 [INFO] --- maven-source-plugin:3.0.1:jar-no-fork 
(attach-sources) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-15T20:10:41.009 [INFO] Building jar: 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-sources.jar
2017-05-15T20:10:41.199 [INFO] 
2017-05-15T20:10:41.199 [INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork 
(attach-test-sources) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-15T20:10:41.206 [INFO] Building jar: 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-test-sources.jar
2017-05-15T20:10:41.405 [INFO] 
2017-05-15T20:10:41.405 [INFO] --- maven-dependency-plugin:3.0.0:analyze-only 
(default) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-15T20:10:41.876 [INFO] No dependency problems found
2017-05-15T20:10:42.052 [INFO] 
2017-05-15T20:10:42.052 [INFO] --- apache-rat-plugin:0.12:check (default) @ 
beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-15T20:10:42.069 [INFO] Enabled default license matchers.
2017-05-15T20:10:42.070 [INFO] Will parse SCM ignores for exclusions...
2017-05-15T20:10:42.070 [INFO] Finished adding exclusions from SCM ignore files.
2017-05-15T20:10:42.070 [INFO] 61 implicit excludes (use -debug for more 
details).
2017-05-15T20:10:42.070 [INFO] Exclude: **/target/**/*
2017-05-15T20:10:42.070 [INFO] Exclude: **/dependency-reduced-pom.xml
2017-05-15T20:10:42.070 [INFO] Exclude: **/hs_err_pid*.log
2017-05-15T20:10:42.070 [INFO] Exclude: .github/**/*
2017-05-15T20:10:42.070 [INFO] Exclude: **/*.iml
2017-05-15T20:10:42.070 [INFO] Exclude: **/.idea/**/*
2017-05-15T20:10:42.070 [INFO] Exclude: **/*.egg-info/**/*
2017-05-15T20:10:42.070 [INFO] Exclude: **/package-list
2017-05-15T20:10:42.070 [INFO] Exclude: **/user.avsc
2017-05-15T20:10:42.070 [INFO] Exclude: 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3136

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #10

[Apache Jenkins Server]

See 


Changes:

[github] readAvros should't have proto Message upper bound

--
[...truncated 1.99 MB...]
2017-05-15T19:55:29.728 [INFO] Excluding org.slf4j:jcl-over-slf4j:jar:1.7.7 
from the shaded jar.
2017-05-15T19:55:29.728 [INFO] Excluding 
io.dropwizard.metrics:metrics-core:jar:3.1.0 from the shaded jar.
2017-05-15T19:55:29.728 [INFO] Excluding 
io.dropwizard.metrics:metrics-jvm:jar:3.1.0 from the shaded jar.
2017-05-15T19:55:29.728 [INFO] Excluding 
com.clearspring.analytics:stream:jar:2.5.2 from the shaded jar.
2017-05-15T19:55:29.728 [INFO] Excluding it.unimi.dsi:fastutil:jar:6.5.7 from 
the shaded jar.
2017-05-15T19:55:29.728 [INFO] Excluding org.apache.thrift:libthrift:jar:0.9.2 
from the shaded jar.
2017-05-15T19:55:29.728 [INFO] Excluding io.netty:netty-all:jar:4.0.39.Final 
from the shaded jar.
2017-05-15T19:55:29.728 [INFO] Excluding org.ow2.asm:asm:jar:5.0.3 from the 
shaded jar.
2017-05-15T19:55:29.728 [INFO] Excluding commons-io:commons-io:jar:2.4 from the 
shaded jar.
2017-05-15T19:55:29.728 [INFO] Excluding 
org.apache.beam:beam-sdks-java-core:jar:2.1.0-SNAPSHOT from the shaded jar.
2017-05-15T19:55:29.728 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-05-15T19:55:29.728 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-05-15T19:55:29.728 [INFO] Excluding net.bytebuddy:byte-buddy:jar:1.6.8 
from the shaded jar.
2017-05-15T19:55:29.728 [INFO] Excluding 
org.apache.logging.log4j:log4j-api:jar:2.6.2 from the shaded jar.
2017-05-15T19:55:32.339 [INFO] Replacing original artifact with shaded artifact.
2017-05-15T19:55:32.339 [INFO] Replacing 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT.jar
 with 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-shaded.jar
2017-05-15T19:55:32.339 [INFO] Replacing original test artifact with shaded 
test artifact.
2017-05-15T19:55:32.339 [INFO] Replacing 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-tests.jar
 with 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-shaded-tests.jar
2017-05-15T19:55:32.511 [INFO] 
2017-05-15T19:55:32.511 [INFO] --- maven-javadoc-plugin:2.10.4:jar 
(attach-javadocs) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-15T19:55:32.688 [INFO] 
2017-05-15T19:55:32.688 [INFO] --- maven-source-plugin:3.0.1:jar-no-fork 
(attach-sources) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-15T19:55:32.694 [INFO] Building jar: 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-sources.jar
2017-05-15T19:55:32.874 [INFO] 
2017-05-15T19:55:32.874 [INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork 
(attach-test-sources) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-15T19:55:32.881 [INFO] Building jar: 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-test-sources.jar
2017-05-15T19:55:33.069 [INFO] 
2017-05-15T19:55:33.069 [INFO] --- maven-dependency-plugin:3.0.0:analyze-only 
(default) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-15T19:55:33.517 [INFO] No dependency problems found
2017-05-15T19:55:33.688 [INFO] 
2017-05-15T19:55:33.688 [INFO] --- apache-rat-plugin:0.12:check (default) @ 
beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-15T19:55:33.705 [INFO] Enabled default license matchers.
2017-05-15T19:55:33.707 [INFO] Will parse SCM ignores for exclusions...
2017-05-15T19:55:33.707 [INFO] Finished adding exclusions from SCM ignore files.
2017-05-15T19:55:33.707 [INFO] 61 implicit excludes (use -debug for more 
details).
2017-05-15T19:55:33.707 [INFO] Exclude: **/target/**/*
2017-05-15T19:55:33.707 [INFO] Exclude: **/dependency-reduced-pom.xml
2017-05-15T19:55:33.707 [INFO] Exclude: **/hs_err_pid*.log
2017-05-15T19:55:33.707 [INFO] Exclude: .github/**/*
2017-05-15T19:55:33.707 [INFO] Exclude: **/*.iml
2017-05-15T19:55:33.707 [INFO] Exclude: **/.idea/**/*
2017-05-15T19:55:33.707 [INFO] Exclude: **/*.egg-info/**/*
2017-05-15T19:55:33.707 [INFO] Exclude: **/package-list
2017-05-15T19:55:33.707 [INFO] Exclude: **/user.avsc
2017-05-15T19:55:33.707 [INFO] Exclude: **/test/resources/**/*.txt
2017-05-15T19:55:33.707 [INFO] Exclude: **/test/**/.placeholder

[jira] [Commented] (BEAM-1663) Generated package names should be deterministic and consistent across runs.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011221#comment-16011221
 ] 

ASF GitHub Bot commented on BEAM-1663:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3151

[BEAM-1663] Use stable naming strategy for ByteBuddy invokers

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

This helps to coalesce related failures across multiple JVM instances, over 
time, across machines, etc.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam bytebuddy-names

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3151.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3151


commit 515c0c846c63e5a05d792ce3b8c60aaa4acdd382
Author: Kenneth Knowles 
Date:   2017-03-09T01:02:05Z

Use stable naming strategy for ByteBuddy invokers

This helps to coalesce related failures across multiple
JVM instances, over time, across machines, etc.




> Generated package names should be deterministic and consistent across runs.
> ---
>
> Key: BEAM-1663
> URL: https://issues.apache.org/jira/browse/BEAM-1663
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core, sdk-java-core
>Reporter: Alex Amato
>Assignee: Kenneth Knowles
>
> This makes it difficult for error/stack trace processing systems to identify 
> that these two errors are the same.
> These two errors contain generated package names with random strings in them, 
> would it be possible to make the generated package names consistent across 
> each job run? From what I understand the random string is to prevent two 
> package names from colliding. Perhaps we could have the name be of the 
> generated packages be based off the functions they are wrapping/referring to?
> YZjse2Tt
> 1TK4aKgJ
> src.test.java.org.apache.beam.examples.cookbook.LoggedException: 
> LoggedException
> at 
> src.test.java.org.apache.beam.examples.cookbook.CloudErrorReporting$WriteLogs$1.processElement
>  (CloudErrorReporting.java:94)
> at 
> src.test.java.org.apache.beam.examples.cookbook.CloudErrorReporting$WriteLogs$1$auxiliary$1TK4aKgJ.invokeProcessElement
>  (Unknown Source)
> ---
> src.test.java.org.apache.beam.examples.cookbook.LoggedException: 
> LoggedException
> at 
> src.test.java.org.apache.beam.examples.cookbook.CloudErrorReporting$WriteLogs$1.processElement
>  (CloudErrorReporting.java:94)
> at 
> src.test.java.org.apache.beam.examples.cookbook.CloudErrorReporting$WriteLogs$1$auxiliary$YZjse2Tt.invokeProcessElement
>  (Unknown Source)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3151: [BEAM-1663] Use stable naming strategy for ByteBudd...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3151

[BEAM-1663] Use stable naming strategy for ByteBuddy invokers

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

This helps to coalesce related failures across multiple JVM instances, over 
time, across machines, etc.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam bytebuddy-names

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3151.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3151


commit 515c0c846c63e5a05d792ce3b8c60aaa4acdd382
Author: Kenneth Knowles 
Date:   2017-03-09T01:02:05Z

Use stable naming strategy for ByteBuddy invokers

This helps to coalesce related failures across multiple
JVM instances, over time, across machines, etc.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2828

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-1868) CreateStreamTest testMultiOutputParDo is flaky on the Spark runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1868:
--
Description: 
Example excerpt from a Jenkins failure:

{code}
Expected: iterable over [<1>, <2>, <3>] in any order
 but: No item matches: <1>, <2>, <3> in []
at 
org.apache.beam.sdk.testing.PAssert$PAssertionSite.capture(PAssert.java:170)
at org.apache.beam.sdk.testing.PAssert.that(PAssert.java:366)
at org.apache.beam.sdk.testing.PAssert.that(PAssert.java:358)
at 
org.apache.beam.runners.spark.translation.streaming.CreateStreamTest.testMultiOutputParDo(CreateStreamTest.java:387)
{code}

  was:
Example failure:

https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-spark/3142/testReport/junit/org.apache.beam.runners.spark.translation.streaming/CreateStreamTest/testLateDataAccumulating/


> CreateStreamTest testMultiOutputParDo is flaky on the Spark runner
> --
>
> Key: BEAM-1868
> URL: https://issues.apache.org/jira/browse/BEAM-1868
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Thomas Groh
>  Labels: flake
>
> Example excerpt from a Jenkins failure:
> {code}
> Expected: iterable over [<1>, <2>, <3>] in any order
>  but: No item matches: <1>, <2>, <3> in []
>   at 
> org.apache.beam.sdk.testing.PAssert$PAssertionSite.capture(PAssert.java:170)
>   at org.apache.beam.sdk.testing.PAssert.that(PAssert.java:366)
>   at org.apache.beam.sdk.testing.PAssert.that(PAssert.java:358)
>   at 
> org.apache.beam.runners.spark.translation.streaming.CreateStreamTest.testMultiOutputParDo(CreateStreamTest.java:387)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1868) CreateStreamTest testMultiOutputParDo is flaky on the Spark runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1868:
--
Summary: CreateStreamTest testMultiOutputParDo is flaky on the Spark runner 
 (was: CreateStreamTest is flaky on the Spark runner)

> CreateStreamTest testMultiOutputParDo is flaky on the Spark runner
> --
>
> Key: BEAM-1868
> URL: https://issues.apache.org/jira/browse/BEAM-1868
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Thomas Groh
>  Labels: flake
>
> Example failure:
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-spark/3142/testReport/junit/org.apache.beam.runners.spark.translation.streaming/CreateStreamTest/testLateDataAccumulating/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #3814

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3150: Cherry-pick of #3149 into release-2.0.0

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/3150

Cherry-pick of #3149 into release-2.0.0

Cherry-pick of #3149 readAvros should't have proto Message upper bound

R: @davorbonaci 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam read-avros

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3150.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3150


commit 1f8b36b3993a069f97c78757b025cdc8d983b1b6
Author: Eugene Kirpichov 
Date:   2017-05-15T19:22:27Z

This closes #3149




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3149: readAvros should't have proto Message upper bound

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3149


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: readAvros should't have proto Message upper bound

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master 6e8d8d657 -> 88ec00860


readAvros should't have proto Message upper bound

Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/de1adefd
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/de1adefd
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/de1adefd

Branch: refs/heads/master
Commit: de1adefda958ae152eae4b7b94b3b78563e130aa
Parents: 6e8d8d6
Author: Neville Li 
Authored: Mon May 15 14:55:09 2017 -0400
Committer: GitHub 
Committed: Mon May 15 14:55:09 2017 -0400

--
 .../src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/de1adefd/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java
index bdc2752e..e293b95 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java
@@ -487,7 +487,7 @@ public class PubsubIO {
* Returns A {@link PTransform} that continuously reads binary encoded Avro 
messages of the
* given type from a Google Cloud Pub/Sub stream.
*/
-  public static  Read readAvros(Class clazz) {
+  public static  Read readAvros(Class clazz) {
 // TODO: Stop using AvroCoder and instead parse the payload directly.
 // We should not be relying on the fact that AvroCoder's wire format is 
identical to
 // the Avro wire format, as the wire format is not part of a coder's API.

[2/2] beam git commit: This closes #3149

[jkff]

This closes #3149


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/88ec0086
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/88ec0086
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/88ec0086

Branch: refs/heads/master
Commit: 88ec00860032078dcd33e07d94642912f6aa7bf7
Parents: 6e8d8d6 de1adef
Author: Eugene Kirpichov 
Authored: Mon May 15 12:22:27 2017 -0700
Committer: Eugene Kirpichov 
Committed: Mon May 15 12:22:27 2017 -0700

--
 .../src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[jira] [Commented] (BEAM-1013) Recheck all existing programming guide code snippets for correctness

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011160#comment-16011160
 ] 

Eugene Kirpichov commented on BEAM-1013:


The style-guide part of this is done. Melissa, is there more to be done here?

> Recheck all existing programming guide code snippets for correctness
> 
>
> Key: BEAM-1013
> URL: https://issues.apache.org/jira/browse/BEAM-1013
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>Assignee: Melissa Pashniak
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1013) Recheck all existing programming guide code snippets for correctness

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov reassigned BEAM-1013:
--

Assignee: Melissa Pashniak  (was: Eugene Kirpichov)

> Recheck all existing programming guide code snippets for correctness
> 
>
> Key: BEAM-1013
> URL: https://issues.apache.org/jira/browse/BEAM-1013
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>Assignee: Melissa Pashniak
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1013) Recheck all existing programming guide code snippets for correctness

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011153#comment-16011153
 ] 

ASF GitHub Bot commented on BEAM-1013:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/243


> Recheck all existing programming guide code snippets for correctness
> 
>
> Key: BEAM-1013
> URL: https://issues.apache.org/jira/browse/BEAM-1013
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>Assignee: Eugene Kirpichov
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #243: [BEAM-1013, BEAM-1353] Website fixups after PTr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/243


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[4/4] beam-site git commit: This closes #243

[jkff]

This closes #243


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/9ffe5ec5
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/9ffe5ec5
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/9ffe5ec5

Branch: refs/heads/asf-site
Commit: 9ffe5ec58b17976c3082bf31e6c1c68ee52b02a5
Parents: 9cc5b22 846a886
Author: Eugene Kirpichov 
Authored: Mon May 15 12:15:37 2017 -0700
Committer: Eugene Kirpichov 
Committed: Mon May 15 12:15:37 2017 -0700

--
 .../pipelines/create-your-pipeline/index.html   |  4 +-
 .../pipelines/design-your-pipeline/index.html   | 20 ++---
 .../documentation/programming-guide/index.html  | 94 
 .../sdks/java-extensions/index.html |  2 +-
 .../mobile-gaming-example/index.html| 20 ++---
 .../get-started/wordcount-example/index.html|  6 +-
 .../pipelines/create-your-pipeline.md   |  4 +-
 .../pipelines/design-your-pipeline.md   | 20 ++---
 src/documentation/programming-guide.md  | 92 ---
 src/documentation/sdks/java-extensions.md   |  2 +-
 src/get-started/mobile-gaming-example.md| 20 ++---
 src/get-started/wordcount-example.md|  6 +-
 12 files changed, 124 insertions(+), 166 deletions(-)
--

[2/4] beam-site git commit: Rewrites the section on Coders to not talk about them as a parsing mechanism

[jkff]

Rewrites the section on Coders to not talk about them as a parsing mechanism


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/0d0da026
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/0d0da026
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/0d0da026

Branch: refs/heads/asf-site
Commit: 0d0da0265d8a3ee07493feec835e56efd6acfd85
Parents: 9cc5b22
Author: Eugene Kirpichov 
Authored: Fri May 12 16:06:09 2017 -0700
Committer: Eugene Kirpichov 
Committed: Mon May 15 11:28:52 2017 -0700

--
 src/documentation/programming-guide.md | 38 ++---
 1 file changed, 8 insertions(+), 30 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/0d0da026/src/documentation/programming-guide.md
--
diff --git a/src/documentation/programming-guide.md 
b/src/documentation/programming-guide.md
index 11ec86d..f70e255 100644
--- a/src/documentation/programming-guide.md
+++ b/src/documentation/programming-guide.md
@@ -1175,11 +1175,9 @@ See the  [Beam-provided I/O Transforms]({{site.baseurl 
}}/documentation/io/built
 
 ## Data encoding and type safety
 
-When you create or output pipeline data, you'll need to specify how the 
elements in your `PCollection`s are encoded and decoded to and from byte 
strings. Byte strings are used for intermediate storage as well reading from 
sources and writing to sinks. The Beam SDKs use objects called coders to 
describe how the elements of a given `PCollection` should be encoded and 
decoded.
+When Beam runners execute your pipeline, they often need to materialize the 
intermediate data in your `PCollection`s, which requires converting elements to 
and from byte strings. The Beam SDKs use objects called `Coder`s to describe 
how the elements of a given `PCollection` may be encoded and decoded.
 
-### Using coders
-
-You typically need to specify a coder when reading data into your pipeline 
from an external source (or creating pipeline data from local data), and also 
when you output pipeline data to an external sink.
+> Note that coders are unrelated to parsing or formatting data when 
interacting with external data sources or sinks. Such parsing or formatting 
should typically be done explicitly, using transforms such as `ParDo` or 
`MapElements`.
 
 {:.language-java}
 In the Beam SDK for Java, the type `Coder` provides the methods required for 
encoding and decoding data. The SDK for Java provides a number of Coder 
subclasses that work with a variety of standard Java types, such as Integer, 
Long, Double, StringUtf8 and more. You can find all of the available Coder 
subclasses in the [Coder 
package](https://github.com/apache/beam/tree/master/sdks/java/core/src/main/java/org/apache/beam/sdk/coders).
@@ -1187,38 +1185,18 @@ In the Beam SDK for Java, the type `Coder` provides the 
methods required for enc
 {:.language-py}
 In the Beam SDK for Python, the type `Coder` provides the methods required for 
encoding and decoding data. The SDK for Python provides a number of Coder 
subclasses that work with a variety of standard Python types, such as primitive 
types, Tuple, Iterable, StringUtf8 and more. You can find all of the available 
Coder subclasses in the 
[apache_beam.coders](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/coders)
 package.
 
-When you read data into a pipeline, the coder indicates how to interpret the 
input data into a language-specific type, such as integer or string. Likewise, 
the coder indicates how the language-specific types in your pipeline should be 
written into byte strings for an output data sink, or to materialize 
intermediate data in your pipeline.
-
-The Beam SDKs set a coder for every `PCollection` in a pipeline, including 
those generated as output from a transform. Most of the time, the Beam SDKs can 
automatically infer the correct coder for an output `PCollection`.
-
 > Note that coders do not necessarily have a 1:1 relationship with types. For 
 > example, the Integer type can have multiple valid coders, and input and 
 > output data can use different Integer coders. A transform might have 
 > Integer-typed input data that uses BigEndianIntegerCoder, and Integer-typed 
 > output data that uses VarIntCoder.
 
-You can explicitly set a `Coder` when inputting or outputting a `PCollection`. 
You set the `Coder` by calling the method 
`.withCoder` setting the `coder` 
argument when you apply your pipeline's read or write transform.
-
-Typically, you set the `Coder` when the coder for a `PCollection` cannot be 
automatically inferred, or when you want to use a different coder than your 
pipeline's default. The following example code reads a set of numbers from 

[1/4] beam-site git commit: [BEAM-1353] Style Guide fixups

[jkff]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 9cc5b2280 -> 9ffe5ec58


[BEAM-1353] Style Guide fixups

Fixes usages of PTransforms affected by changes as part of
https://issues.apache.org/jira/browse/BEAM-1353


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/e78f8f27
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/e78f8f27
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/e78f8f27

Branch: refs/heads/asf-site
Commit: e78f8f276a6cbf3156e0f7af3dd4ae1d9b92ee7a
Parents: 0d0da02
Author: Eugene Kirpichov 
Authored: Fri May 12 16:07:19 2017 -0700
Committer: Eugene Kirpichov 
Committed: Mon May 15 11:28:52 2017 -0700

--
 .../pipelines/create-your-pipeline.md   |  4 +-
 .../pipelines/design-your-pipeline.md   | 20 
 src/documentation/programming-guide.md  | 54 ++--
 src/documentation/sdks/java-extensions.md   |  2 +-
 src/get-started/mobile-gaming-example.md| 20 
 src/get-started/wordcount-example.md|  6 +--
 6 files changed, 53 insertions(+), 53 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/e78f8f27/src/documentation/pipelines/create-your-pipeline.md
--
diff --git a/src/documentation/pipelines/create-your-pipeline.md 
b/src/documentation/pipelines/create-your-pipeline.md
index b765467..cbf7d31 100644
--- a/src/documentation/pipelines/create-your-pipeline.md
+++ b/src/documentation/pipelines/create-your-pipeline.md
@@ -42,7 +42,7 @@ The following example code shows how to `apply` a 
`TextIO.Read` root transform t
 
 ```java
 PCollection lines = p.apply(
-  "ReadLines", TextIO.Read.from("gs://some/inputData.txt"));
+  "ReadLines", TextIO.read().from("gs://some/inputData.txt"));
 ```
 
 ## Applying Transforms to Process Pipeline Data
@@ -68,7 +68,7 @@ The following example code shows how to `apply` a 
`TextIO.Write` transform to wr
 ```java
 PCollection filteredWords = ...;
 
-filteredWords.apply("WriteMyFile", 
TextIO.Write.to("gs://some/outputData.txt"));
+filteredWords.apply("WriteMyFile", 
TextIO.write().to("gs://some/outputData.txt"));
 ```
 
 ## Running Your Pipeline

http://git-wip-us.apache.org/repos/asf/beam-site/blob/e78f8f27/src/documentation/pipelines/design-your-pipeline.md
--
diff --git a/src/documentation/pipelines/design-your-pipeline.md 
b/src/documentation/pipelines/design-your-pipeline.md
index c40803c..ce6a734 100644
--- a/src/documentation/pipelines/design-your-pipeline.md
+++ b/src/documentation/pipelines/design-your-pipeline.md
@@ -103,13 +103,7 @@ final TupleTag startsWithATag = new 
TupleTag(){};
 final TupleTag startsWithBTag = new TupleTag(){};
 
 PCollectionTuple mixedCollection =
-dbRowCollection.apply(
-ParDo
-// Specify main output. In this example, it is the output
-// with tag startsWithATag.
-.withOutputTags(startsWithATag,
-// Specify the output with tag startsWithBTag, as a TupleTagList.
-TupleTagList.of(startsWithBTag))
+dbRowCollection.apply(ParDo
 .of(new DoFn() {
   @ProcessElement
   public void processElement(ProcessContext c) {
@@ -121,8 +115,12 @@ PCollectionTuple mixedCollection =
   c.output(startsWithBTag, c.element());
 }
   }
-}
-));
+})
+// Specify main output. In this example, it is the output
+// with tag startsWithATag.
+.withOutputTags(startsWithATag,
+// Specify the output with tag startsWithBTag, as a TupleTagList.
+TupleTagList.of(startsWithBTag)));
 
 // Get subset of the output with tag startsWithATag.
 mixedCollection.get(startsWithATag).apply(...);
@@ -159,7 +157,7 @@ mergedCollectionWithFlatten.apply(...);
 
 ## Multiple sources
 
-Your pipeline can read its input from one or more sources. If your pipeline 
reads from multiple sources and the data from those sources is related, it can 
be useful to join the inputs together. In the example illustrated in Figure 5 
below, the pipeline reads names and addresses from a database table, and names 
and order numbers from a text file. The pipeline then uses `CoGroupByKey` to 
join this information, where the key is the name; the resulting `PCollection` 
contains all the combinations of names, addresses, and orders.
+Your pipeline can read its input from one or more sources. If your pipeline 
reads from multiple sources and the data from those sources is related, it can 
be useful to join the inputs together. In the example illustrated in Figure 5 
below, the 

[3/4] beam-site git commit: Regenerate website

[jkff]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/846a8863
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/846a8863
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/846a8863

Branch: refs/heads/asf-site
Commit: 846a886371ca91513faf2f1db61641428ea05c2a
Parents: e78f8f2
Author: Eugene Kirpichov 
Authored: Mon May 15 12:15:03 2017 -0700
Committer: Eugene Kirpichov 
Committed: Mon May 15 12:15:03 2017 -0700

--
 .../pipelines/create-your-pipeline/index.html   |  4 +-
 .../pipelines/design-your-pipeline/index.html   | 20 ++---
 .../documentation/programming-guide/index.html  | 94 
 .../sdks/java-extensions/index.html |  2 +-
 .../mobile-gaming-example/index.html| 20 ++---
 .../get-started/wordcount-example/index.html|  6 +-
 6 files changed, 63 insertions(+), 83 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/846a8863/content/documentation/pipelines/create-your-pipeline/index.html
--
diff --git a/content/documentation/pipelines/create-your-pipeline/index.html 
b/content/documentation/pipelines/create-your-pipeline/index.html
index 6cfe938..4f31a5b 100644
--- a/content/documentation/pipelines/create-your-pipeline/index.html
+++ b/content/documentation/pipelines/create-your-pipeline/index.html
@@ -197,7 +197,7 @@
 The following example code shows how to apply a TextIO.Read root transform to read data from a 
text file. The transform is applied to a Pipeline object p, and returns a pipeline data set in the form 
of a PCollectionString:
 
 PCollectionString 
lines = p.apply(
-  "ReadLines", TextIO.Read.from("gs://some/inputData.txt"));
+  "ReadLines", TextIO.read().from("gs://some/inputData.txt"));
 
 
 
@@ -223,7 +223,7 @@
 
 PCollectionString 
filteredWords = ...;
 
-filteredWords.apply("WriteMyFile", TextIO.Write.to("gs://some/outputData.txt"));
+filteredWords.apply("WriteMyFile", TextIO.write().to("gs://some/outputData.txt"));
 
 
 

http://git-wip-us.apache.org/repos/asf/beam-site/blob/846a8863/content/documentation/pipelines/design-your-pipeline/index.html
--
diff --git a/content/documentation/pipelines/design-your-pipeline/index.html 
b/content/documentation/pipelines/design-your-pipeline/index.html
index 2185418..2e7018e 100644
--- a/content/documentation/pipelines/design-your-pipeline/index.html
+++ b/content/documentation/pipelines/design-your-pipeline/index.html
@@ -259,13 +259,7 @@
 final TupleTagString 
startsWithBTag = new TupleTagString(){};
 
 PCollectionTuple mixedCollection 
=
-dbRowCollection.apply(
-ParDo
-// Specify main output. In this example, it is the 
output
-// with tag startsWithATag.
-.withOutputTags(startsWithATag,
-// Specify the output with tag startsWithBTag, as a 
TupleTagList.
-TupleTagList.of(startsWithBTag))
+dbRowCollection.apply(ParDo
 .of(new DoFnString, 
String() {
   @ProcessElement
   public void processElement(ProcessContext c) {
@@ -277,8 +271,12 @@
   c.output(startsWithBTag, c.element());
 }
   }
-}
-));
+})
+// Specify main output. In this example, it is the 
output
+// with tag startsWithATag.
+.withOutputTags(startsWithATag,
+// Specify the output with tag startsWithBTag, as a 
TupleTagList.
+TupleTagList.of(startsWithBTag)));
 
 // Get subset of the output with tag startsWithATag.
 mixedCollection.get(startsWithATag).apply(...);
@@ -317,7 +315,7 @@
 
 Multiple sources
 
-Your pipeline can read its input from one or more sources. If your pipeline 
reads from multiple sources and the data from those sources is related, it can 
be useful to join the inputs together. In the example illustrated in Figure 5 
below, the pipeline reads names and addresses from a database table, and names 
and order numbers from a text file. The pipeline then uses CoGroupByKey to join this information, where 
the key is the name; the resulting PCollection contains all the combinations of 
names, addresses, and orders.
+Your pipeline can read its input from one or more sources. If your pipeline 
reads from multiple sources and the data from those sources is related, it can 
be useful to join the inputs together. In the example illustrated in Figure 5 
below, the pipeline reads names and addresses from a database table, and names 
and order numbers from a Kafka topic. The pipeline then uses CoGroupByKey to join this information, where 
the key is the name; the resulting PCollection