Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2715

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1534) Create a dockerized developer environment for Beam

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102747#comment-16102747
 ] 

Jean-Baptiste Onofré commented on BEAM-1534:


By the way, we also have a related Jira about creating Beam distributions. I 
plan to move forward on that one (as part of the build process by profile).

> Create a dockerized developer environment for Beam
> --
>
> Key: BEAM-1534
> URL: https://issues.apache.org/jira/browse/BEAM-1534
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-java
>Reporter: Nitin Lamba
>Assignee: Ekrem Aksoy
>
> This will help create a repeatable developer environment setup.
> Other Apache projects can be used as a reference:
> https://github.com/apache/ambari/tree/trunk/dev-support/docker
> https://github.com/apache/geode/blob/develop/docker/Dockerfile



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2687) Python SDK support for Stateful Processing

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-2687:
-

 Summary: Python SDK support for Stateful Processing
 Key: BEAM-2687
 URL: https://issues.apache.org/jira/browse/BEAM-2687
 Project: Beam
  Issue Type: New Feature
  Components: sdk-py
Reporter: Ahmet Altay


Python SDK should support stateful processing 
(https://beam.apache.org/blog/2017/02/13/stateful-processing.html)

In the meantime, runner capability matrix should be updated to show the lack of 
this feature (https://beam.apache.org/documentation/runners/capability-matrix/)

Use this as an umbrella issue for all related issues.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #2809

[Apache Jenkins Server]

See 


--
[...truncated 558.38 KB...]
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_undeclared_outputs (apache_beam.transforms.ptransform_test.PTransformTest) 
... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.4.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_multiple_empty_outputs 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.4.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_dict_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ... 
ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.4.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_list_and_as_dict_side_inputs 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
DEPRECATION: pip install --download has been

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3659

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3650: Translate ParDo and GroupByKey operations through t...

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/3650

Translate ParDo and GroupByKey operations through the Runner API

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam runner-api-pardo

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3650.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3650


commit d8d6839b6b574dabf054b29dcdf04de6c665a111
Author: Robert Bradshaw 
Date:   2017-07-26T23:30:56Z

Translate ParDo operations through the Runner API.

commit 4e72e07181d02ff39daa6ecfcf264f2af3f37e6b
Author: Robert Bradshaw 
Date:   2017-07-27T01:01:49Z

Streaming fixes.

commit f30c75f563dc2be24c805642c59b099f4287472e
Author: Robert Bradshaw 
Date:   2017-07-27T01:21:20Z

Translate GroupByKey[Only] through the Runner API.

commit a18966901e878aabd314ab3900d37078cd572d7c
Author: Robert Bradshaw 
Date:   2017-07-27T01:21:47Z

More informative references in the proto representation.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3658

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2714

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4463

[Apache Jenkins Server]

See 


--
[...truncated 1.01 MB...]
2017-07-27T00:31:41.009 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/net/java/dev/jna/jna/4.0.0/jna-4.0.0.pom 
(2 KB at 46.2 KB/sec)
2017-07-27T00:31:41.012 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/github/jbellis/jamm/0.3.0/jamm-0.3.0.pom
2017-07-27T00:31:41.039 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/jbellis/jamm/0.3.0/jamm-0.3.0.pom
 (5 KB at 164.4 KB/sec)
2017-07-27T00:31:41.042 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-all/4.0.39.Final/netty-all-4.0.39.Final.pom
2017-07-27T00:31:41.071 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-all/4.0.39.Final/netty-all-4.0.39.Final.pom
 (18 KB at 592.7 KB/sec)
2017-07-27T00:31:41.073 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-parent/4.0.39.Final/netty-parent-4.0.39.Final.pom
2017-07-27T00:31:41.107 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-parent/4.0.39.Final/netty-parent-4.0.39.Final.pom
 (44 KB at 1289.3 KB/sec)
2017-07-27T00:31:41.119 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/fusesource/sigar/1.6.4/sigar-1.6.4.pom
2017-07-27T00:31:41.145 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/fusesource/sigar/1.6.4/sigar-1.6.4.pom 
(2 KB at 47.4 KB/sec)
2017-07-27T00:31:41.147 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/fusesource/fusesource-pom/1.5/fusesource-pom-1.5.pom
2017-07-27T00:31:41.174 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/fusesource/fusesource-pom/1.5/fusesource-pom-1.5.pom
 (15 KB at 528.5 KB/sec)
2017-07-27T00:31:41.177 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/jdt/core/compiler/ecj/4.4.2/ecj-4.4.2.pom
2017-07-27T00:31:41.204 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/jdt/core/compiler/ecj/4.4.2/ecj-4.4.2.pom
 (3 KB at 75.0 KB/sec)
2017-07-27T00:31:41.206 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/caffinitas/ohc/ohc-core/0.4.3/ohc-core-0.4.3.pom
2017-07-27T00:31:41.234 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/caffinitas/ohc/ohc-core/0.4.3/ohc-core-0.4.3.pom
 (11 KB at 378.6 KB/sec)
2017-07-27T00:31:41.236 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/caffinitas/ohc/ohc-parent/0.4.3/ohc-parent-0.4.3.pom
2017-07-27T00:31:41.266 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/caffinitas/ohc/ohc-parent/0.4.3/ohc-parent-0.4.3.pom
 (17 KB at 543.3 KB/sec)
2017-07-27T00:31:41.269 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/github/ben-manes/caffeine/caffeine/2.2.6/caffeine-2.2.6.pom
2017-07-27T00:31:41.296 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/ben-manes/caffeine/caffeine/2.2.6/caffeine-2.2.6.pom
 (5 KB at 166.0 KB/sec)
2017-07-27T00:31:41.302 [INFO] Downloading: 
http://repository.apache.org/snapshots/io/netty/netty-codec-http2/maven-metadata.xml
2017-07-27T00:31:41.302 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-codec-http2/maven-metadata.xml
2017-07-27T00:31:41.333 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-codec-http2/maven-metadata.xml
 (2 KB at 41.1 KB/sec)
2017-07-27T00:31:41.465 [INFO] Downloading: 
https://oss.sonatype.org/content/repositories/snapshots/io/netty/netty-codec-http2/maven-metadata.xml
2017-07-27T00:31:41.465 [INFO] Downloading: 
https://repository.apache.org/content/repositories/snapshots/io/netty/netty-codec-http2/maven-metadata.xml
2017-07-27T00:31:41.732 [INFO] Downloaded: 
https://oss.sonatype.org/content/repositories/snapshots/io/netty/netty-codec-http2/maven-metadata.xml
 (594 B at 2.2 KB/sec)
[JENKINS] Archiving disabled
2017-07-27T00:31:42.609 [INFO]  
   
2017-07-27T00:31:42.609 [INFO] 

2017-07-27T00:31:42.609 [INFO] Skipping Apache Beam :: Parent
2017-07-27T00:31:42.609 [INFO] This project has been banned from the build due 
to previous failures.
2017-07-27T00:31:42.609 [INFO] 

[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archivi

[jira] [Commented] (BEAM-2686) PTransform's should always set a Coder on their output PCollections

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102521#comment-16102521
 ] 

ASF GitHub Bot commented on BEAM-2686:
--

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/3649

[BEAM-2686] Towards deprecating PCollection.setCoder()

https://issues.apache.org/jira/browse/BEAM-2686

- Renames Source.getDefaultOutputCoder to Source.getOutputCoder (in a 
compatible way), to avoid the implication that it is optional: it is not 
optional.
- Requires specifying a Coder on PCollection.createPrimitiveOutputInternal: 
as of this PR it can still be null (in which case coder will stay unset), but 
eventually it won't be
- Removes all overrides of PTransform.getDefaultOutputCoder and deprecates 
it: instead, PTransform's should explicitly .setCoder() on their output 
collection

This PR is 3 boring, almost mechanical commits, each of which is large but 
should be individually easy to review.

The next step (in a separate PR) will be getting rid of .setCoder() calls 
in the SDK. Nearly all such calls are on results of ParDo.of() and 
MapElements.via() and such - I'll need to add optional .withOutputCoder() 
parameters to those transforms.

R: @robertwb 
CC: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam set-coder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3649.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3649


commit c71762662812bb9b02b747790280c5d0a542b8e7
Author: Eugene Kirpichov 
Date:   2017-07-26T23:05:27Z

Makes all Source classes override getOutputCoder instead of 
getDefaultOutputCoder

commit 7bb48421f739eef4ebca0faedae2b8af69cae567
Author: Eugene Kirpichov 
Date:   2017-07-27T00:14:58Z

Requires specifying a Coder on PCollection.createPrimitiveOutputInternal

The coder can still be null, in which case it is left unspecified.

commit 3653e56b885eeccf9d749d7d5b32969e9af5e043
Author: Eugene Kirpichov 
Date:   2017-07-26T23:33:36Z

Removes all overrides of PTransform.getDefaultOutputCoder and deprecates it




> PTransform's should always set a Coder on their output PCollections
> ---
>
> Key: BEAM-2686
> URL: https://issues.apache.org/jira/browse/BEAM-2686
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>
> See discussion on 
> https://lists.apache.org/thread.html/1dde0b5a93c2983cbab5f68ce7c74580102f5bb2baaa816585d7eabb@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3649: [BEAM-2686] Towards deprecating PCollection.setCode...

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/3649

[BEAM-2686] Towards deprecating PCollection.setCoder()

https://issues.apache.org/jira/browse/BEAM-2686

- Renames Source.getDefaultOutputCoder to Source.getOutputCoder (in a 
compatible way), to avoid the implication that it is optional: it is not 
optional.
- Requires specifying a Coder on PCollection.createPrimitiveOutputInternal: 
as of this PR it can still be null (in which case coder will stay unset), but 
eventually it won't be
- Removes all overrides of PTransform.getDefaultOutputCoder and deprecates 
it: instead, PTransform's should explicitly .setCoder() on their output 
collection

This PR is 3 boring, almost mechanical commits, each of which is large but 
should be individually easy to review.

The next step (in a separate PR) will be getting rid of .setCoder() calls 
in the SDK. Nearly all such calls are on results of ParDo.of() and 
MapElements.via() and such - I'll need to add optional .withOutputCoder() 
parameters to those transforms.

R: @robertwb 
CC: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam set-coder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3649.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3649


commit c71762662812bb9b02b747790280c5d0a542b8e7
Author: Eugene Kirpichov 
Date:   2017-07-26T23:05:27Z

Makes all Source classes override getOutputCoder instead of 
getDefaultOutputCoder

commit 7bb48421f739eef4ebca0faedae2b8af69cae567
Author: Eugene Kirpichov 
Date:   2017-07-27T00:14:58Z

Requires specifying a Coder on PCollection.createPrimitiveOutputInternal

The coder can still be null, in which case it is left unspecified.

commit 3653e56b885eeccf9d749d7d5b32969e9af5e043
Author: Eugene Kirpichov 
Date:   2017-07-26T23:33:36Z

Removes all overrides of PTransform.getDefaultOutputCoder and deprecates it




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2686) PTransform's should always set a Coder on their output PCollections

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-2686:
--

 Summary: PTransform's should always set a Coder on their output 
PCollections
 Key: BEAM-2686
 URL: https://issues.apache.org/jira/browse/BEAM-2686
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Eugene Kirpichov


See discussion on 
https://lists.apache.org/thread.html/1dde0b5a93c2983cbab5f68ce7c74580102f5bb2baaa816585d7eabb@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #2808

[Apache Jenkins Server]

See 


Changes:

[robertwb] Migrate checkCombineFn in TestUtils to CombineFnTester

--
[...truncated 562.27 KB...]
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.4.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_list_and_as_dict_side_inputs 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.4.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_list_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ... 
ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.4.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_singleton_with_different_defaults 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.4.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_singleton_without_unique_labels 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcre

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2713

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2672) test merge in BeamSqlUdaf

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102472#comment-16102472
 ] 

ASF GitHub Bot commented on BEAM-2672:
--

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/3648

sync-up from master branch

BEAM-2672 requires a new function added in #3628 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/beam master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3648.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3648


commit 4192ac6c72765e8c55480eec66c499ef0798ecf3
Author: Stephen Sisk 
Date:   2017-06-14T16:57:35Z

Add maven support for invoking perfkit benchmarker to run IO ITs

commit 2f9cbec803791fd4e17fb28ec5590649aabb9622
Author: Stephen Sisk 
Date:   2017-07-20T18:38:40Z

io-it-suite-local independent of io-it-suite, k8s properties -> root pom

commit 7e63d2cf6c0ea0da4ce68b72536522f68b4a5272
Author: Thomas Groh 
Date:   2017-07-20T20:34:49Z

This closes #3588

commit c52a908cba7765e120a94909ab02c548d1a124ad
Author: Eugene Kirpichov 
Date:   2017-07-18T20:40:52Z

Minor changes to AvroSource in preparation for refactoring

commit d4026da1ad1fa0864052b85a66c4af5975327e9f
Author: Eugene Kirpichov 
Date:   2017-07-18T21:09:03Z

Gets rid of opening Avro files in createForSubrangeOfFile codepath

commit 1d9160fa3337704b9fbdc423796925be78b0087e
Author: Eugene Kirpichov 
Date:   2017-07-20T23:59:27Z

This closes #3590: [BEAM-2628] Makes AvroSource not open files while 
splitting

commit cfadecb56c64aa155c8b5fd0d8a6654ceb918eba
Author: Kenneth Knowles 
Date:   2017-07-18T05:03:09Z

Uniquify application nodes in TextIOReadTest and only run pipeline once

 - These nodes have never had stable unique names, but we have always 
mutated
   the pipeline before running it so they got lucky.
 - A pipeline can also be mutated when run() so it should be considered dead
   after run() is called.

This fixes both issues by uniquifying the names and running a given pipeline
only once.

commit c2110c97d530b8e90387bf99b3ac7d36201d85c7
Author: Thomas Groh 
Date:   2017-07-19T17:55:33Z

Register a PTransformTranslator for Combine

Include the Combine Payload in the Runner API Graph.

Add getCombineFn(AppliedPTransform) to extract the CombineFn from an
arbitrary transform.

Update Pipeline Translation tests to include accumulator coders.

commit b6f126dc6000ceba410d5074fd65af141b7ab3c6
Author: Thomas Groh 
Date:   2017-07-21T17:03:30Z

This closes #3597

commit 01492e69d1b1eedb99931f7071a9dae9d60d25fe
Author: Kenneth Knowles 
Date:   2017-07-20T03:58:36Z

Use RehydratedComponents for memoized rehydration

commit f870bf5164b00a1379f1e33a5ef27b3701a87c08
Author: Kenneth Knowles 
Date:   2017-07-21T18:06:02Z

This closes #3601: Use RehydratedComponents for memoized rehydration

commit e38dc5fbe1f45fe282b4cd96b858631db73130e9
Author: Kenneth Knowles 
Date:   2017-06-07T20:58:11Z

Use dehydration-insensitive APIs in WindowEvaluatorFactory

commit 8bd64759616a2f262f60ce2998b98f0dd1dd2000
Author: Kenneth Knowles 
Date:   2017-07-21T18:36:46Z

This closes #3609: Use dehydration-insensitive APIs in 
WindowEvaluatorFactory

commit fa3a5abbc94db629feae8d7d73a31e7dda06bf76
Author: Kenneth Knowles 
Date:   2017-06-07T21:35:09Z

Use dehydration-insensitive APIs in ParDoEvaluatorFactory

commit 4b355844a4920bc9faba75f7cd61008bedebaf29
Author: Kenneth Knowles 
Date:   2017-07-21T18:50:48Z

This closes #3610: Use dehydration-insensitive APIs in ParDoEvaluatorFactory

commit 81c2e90f3a9d109121b686c25b642cc9f16ced43
Author: Kenneth Knowles 
Date:   2017-07-21T19:21:05Z

This closes #3608: Uniquify application nodes in TextIOReadTest and only 
run pipeline once

commit 4f7d0b4ed70b40443aede1ce28d0939f08b1d04f
Author: Eugene Kirpichov 
Date:   2017-07-22T01:01:14Z

Unbreaks AvroSource compatibility with Dataflow worker

commit f39874860027832fdeff957afdc5cc8326de6d30
Author: Eugene Kirpichov 
Date:   2017-07-22T01:23:48Z

This closes #3616: Unbreaks AvroSource compatibility with Dataflow worker

commit a7c18495f91c9d7537271c9595b3ac4d24003ece
Author: Luke Cwik 
Date:   2017-07-23T21:00:17Z

[BEAM-2662] Fix Java quickstart for Spark to include Spark version

commit b1adbfbcf0a949a92a022c86b061ba3b64b69c9b
Author: Pei He 
Date:   2017-07-19T12:38:49Z

ReshuffleTest: replace Iterable equal tests with matchers.

commit 49d4cf4de48f121f583ddf7e2b7d0e56370a735c
Author: Pei He 
Date:   2017-07-24T02:53:27Z

This closes #3593

commit 5f1d1365ba73aeac5a76a31c64ead2d7e94e4040
Author: Jean-Baptiste Onofré 
Date:   2

[GitHub] beam pull request #3648: sync-up from master branch

[XuMingmin]

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/3648

sync-up from master branch

BEAM-2672 requires a new function added in #3628 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/beam master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3648.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3648


commit 4192ac6c72765e8c55480eec66c499ef0798ecf3
Author: Stephen Sisk 
Date:   2017-06-14T16:57:35Z

Add maven support for invoking perfkit benchmarker to run IO ITs

commit 2f9cbec803791fd4e17fb28ec5590649aabb9622
Author: Stephen Sisk 
Date:   2017-07-20T18:38:40Z

io-it-suite-local independent of io-it-suite, k8s properties -> root pom

commit 7e63d2cf6c0ea0da4ce68b72536522f68b4a5272
Author: Thomas Groh 
Date:   2017-07-20T20:34:49Z

This closes #3588

commit c52a908cba7765e120a94909ab02c548d1a124ad
Author: Eugene Kirpichov 
Date:   2017-07-18T20:40:52Z

Minor changes to AvroSource in preparation for refactoring

commit d4026da1ad1fa0864052b85a66c4af5975327e9f
Author: Eugene Kirpichov 
Date:   2017-07-18T21:09:03Z

Gets rid of opening Avro files in createForSubrangeOfFile codepath

commit 1d9160fa3337704b9fbdc423796925be78b0087e
Author: Eugene Kirpichov 
Date:   2017-07-20T23:59:27Z

This closes #3590: [BEAM-2628] Makes AvroSource not open files while 
splitting

commit cfadecb56c64aa155c8b5fd0d8a6654ceb918eba
Author: Kenneth Knowles 
Date:   2017-07-18T05:03:09Z

Uniquify application nodes in TextIOReadTest and only run pipeline once

 - These nodes have never had stable unique names, but we have always 
mutated
   the pipeline before running it so they got lucky.
 - A pipeline can also be mutated when run() so it should be considered dead
   after run() is called.

This fixes both issues by uniquifying the names and running a given pipeline
only once.

commit c2110c97d530b8e90387bf99b3ac7d36201d85c7
Author: Thomas Groh 
Date:   2017-07-19T17:55:33Z

Register a PTransformTranslator for Combine

Include the Combine Payload in the Runner API Graph.

Add getCombineFn(AppliedPTransform) to extract the CombineFn from an
arbitrary transform.

Update Pipeline Translation tests to include accumulator coders.

commit b6f126dc6000ceba410d5074fd65af141b7ab3c6
Author: Thomas Groh 
Date:   2017-07-21T17:03:30Z

This closes #3597

commit 01492e69d1b1eedb99931f7071a9dae9d60d25fe
Author: Kenneth Knowles 
Date:   2017-07-20T03:58:36Z

Use RehydratedComponents for memoized rehydration

commit f870bf5164b00a1379f1e33a5ef27b3701a87c08
Author: Kenneth Knowles 
Date:   2017-07-21T18:06:02Z

This closes #3601: Use RehydratedComponents for memoized rehydration

commit e38dc5fbe1f45fe282b4cd96b858631db73130e9
Author: Kenneth Knowles 
Date:   2017-06-07T20:58:11Z

Use dehydration-insensitive APIs in WindowEvaluatorFactory

commit 8bd64759616a2f262f60ce2998b98f0dd1dd2000
Author: Kenneth Knowles 
Date:   2017-07-21T18:36:46Z

This closes #3609: Use dehydration-insensitive APIs in 
WindowEvaluatorFactory

commit fa3a5abbc94db629feae8d7d73a31e7dda06bf76
Author: Kenneth Knowles 
Date:   2017-06-07T21:35:09Z

Use dehydration-insensitive APIs in ParDoEvaluatorFactory

commit 4b355844a4920bc9faba75f7cd61008bedebaf29
Author: Kenneth Knowles 
Date:   2017-07-21T18:50:48Z

This closes #3610: Use dehydration-insensitive APIs in ParDoEvaluatorFactory

commit 81c2e90f3a9d109121b686c25b642cc9f16ced43
Author: Kenneth Knowles 
Date:   2017-07-21T19:21:05Z

This closes #3608: Uniquify application nodes in TextIOReadTest and only 
run pipeline once

commit 4f7d0b4ed70b40443aede1ce28d0939f08b1d04f
Author: Eugene Kirpichov 
Date:   2017-07-22T01:01:14Z

Unbreaks AvroSource compatibility with Dataflow worker

commit f39874860027832fdeff957afdc5cc8326de6d30
Author: Eugene Kirpichov 
Date:   2017-07-22T01:23:48Z

This closes #3616: Unbreaks AvroSource compatibility with Dataflow worker

commit a7c18495f91c9d7537271c9595b3ac4d24003ece
Author: Luke Cwik 
Date:   2017-07-23T21:00:17Z

[BEAM-2662] Fix Java quickstart for Spark to include Spark version

commit b1adbfbcf0a949a92a022c86b061ba3b64b69c9b
Author: Pei He 
Date:   2017-07-19T12:38:49Z

ReshuffleTest: replace Iterable equal tests with matchers.

commit 49d4cf4de48f121f583ddf7e2b7d0e56370a735c
Author: Pei He 
Date:   2017-07-24T02:53:27Z

This closes #3593

commit 5f1d1365ba73aeac5a76a31c64ead2d7e94e4040
Author: Jean-Baptiste Onofré 
Date:   2017-07-24T04:57:27Z

[BEAM-2662] This closes #3620

commit b03c4f0790fa639d739f7f3fdeaa4a703fadb8fa
Author: Aljoscha Krettek 
Date:   2017-07-12T12:39:58Z

[BEAM-2571] Clarify pushedback variable name in DoFnOperator

commit 8449931708338dd854b

[jira] [Commented] (BEAM-2558) Add a CombineFnTester

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102466#comment-16102466
 ] 

ASF GitHub Bot commented on BEAM-2558:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3628


> Add a CombineFnTester
> -
>
> Key: BEAM-2558
> URL: https://issues.apache.org/jira/browse/BEAM-2558
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> This should be in the style of {{CoderProperties}} or {{EqualsTester}}: the 
> user should provide some inputs, and it should exercise potential paths by 
> which those elements may be accumulated and ensure that they all produce the 
> same results.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[1/2] beam git commit: Closes #3628

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master 673937b92 -> e206b1a41


Closes #3628


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e206b1a4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e206b1a4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e206b1a4

Branch: refs/heads/master
Commit: e206b1a41c1f22330cb78fd1082425ea1e74dbb4
Parents: 673937b 51867ad
Author: Robert Bradshaw 
Authored: Wed Jul 26 16:31:13 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Jul 26 16:31:13 2017 -0700

--
 .../beam/sdk/testing/CombineFnTester.java   | 147 ++
 .../java/org/apache/beam/sdk/TestUtils.java |  88 --
 .../beam/sdk/testing/CombineFnTesterTest.java   | 276 +++
 .../transforms/ApproximateQuantilesTest.java|  24 +-
 .../apache/beam/sdk/transforms/CombineTest.java |  12 +-
 .../org/apache/beam/sdk/transforms/MaxTest.java |   8 +-
 .../apache/beam/sdk/transforms/MeanTest.java|   4 +-
 .../org/apache/beam/sdk/transforms/MinTest.java |   8 +-
 .../org/apache/beam/sdk/transforms/SumTest.java |   8 +-
 9 files changed, 455 insertions(+), 120 deletions(-)
--

[2/2] beam git commit: Migrate checkCombineFn in TestUtils to CombineFnTester

[robertwb]

Migrate checkCombineFn in TestUtils to CombineFnTester

This makes CombineFnTester significantly more discoverable, and usable
without having dependencies on the test JAR.

Update existing tests.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/51867adf
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/51867adf
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/51867adf

Branch: refs/heads/master
Commit: 51867adfea0d78a7dcb0fb09b5c257cf8e0e2ff4
Parents: 673937b
Author: Thomas Groh 
Authored: Thu Jul 20 13:36:04 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Jul 26 16:31:13 2017 -0700

--
 .../beam/sdk/testing/CombineFnTester.java   | 147 ++
 .../java/org/apache/beam/sdk/TestUtils.java |  88 --
 .../beam/sdk/testing/CombineFnTesterTest.java   | 276 +++
 .../transforms/ApproximateQuantilesTest.java|  24 +-
 .../apache/beam/sdk/transforms/CombineTest.java |  12 +-
 .../org/apache/beam/sdk/transforms/MaxTest.java |   8 +-
 .../apache/beam/sdk/transforms/MeanTest.java|   4 +-
 .../org/apache/beam/sdk/transforms/MinTest.java |   8 +-
 .../org/apache/beam/sdk/transforms/SumTest.java |   8 +-
 9 files changed, 455 insertions(+), 120 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/51867adf/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/CombineFnTester.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/CombineFnTester.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/CombineFnTester.java
new file mode 100644
index 000..efd2af3
--- /dev/null
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/CombineFnTester.java
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.testing;
+
+import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertThat;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.sdk.transforms.Combine.CombineFn;
+import org.hamcrest.Matcher;
+
+/**
+ * Utilities for testing {@link CombineFn CombineFns}. Ensures that the {@link 
CombineFn} gives
+ * correct results across various permutations and shardings of the input.
+ */
+public class CombineFnTester {
+  /**
+   * Tests that the the {@link CombineFn}, when applied to the provided input, 
produces the provided
+   * output. Tests a variety of permutations of the input.
+   */
+  public static  void testCombineFn(
+  CombineFn fn, List input, final OutputT 
expected) {
+testCombineFn(fn, input, is(expected));
+Collections.shuffle(input);
+testCombineFn(fn, input, is(expected));
+  }
+
+  public static  void testCombineFn(
+  CombineFn fn, List input, Matcher matcher) {
+int size = input.size();
+checkCombineFnShardsMultipleOrders(fn, Collections.singletonList(input), 
matcher);
+checkCombineFnShardsMultipleOrders(fn, shardEvenly(input, 2), matcher);
+if (size > 4) {
+  checkCombineFnShardsMultipleOrders(fn, shardEvenly(input, size / 2), 
matcher);
+  checkCombineFnShardsMultipleOrders(
+  fn, shardEvenly(input, (int) (size / Math.sqrt(size))), matcher);
+}
+checkCombineFnShardsMultipleOrders(fn, shardExponentially(input, 1.4), 
matcher);
+checkCombineFnShardsMultipleOrders(fn, shardExponentially(input, 2), 
matcher);
+checkCombineFnShardsMultipleOrders(fn, shardExponentially(input, Math.E), 
matcher);
+  }
+
+  private static  void 
checkCombineFnShardsMultipleOrders(
+  CombineFn fn,
+  List> shards,
+  Matcher matcher) {
+checkCombineFnShardsSingleMerge(fn, shards, matcher);
+checkCombineFnShardsWithEmptyAccumulators(fn, shards, matcher);
+checkCombineFnShardsIncrementalMerging(fn, shards, matcher);
+Collections.shuffle(shards);
+checkCombineFnShardsSingleMerge(fn, shards, matcher);
+checkCombineFnShardsWithEmptyAccumulators(fn, sha

[GitHub] beam pull request #3628: [BEAM-2558] Migrate checkCombineFn in TestUtils to ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3628


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3657

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #4461

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2490) ReadFromText function is not taking all data with glob operator (*)

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102436#comment-16102436
 ] 

Guillermo Rodríguez Cano commented on BEAM-2490:


Sorry for the delay, I have been busy with other stuff at work and couldn't 
test this when I wanted.
Will do test tomorrow morning and get back asap.

> ReadFromText function is not taking all data with glob operator (*) 
> 
>
> Key: BEAM-2490
> URL: https://issues.apache.org/jira/browse/BEAM-2490
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Affects Versions: 2.0.0
> Environment: Usage with Google Cloud Platform: Dataflow runner
>Reporter: Olivier NGUYEN QUOC
>Assignee: Chamikara Jayalath
> Fix For: Not applicable
>
>
> I run a very simple pipeline:
> * Read my files from Google Cloud Storage
> * Split with '\n' char
> * Write in on a Google Cloud Storage
> I have 8 files that match with the pattern:
> * my_files_2016090116_20160902_060051_xx.csv.gz (229.25 MB)
> * my_files_2016090117_20160902_060051_xx.csv.gz (184.1 MB)
> * my_files_2016090118_20160902_060051_xx.csv.gz (171.73 MB)
> * my_files_2016090119_20160902_060051_xx.csv.gz (151.34 MB)
> * my_files_2016090120_20160902_060051_xx.csv.gz (129.69 MB)
> * my_files_2016090121_20160902_060051_xx.csv.gz (151.7 MB)
> * my_files_2016090122_20160902_060051_xx.csv.gz (346.46 MB)
> * my_files_2016090122_20160902_060051_xx.csv.gz (222.57 MB)
> This code should take them all:
> {code:python}
> beam.io.ReadFromText(
>   "gs://_folder1/my_files_20160901*.csv.gz",
>   skip_header_lines=1,
>   compression_type=beam.io.filesystem.CompressionTypes.GZIP
>   )
> {code}
> It runs well but there is only a 288.62 MB file in output of this pipeline 
> (instead of a 1.5 GB file).
> The whole pipeline code:
> {code:python}
> data = (p | 'ReadMyFiles' >> beam.io.ReadFromText(
>   "gs://_folder1/my_files_20160901*.csv.gz",
>   skip_header_lines=1,
>   compression_type=beam.io.filesystem.CompressionTypes.GZIP
>   )
>| 'SplitLines' >> beam.FlatMap(lambda x: x.split('\n'))
> )
> output = (
>   data| "Write" >> beam.io.WriteToText('gs://XXX_folder2/test.csv', 
> num_shards=1)
> )
> {code}
> Dataflow indicates me that the estimated size of the output after the 
> ReadFromText step is 602.29 MB only, which not correspond to any unique input 
> file size nor the overall file size matching with the pattern.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #2807

[Apache Jenkins Server]

See 


Changes:

[robertwb] Translate combining operations through the Runner API.

--
[...truncated 558.89 KB...]
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.4.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_dict_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ... 
ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.4.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_undeclared_outputs (apache_beam.transforms.ptransform_test.PTransformTest) 
... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.4.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_list_and_as_dict_side_inp

[jira] [Commented] (BEAM-2566) Java SDK harness should not depend on any runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102415#comment-16102415
 ] 

ASF GitHub Bot commented on BEAM-2566:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3647

[BEAM-2566] Remove References to CloudObject from the Java Harness

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---
Migrates to using the shared Runner API definitions.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam rm_cloud_object

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3647.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3647


commit 0119570cdfc855f182cd0e5650e5f2b22e11b3d6
Author: Thomas Groh 
Date:   2017-07-26T22:34:23Z

Remove References to CloudObject from the Java Harness

Migrates to using the shared Runner API definitions.




> Java SDK harness should not depend on any runner
> 
>
> Key: BEAM-2566
> URL: https://issues.apache.org/jira/browse/BEAM-2566
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> Right now there is a dependency on the Dataflow runner. I believe this is 
> legacy due to using {{CloudObject}} temporarily but I do not claim to 
> understand the full nature of the dependency.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3647: [BEAM-2566] Remove References to CloudObject from t...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3647

[BEAM-2566] Remove References to CloudObject from the Java Harness

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---
Migrates to using the shared Runner API definitions.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam rm_cloud_object

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3647.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3647


commit 0119570cdfc855f182cd0e5650e5f2b22e11b3d6
Author: Thomas Groh 
Date:   2017-07-26T22:34:23Z

Remove References to CloudObject from the Java Harness

Migrates to using the shared Runner API definitions.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2712

[Apache Jenkins Server]

See 

[jira] [Closed] (BEAM-1820) Source.getDefaultOutputCoder() should be @Nullable

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-1820.
--
   Resolution: Won't Fix
Fix Version/s: Not applicable

The consensus on dev@ is that instead, PTransform's should always provide a 
Coder for their output; and the natural generalization to Source is that 
sources also always should provide a Coder for their output; letting the user 
configure the Source with an explicit Coder if necessary.

> Source.getDefaultOutputCoder() should be @Nullable
> --
>
> Key: BEAM-1820
> URL: https://issues.apache.org/jira/browse/BEAM-1820
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Łukasz Gajowy
>  Labels: easyfix, starter
> Fix For: Not applicable
>
>
> Source.getDefaultOutputCoder() returns a coder for elements produced by the 
> source.
> However, the Source objects are nearly always hidden from the user and 
> instead encapsulated in a transform. Often, an enclosing transform has a 
> better idea of what coder should be used to encode these elements (e.g. a 
> user supplied a Coder to that transform's configuration). In that case, it'd 
> be good if Source.getDefaultOutputCoder() could just return null, and coder 
> would have to be handled by the enclosing transform or perhaps specified on 
> the output of that transform explicitly.
> Right now there's a bunch of code in the SDK and runners that assumes 
> Source.getDefaultOutputCoder() returns non-null. That code would need to be 
> fixed to instead use the coder set on the collection produced by 
> Read.from(source).
> It all appears pretty easy to fix, so this is a good starter item.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-1820) Source.getDefaultOutputCoder() should be @Nullable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102354#comment-16102354
 ] 

ASF GitHub Bot commented on BEAM-1820:
--

Github user jkff closed the pull request at:

https://github.com/apache/beam/pull/3634


> Source.getDefaultOutputCoder() should be @Nullable
> --
>
> Key: BEAM-1820
> URL: https://issues.apache.org/jira/browse/BEAM-1820
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Łukasz Gajowy
>  Labels: easyfix, starter
>
> Source.getDefaultOutputCoder() returns a coder for elements produced by the 
> source.
> However, the Source objects are nearly always hidden from the user and 
> instead encapsulated in a transform. Often, an enclosing transform has a 
> better idea of what coder should be used to encode these elements (e.g. a 
> user supplied a Coder to that transform's configuration). In that case, it'd 
> be good if Source.getDefaultOutputCoder() could just return null, and coder 
> would have to be handled by the enclosing transform or perhaps specified on 
> the output of that transform explicitly.
> Right now there's a bunch of code in the SDK and runners that assumes 
> Source.getDefaultOutputCoder() returns non-null. That code would need to be 
> fixed to instead use the coder set on the collection produced by 
> Read.from(source).
> It all appears pretty easy to fix, so this is a good starter item.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3634: [BEAM-1820] Source.getDefaultOutputCoder() throws C...

[jkff]

Github user jkff closed the pull request at:

https://github.com/apache/beam/pull/3634


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3640: Translate combining operations through the Runner A...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3640


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Translate combining operations through the Runner API.

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master b67a30bf1 -> 673937b92


Translate combining operations through the Runner API.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/82cc1e1a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/82cc1e1a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/82cc1e1a

Branch: refs/heads/master
Commit: 82cc1e1ad54d55d87ae912a96c148fd16503d017
Parents: b67a30b
Author: Robert Bradshaw 
Authored: Tue Jul 25 16:12:16 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Jul 26 14:55:39 2017 -0700

--
 .../runners/dataflow/dataflow_runner.py |  4 +--
 sdks/python/apache_beam/transforms/core.py  | 37 +++-
 .../python/apache_beam/transforms/ptransform.py | 12 +--
 sdks/python/apache_beam/utils/urns.py   |  5 ++-
 4 files changed, 52 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/82cc1e1a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py 
b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
index aec7d00..d653e91 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
@@ -591,8 +591,8 @@ class DataflowRunner(PipelineRunner):
  PropertyNames.OUTPUT_NAME: input_step.get_output(input_tag)})
 # Note that the accumulator must not have a WindowedValue encoding, while
 # the output of this step does in fact have a WindowedValue encoding.
-accumulator_encoding = self._get_encoded_output_coder(transform_node,
-  window_value=False)
+accumulator_encoding = self._get_cloud_encoding(
+transform_node.fn.get_accumulator_coder())
 output_encoding = self._get_encoded_output_coder(transform_node)
 
 step.encoding = output_encoding

http://git-wip-us.apache.org/repos/asf/beam/blob/82cc1e1a/sdks/python/apache_beam/transforms/core.py
--
diff --git a/sdks/python/apache_beam/transforms/core.py 
b/sdks/python/apache_beam/transforms/core.py
index 92b8737..25fe39f 100644
--- a/sdks/python/apache_beam/transforms/core.py
+++ b/sdks/python/apache_beam/transforms/core.py
@@ -25,6 +25,7 @@ import types
 
 from apache_beam import pvalue
 from apache_beam import typehints
+from apache_beam import coders
 from apache_beam.coders import typecoders
 from apache_beam.internal import util
 from apache_beam.portability.api import beam_runner_api_pb2
@@ -311,7 +312,7 @@ class CallableWrapperDoFn(DoFn):
 return getattr(self._fn, '_argspec_fn', self._fn)
 
 
-class CombineFn(WithTypeHints, HasDisplayData):
+class CombineFn(WithTypeHints, HasDisplayData, urns.RunnerApiFn):
   """A function object used by a Combine transform with custom processing.
 
   A CombineFn specifies how multiple values in all or part of a PCollection can
@@ -430,6 +431,11 @@ class CombineFn(WithTypeHints, HasDisplayData):
   def maybe_from_callable(fn):
 return fn if isinstance(fn, CombineFn) else CallableWrapperCombineFn(fn)
 
+  def get_accumulator_coder(self):
+return coders.registry.get_coder(object)
+
+  urns.RunnerApiFn.register_pickle_urn(urns.PICKLED_COMBINE_FN)
+
 
 class CallableWrapperCombineFn(CombineFn):
   """For internal use only; no backwards-compatibility guarantees.
@@ -816,6 +822,13 @@ def Filter(fn, *args, **kwargs):  # pylint: 
disable=invalid-name
   return pardo
 
 
+def _combine_payload(combine_fn, context):
+  return beam_runner_api_pb2.CombinePayload(
+  combine_fn=combine_fn.to_runner_api(context),
+  accumulator_coder_id=context.coders.get_id(
+  combine_fn.get_accumulator_coder()))
+
+
 class CombineGlobally(PTransform):
   """A CombineGlobally transform.
 
@@ -973,6 +986,17 @@ class CombinePerKey(PTransformWithSideInputs):
 return pcoll | GroupByKey() | 'Combine' >> CombineValues(
 self.fn, *args, **kwargs)
 
+  def to_runner_api_parameter(self, context):
+return (
+urns.COMBINE_PER_KEY_TRANSFORM,
+_combine_payload(self.fn, context))
+
+  @PTransform.register_urn(
+  urns.COMBINE_PER_KEY_TRANSFORM, beam_runner_api_pb2.CombinePayload)
+  def from_runner_api_parameter(combine_payload, context):
+return CombinePerKey(
+CombineFn.from_runner_api(combine_payload.combine_fn, context))
+
 
 # TODO(robertwb): Rename to CombineGroupedValues?
 class CombineValues(PTransformWithSideInputs):
@@ -995,6 +1019,17 @@ class CombineValues(PTransformWithSideInputs):
 CombineValuesDoFn(key_type, self.fn, runtime_type_check),
 *args, **kwargs)
 
+  de

[2/2] beam git commit: Closes #3640

[robertwb]

Closes #3640


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/673937b9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/673937b9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/673937b9

Branch: refs/heads/master
Commit: 673937b92a1496a18877e30689969f19432e7077
Parents: b67a30b 82cc1e1
Author: Robert Bradshaw 
Authored: Wed Jul 26 14:56:10 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Jul 26 14:56:10 2017 -0700

--
 .../runners/dataflow/dataflow_runner.py |  4 +--
 sdks/python/apache_beam/transforms/core.py  | 37 +++-
 .../python/apache_beam/transforms/ptransform.py | 12 +--
 sdks/python/apache_beam/utils/urns.py   |  5 ++-
 4 files changed, 52 insertions(+), 6 deletions(-)
--

[jira] [Created] (BEAM-2685) Fix issue with abc.ABCMeta causing dill failures

[Robert Bradshaw (JIRA)]

Robert Bradshaw created BEAM-2685:
-

 Summary: Fix issue with abc.ABCMeta causing dill failures
 Key: BEAM-2685
 URL: https://issues.apache.org/jira/browse/BEAM-2685
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Affects Versions: 2.0.0
Reporter: Robert Bradshaw
Assignee: Ahmet Altay
Priority: Minor
 Fix For: Not applicable


This may be a bug with Dill that needs to be filed upstream. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #2806

[Apache Jenkins Server]

See 


--
[...truncated 568.35 KB...]
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_list_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ... 
ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.4.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_list_and_as_dict_side_inputs 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.4.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_singleton_with_different_defaults 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.4.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_singleton_without_unique_labels 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements

[jira] [Commented] (BEAM-2676) move BeamSqlRow and BeamSqlRowType to sdk/java/core

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102317#comment-16102317
 ] 

Xu Mingmin commented on BEAM-2676:
--

SGTM, then we've all the type classes in one package. [~robertwb] make sense to 
you?

> move BeamSqlRow and BeamSqlRowType to sdk/java/core
> ---
>
> Key: BEAM-2676
> URL: https://issues.apache.org/jira/browse/BEAM-2676
> Project: Beam
>  Issue Type: Test
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
>
> BeamSqlRow/BeamSqlRowType is the fundamental of structured data processing in 
> Beam, like joins, simple projections/expansions. It's more visible to move 
> them to sdk-java-core.
> It contains two parts:
> 1). remove SQL word in the name,
> BeamSqlRow --> BeamRow
> BeamSqlRowType --> BeamRowType
> 2). move from package {{org.apache.beam.dsls.sql.schema}} to 
> {{org.apache.beam.sdk.sd}} (sd stands for structure data), in module 
> {{beam-sdks-java-core}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3656

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3655

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #4460

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2711

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2710

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-2684) AmqpIOTest is flaky

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-2684:
--

 Summary: AmqpIOTest is flaky
 Key: BEAM-2684
 URL: https://issues.apache.org/jira/browse/BEAM-2684
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Eugene Kirpichov
Assignee: Jean-Baptiste Onofré


This test is often timing out, and has been doing that for a while, causing 
unrelated PRs to fail randomly. I've gotten into the habit of excluding 
sdks/java/io/amqp when running "mvn verify" and I suppose it's not a good habit 
:) 

Example failure: 
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/13424/console



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3646: Do not merge: scratch work 2

[charlesccychen]

GitHub user charlesccychen opened a pull request:

https://github.com/apache/beam/pull/3646

Do not merge: scratch work 2

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/charlesccychen/beam streaming-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3646.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3646


commit c2b3f3222baf5dd7b1f107dc0bfcc7f8b5d49111
Author: Charles Chen 
Date:   2017-04-21T05:51:27Z

Don't buffer elements in GroupByKeyOnly.

commit 3383c260855150dc9f0182a15f37136ea1b1c93c
Author: Charles Chen 
Date:   2017-05-24T21:50:02Z

streaming test

commit 038f3f33297421237814124887564c6f2620711a
Author: Charles Chen 
Date:   2017-05-24T21:53:09Z

Working TestStream evaluator

commit fc661f6e22fb2d4b648af345527ed67a32776754
Author: Charles Chen 
Date:   2017-05-24T21:54:09Z

Slightly better watermark propagation, debugging print cleanup.

commit 053c7ae370df66d224469129f91fe8028ff3bcf4
Author: Charles Chen 
Date:   2017-05-24T21:55:42Z

Factor out GroupAlsoByWindows so that it can be overridden by the 
DirectRunner.

commit 8854c53c1bc75631bd3b09b3f7ed6412eb93f844
Author: Charles Chen 
Date:   2017-05-24T21:56:53Z

Skeleton implementation of GroupAlsoByKeyEvaluator.

commit ae0d80d0b945e8c50426696495e893565dc7db4a
Author: Charles Chen 
Date:   2017-04-24T15:16:05Z

Rename old non-keyed state to legacy_state, introduce KeyedWorkItem, 
prepare to expose DirectUnmergedState API.

commit 8be01a96055fb92013f6fc5aed8edc2af6e75e42
Author: Charles Chen 
Date:   2017-05-24T21:57:30Z

Mostly working streaming GroupAlsoByWindow.

commit bf38fe7afd219144e068d5dbd7035449f83bb723
Author: Charles Chen 
Date:   2017-05-24T21:58:08Z

Introduce keyed bundles, KeyedWorkItems.

commit 97de55600417145702576424dc7512de3f981838
Author: Charles Chen 
Date:   2017-05-24T22:01:59Z

Fix rebase to head after Beam 2.0 release.

commit 341fa9245e5d67cc5fab4244bfeca5ea49c72231
Author: Charles Chen 
Date:   2017-05-25T21:22:03Z

Clarify transform_keyed_states.

commit 1e83d5e900c7039ccf89d989081ae50b3b3e510f
Author: Charles Chen 
Date:   2017-05-26T00:38:31Z

Support timer firings.

commit 256309f2f3b01b230b258b8001a494d75e3e1713
Author: Charles Chen 
Date:   2017-05-26T01:08:44Z

Reintroduce hacks from head.

commit 25ce9f18a124e8d0aa80e05a516699da6a067c19
Author: Charles Chen 
Date:   2017-05-26T21:12:59Z

Add debugging output.

commit 7be85c67c5c405c16c146e5605dd2a5bb9bd3bdd
Author: Charles Chen 
Date:   2017-05-26T22:56:32Z

Don't hold input watermark on pending elements.

commit 1df07a5b46811d0d3a383b156121d77f5f37aa57
Author: Charles Chen 
Date:   2017-05-30T18:26:03Z

Finish moving earliest hold extraction to evaluators, dbg output++

commit e52ba42a9efe0961ed57da52e93fdd8e5970eedc
Author: Charles Chen 
Date:   2017-05-31T22:46:06Z

Batch WordCount works in streaming mode!!

Fixes earliest hold to be < instead of <=.

commit e5bb716e00ac02679edc3a357153009b862bb4d0
Author: Charles Chen 
Date:   2017-06-05T22:20:37Z

Working batch and streaming mode execution, except for certain empty side
inputs.

commit 91873cc1422d5bc7812464feea6789a1c4242577
Author: Robert Bradshaw 
Date:   2017-06-07T15:48:49Z

PubSubIO for direct runner.

commit e05778fc1245b75a34f63b4a29d1e7c2decab053
Author: Charles Chen 
Date:   2017-06-07T17:14:07Z

Merge pull request #1 from robertwb/streaming

PubSubIO for direct runner.

commit 12a95178163ef3bf56f49f6276194e2ab0068999
Author: Charles Chen 
Date:   2017-06-08T22:12:14Z

Use keyed holds in watermark manager

commit b90a2e45b6b39d3d6c513218a1ec3ae9e1edca27
Author: Charles Chen 
Date:   2017-06-0

[1/4] beam git commit: Disable abc metaclass due to issues with pickling.

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master 69a51c988 -> b67a30bf1


Disable abc metaclass due to issues with pickling.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/24b39e45
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/24b39e45
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/24b39e45

Branch: refs/heads/master
Commit: 24b39e457519f45df55a90a6c6c92fa8df5128b7
Parents: 22e49e6
Author: Robert Bradshaw 
Authored: Mon Jul 24 17:19:47 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Jul 26 10:55:53 2017 -0700

--
 sdks/python/apache_beam/utils/urns.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/24b39e45/sdks/python/apache_beam/utils/urns.py
--
diff --git a/sdks/python/apache_beam/utils/urns.py 
b/sdks/python/apache_beam/utils/urns.py
index c155cfd..9e4635d 100644
--- a/sdks/python/apache_beam/utils/urns.py
+++ b/sdks/python/apache_beam/utils/urns.py
@@ -53,7 +53,8 @@ class RunnerApiFn(object):
   to register serialization via pickling.
   """
 
-  __metaclass__ = abc.ABCMeta
+  # TODO(robertwb): Figure out issue with dill + local classes + abc metaclass
+  # __metaclass__ = abc.ABCMeta
 
   _known_urns = {}
 

[GitHub] beam pull request #3635: Translate reads/sources through the Runner API

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3635


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[4/4] beam git commit: Closes #3635

[robertwb]

Closes #3635


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b67a30bf
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b67a30bf
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b67a30bf

Branch: refs/heads/master
Commit: b67a30bf1586990c56fc19e1510417bdab71f280
Parents: 69a51c9 2d307f2
Author: Robert Bradshaw 
Authored: Wed Jul 26 10:55:54 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Jul 26 10:55:54 2017 -0700

--
 sdks/python/apache_beam/io/gcp/pubsub.py|  3 ++
 sdks/python/apache_beam/io/iobase.py| 31 +++-
 .../runners/dataflow/native_io/iobase.py|  6 +++-
 sdks/python/apache_beam/utils/urns.py   |  6 +++-
 4 files changed, 43 insertions(+), 3 deletions(-)
--

[2/4] beam git commit: Use runner API for read operation.

[robertwb]

Use runner API for read operation.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/22e49e60
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/22e49e60
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/22e49e60

Branch: refs/heads/master
Commit: 22e49e6035c54f95c604be8cf15daecba458240a
Parents: 69a51c9
Author: Robert Bradshaw 
Authored: Mon Jul 17 16:51:52 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Jul 26 10:55:53 2017 -0700

--
 sdks/python/apache_beam/io/gcp/pubsub.py|  3 +++
 sdks/python/apache_beam/io/iobase.py| 27 +++-
 .../runners/dataflow/native_io/iobase.py|  8 +-
 sdks/python/apache_beam/utils/urns.py   |  3 +++
 4 files changed, 39 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/22e49e60/sdks/python/apache_beam/io/gcp/pubsub.py
--
diff --git a/sdks/python/apache_beam/io/gcp/pubsub.py 
b/sdks/python/apache_beam/io/gcp/pubsub.py
index 32d388a..7d1f355 100644
--- a/sdks/python/apache_beam/io/gcp/pubsub.py
+++ b/sdks/python/apache_beam/io/gcp/pubsub.py
@@ -183,6 +183,9 @@ class _PubSubPayloadSource(dataflow_io.NativeSource):
 raise NotImplementedError(
 'PubSubPayloadSource is not supported in local execution.')
 
+  def is_bounded(self):
+return False
+
 
 class _PubSubPayloadSink(dataflow_io.NativeSink):
   """Sink for the payload of a message as bytes to a Cloud Pub/Sub topic."""

http://git-wip-us.apache.org/repos/asf/beam/blob/22e49e60/sdks/python/apache_beam/io/iobase.py
--
diff --git a/sdks/python/apache_beam/io/iobase.py 
b/sdks/python/apache_beam/io/iobase.py
index 7e40d83..bdd06e1 100644
--- a/sdks/python/apache_beam/io/iobase.py
+++ b/sdks/python/apache_beam/io/iobase.py
@@ -37,6 +37,7 @@ import uuid
 
 from apache_beam import pvalue
 from apache_beam import coders
+from apache_beam.portability.api import beam_runner_api_pb2
 from apache_beam.pvalue import AsIter
 from apache_beam.pvalue import AsSingleton
 from apache_beam.transforms import core
@@ -44,6 +45,7 @@ from apache_beam.transforms import ptransform
 from apache_beam.transforms import window
 from apache_beam.transforms.display import HasDisplayData
 from apache_beam.transforms.display import DisplayDataItem
+from apache_beam.utils import urns
 from apache_beam.utils.windowed_value import WindowedValue
 
 __all__ = ['BoundedSource', 'RangeTracker', 'Read', 'Sink', 'Write', 'Writer']
@@ -70,7 +72,7 @@ SourceBundle = namedtuple(
 'weight source start_position stop_position')
 
 
-class BoundedSource(HasDisplayData):
+class BoundedSource(HasDisplayData, urns.RunnerApiFn):
   """A source that reads a finite amount of input records.
 
   This class defines following operations which can be used to read the source
@@ -189,6 +191,11 @@ class BoundedSource(HasDisplayData):
 """
 return coders.registry.get_coder(object)
 
+  def is_bounded(self):
+return True
+
+  urns.RunnerApiFn.register_pickle_urn(urns.PICKLED_SOURCE)
+
 
 class RangeTracker(object):
   """A thread safe object used by Dataflow source framework.
@@ -820,6 +827,24 @@ class Read(ptransform.PTransform):
   label='Read Source'),
 'source_dd': self.source}
 
+  def to_runner_api_parameter(self, context):
+return (urns.READ_TRANSFORM,
+beam_runner_api_pb2.ReadPayload(
+source=self.source.to_runner_api(context),
+is_bounded=beam_runner_api_pb2.BOUNDED
+if self.source.is_bounded()
+else beam_runner_api_pb2.UNBOUNDED))
+
+  @staticmethod
+  def from_runner_api_parameter(parameter, context):
+return Read(BoundedSource.from_runner_api(parameter.source, context))
+
+
+ptransform.PTransform.register_urn(
+urns.READ_TRANSFORM,
+beam_runner_api_pb2.ReadPayload,
+Read.from_runner_api_parameter)
+
 
 class Write(ptransform.PTransform):
   """A ``PTransform`` that writes to a sink.

http://git-wip-us.apache.org/repos/asf/beam/blob/22e49e60/sdks/python/apache_beam/runners/dataflow/native_io/iobase.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/native_io/iobase.py 
b/sdks/python/apache_beam/runners/dataflow/native_io/iobase.py
index c1f4238..f29b714 100644
--- a/sdks/python/apache_beam/runners/dataflow/native_io/iobase.py
+++ b/sdks/python/apache_beam/runners/dataflow/native_io/iobase.py
@@ -25,6 +25,7 @@ import logging
 from apache_beam import pvalue
 from apache_beam.transforms import ptransform
 from apache_beam.transforms.display import HasDisplayData
+from apache_beam.utils import urns

[3/4] beam git commit: Create common superclass for all Sources.

[robertwb]

Create common superclass for all Sources.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2d307f29
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2d307f29
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2d307f29

Branch: refs/heads/master
Commit: 2d307f29b77e6aeb28f624f50246e99422a1173d
Parents: 24b39e4
Author: Robert Bradshaw 
Authored: Tue Jul 25 13:33:03 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Jul 26 10:55:54 2017 -0700

--
 sdks/python/apache_beam/io/iobase.py| 12 
 .../apache_beam/runners/dataflow/native_io/iobase.py|  6 ++
 2 files changed, 10 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2d307f29/sdks/python/apache_beam/io/iobase.py
--
diff --git a/sdks/python/apache_beam/io/iobase.py 
b/sdks/python/apache_beam/io/iobase.py
index bdd06e1..db75fe3 100644
--- a/sdks/python/apache_beam/io/iobase.py
+++ b/sdks/python/apache_beam/io/iobase.py
@@ -72,7 +72,13 @@ SourceBundle = namedtuple(
 'weight source start_position stop_position')
 
 
-class BoundedSource(HasDisplayData, urns.RunnerApiFn):
+class SourceBase(HasDisplayData, urns.RunnerApiFn):
+  """Base class for all sources that can be passed to beam.io.Read(...).
+  """
+  urns.RunnerApiFn.register_pickle_urn(urns.PICKLED_SOURCE)
+
+
+class BoundedSource(SourceBase):
   """A source that reads a finite amount of input records.
 
   This class defines following operations which can be used to read the source
@@ -194,8 +200,6 @@ class BoundedSource(HasDisplayData, urns.RunnerApiFn):
   def is_bounded(self):
 return True
 
-  urns.RunnerApiFn.register_pickle_urn(urns.PICKLED_SOURCE)
-
 
 class RangeTracker(object):
   """A thread safe object used by Dataflow source framework.
@@ -837,7 +841,7 @@ class Read(ptransform.PTransform):
 
   @staticmethod
   def from_runner_api_parameter(parameter, context):
-return Read(BoundedSource.from_runner_api(parameter.source, context))
+return Read(SourceBase.from_runner_api(parameter.source, context))
 
 
 ptransform.PTransform.register_urn(

http://git-wip-us.apache.org/repos/asf/beam/blob/2d307f29/sdks/python/apache_beam/runners/dataflow/native_io/iobase.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/native_io/iobase.py 
b/sdks/python/apache_beam/runners/dataflow/native_io/iobase.py
index f29b714..2f2316f 100644
--- a/sdks/python/apache_beam/runners/dataflow/native_io/iobase.py
+++ b/sdks/python/apache_beam/runners/dataflow/native_io/iobase.py
@@ -23,9 +23,9 @@ For internal use only; no backwards-compatibility guarantees.
 import logging
 
 from apache_beam import pvalue
+from apache_beam.io import iobase
 from apache_beam.transforms import ptransform
 from apache_beam.transforms.display import HasDisplayData
-from apache_beam.utils import urns
 
 
 def _dict_printable_fields(dict_object, skip_fields):
@@ -43,7 +43,7 @@ _minor_fields = ['coder', 'key_coder', 'value_coder',
  'compression_type']
 
 
-class NativeSource(HasDisplayData, urns.RunnerApiFn):
+class NativeSource(iobase.SourceBase):
   """A source implemented by Dataflow service.
 
   This class is to be only inherited by sources natively implemented by Cloud
@@ -65,8 +65,6 @@ class NativeSource(HasDisplayData, urns.RunnerApiFn):
 vals=', '.join(_dict_printable_fields(self.__dict__,
   _minor_fields)))
 
-  urns.RunnerApiFn.register_pickle_urn(urns.PICKLED_SOURCE)
-
 
 class NativeSourceReader(object):
   """A reader for a source implemented by Dataflow service."""

[GitHub] beam pull request #3645: Fixes BigQueryIO.Write javadoc for to(ValueInSingle...

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/3645

Fixes BigQueryIO.Write javadoc for to(ValueInSingleWindow)

R: @reuvenlax 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam bqio-javadoc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3645.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3645






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-2681) TransformHierarchy.Node getInputs() TupleTags doesn't match previous ParDo.MultiOutput's getOutputs().

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-2681.
---
   Resolution: Not A Problem
Fix Version/s: Not applicable

This is expected. A {{TupleTag}} is a local identifier of a {{PCollection}}; it 
should only be used in the context in which it is declared.

If this was not the case, meaningful expansion of a {{PCollectionTuple}} would 
not be possible.

> TransformHierarchy.Node getInputs() TupleTags doesn't match previous 
> ParDo.MultiOutput's getOutputs().
> --
>
> Key: BEAM-2681
> URL: https://issues.apache.org/jira/browse/BEAM-2681
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
>Assignee: Thomas Groh
>Priority: Minor
> Fix For: Not applicable
>
>
> For example,
> ParDo.MultiOutput ->PCollectionTuple [tag1 -> pc1 (tag_pc1), tag2 -> pc2 
> (tag_pc2)]
> pc1 (tag_pc1) -> transform_a
> pc2 (tag_pc2) -> transform_b
> During translation, calling getOutputs() for Node ParDo.MultiOutput will 
> returns map keyed by tag1 and tag2, and calling getInputs() for Node 
> transform_a and transform_b will returns map keyed by tag_pc1 and tag_pc2 
> respectively.
> Every runner will need to add special case for ParDo translation in order to 
> handle this.
> Is this a bug or intended? (If it is intended, should we document it 
> somewhere for runner implementors?)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3654

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1534) Create a dockerized developer environment for Beam

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101866#comment-16101866
 ] 

Ismaël Mejía commented on BEAM-1534:


Hi, I would like to know if you there is progress on this issue. I ended up 
writing some docker scripts to validate the releases/prs and I wanted to know 
if I can take the ticket to contribute them.

https://github.com/iemejia/beam/tree/BEAM-1534-dev/sdks/java/build-tools/src/main/resources/docker

> Create a dockerized developer environment for Beam
> --
>
> Key: BEAM-1534
> URL: https://issues.apache.org/jira/browse/BEAM-1534
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-java
>Reporter: Nitin Lamba
>Assignee: Ekrem Aksoy
>
> This will help create a repeatable developer environment setup.
> Other Apache projects can be used as a reference:
> https://github.com/apache/ambari/tree/trunk/dev-support/docker
> https://github.com/apache/geode/blob/develop/docker/Dockerfile



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #4458

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2709

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-2658) SerializableCoder has higher precedence over ProtoCoder in CoderRegistry#getCoder

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-2658.
-
   Resolution: Fixed
Fix Version/s: 2.2.0

> SerializableCoder has higher precedence over ProtoCoder in 
> CoderRegistry#getCoder
> -
>
> Key: BEAM-2658
> URL: https://issues.apache.org/jira/browse/BEAM-2658
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.0.0
>Reporter: Neville Li
>Assignee: Neville Li
>Priority: Minor
> Fix For: 2.2.0
>
>
> {code}
> import com.google.protobuf.Timestamp;
> import org.apache.beam.sdk.Pipeline;
> import org.apache.beam.sdk.coders.CannotProvideCoderException;
> import org.apache.beam.sdk.coders.Coder;
> import org.apache.beam.sdk.options.PipelineOptions;
> import org.apache.beam.sdk.options.PipelineOptionsFactory;
> public class CoderTest {
>   public static void main(String[] args) throws CannotProvideCoderException {
> PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
> Pipeline pipeline = Pipeline.create(options);
> Coder coder = 
> pipeline.getCoderRegistry().getCoder(Timestamp.class);
> // class org.apache.beam.sdk.coders.SerializableCoder
> System.out.println(coder.getClass());
>   }
> }
> {code}
> Right now we're sorting {{CoderProviderRegistrar}}s by canonical name but 
> {{SerializableCoderProvider}} should be added last as a fallback if there're 
> other {{CoderProvider}}s that support the same type.
> {code}
> Set registrars = 
> Sets.newTreeSet(ObjectsClassComparator.INSTANCE);
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-2658) SerializableCoder has higher precedence over ProtoCoder in CoderRegistry#getCoder

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-2658:
---

Assignee: Neville Li  (was: Davor Bonaci)

> SerializableCoder has higher precedence over ProtoCoder in 
> CoderRegistry#getCoder
> -
>
> Key: BEAM-2658
> URL: https://issues.apache.org/jira/browse/BEAM-2658
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.0.0
>Reporter: Neville Li
>Assignee: Neville Li
>Priority: Minor
>
> {code}
> import com.google.protobuf.Timestamp;
> import org.apache.beam.sdk.Pipeline;
> import org.apache.beam.sdk.coders.CannotProvideCoderException;
> import org.apache.beam.sdk.coders.Coder;
> import org.apache.beam.sdk.options.PipelineOptions;
> import org.apache.beam.sdk.options.PipelineOptionsFactory;
> public class CoderTest {
>   public static void main(String[] args) throws CannotProvideCoderException {
> PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
> Pipeline pipeline = Pipeline.create(options);
> Coder coder = 
> pipeline.getCoderRegistry().getCoder(Timestamp.class);
> // class org.apache.beam.sdk.coders.SerializableCoder
> System.out.println(coder.getClass());
>   }
> }
> {code}
> Right now we're sorting {{CoderProviderRegistrar}}s by canonical name but 
> {{SerializableCoderProvider}} should be added last as a fallback if there're 
> other {{CoderProvider}}s that support the same type.
> {code}
> Set registrars = 
> Sets.newTreeSet(ObjectsClassComparator.INSTANCE);
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2658) SerializableCoder has higher precedence over ProtoCoder in CoderRegistry#getCoder

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101760#comment-16101760
 ] 

ASF GitHub Bot commented on BEAM-2658:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3617


> SerializableCoder has higher precedence over ProtoCoder in 
> CoderRegistry#getCoder
> -
>
> Key: BEAM-2658
> URL: https://issues.apache.org/jira/browse/BEAM-2658
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.0.0
>Reporter: Neville Li
>Assignee: Davor Bonaci
>Priority: Minor
>
> {code}
> import com.google.protobuf.Timestamp;
> import org.apache.beam.sdk.Pipeline;
> import org.apache.beam.sdk.coders.CannotProvideCoderException;
> import org.apache.beam.sdk.coders.Coder;
> import org.apache.beam.sdk.options.PipelineOptions;
> import org.apache.beam.sdk.options.PipelineOptionsFactory;
> public class CoderTest {
>   public static void main(String[] args) throws CannotProvideCoderException {
> PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
> Pipeline pipeline = Pipeline.create(options);
> Coder coder = 
> pipeline.getCoderRegistry().getCoder(Timestamp.class);
> // class org.apache.beam.sdk.coders.SerializableCoder
> System.out.println(coder.getClass());
>   }
> }
> {code}
> Right now we're sorting {{CoderProviderRegistrar}}s by canonical name but 
> {{SerializableCoderProvider}} should be added last as a fallback if there're 
> other {{CoderProvider}}s that support the same type.
> {code}
> Set registrars = 
> Sets.newTreeSet(ObjectsClassComparator.INSTANCE);
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3617: [BEAM-2658] register SerializableCoder last

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3617


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: [BEAM-2658] register SerializableCoder last

[lcwik]

[BEAM-2658] register SerializableCoder last

This closes #3617


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/69a51c98
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/69a51c98
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/69a51c98

Branch: refs/heads/master
Commit: 69a51c98865b6a09ac9d6c4402d38854223b4281
Parents: bc72c94 809a7f8
Author: Luke Cwik 
Authored: Wed Jul 26 07:54:04 2017 -0700
Committer: Luke Cwik 
Committed: Wed Jul 26 07:54:04 2017 -0700

--
 .../apache/beam/sdk/coders/CoderRegistry.java   |  4 ++
 .../apache/beam/sdk/coders/DefaultCoder.java|  2 -
 .../beam/sdk/coders/SerializableCoder.java  |  2 -
 .../beam/sdk/coders/CoderRegistryTest.java  | 69 
 4 files changed, 73 insertions(+), 4 deletions(-)
--

[1/2] beam git commit: [BEAM-2658] register DefaultCoder first and SerializableCoder last

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master bc72c9405 -> 69a51c988


[BEAM-2658] register DefaultCoder first and SerializableCoder last


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/809a7f86
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/809a7f86
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/809a7f86

Branch: refs/heads/master
Commit: 809a7f861bcdfb2ccba3690b439940c702c49701
Parents: bc72c94
Author: Neville Li 
Authored: Sat Jul 22 00:14:03 2017 -0700
Committer: Luke Cwik 
Committed: Wed Jul 26 07:53:31 2017 -0700

--
 .../apache/beam/sdk/coders/CoderRegistry.java   |  4 ++
 .../apache/beam/sdk/coders/DefaultCoder.java|  2 -
 .../beam/sdk/coders/SerializableCoder.java  |  2 -
 .../beam/sdk/coders/CoderRegistryTest.java  | 69 
 4 files changed, 73 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/809a7f86/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java
index 53cb6d3..48389b1 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java
@@ -147,9 +147,13 @@ public class CoderRegistry {
 Set registrars = 
Sets.newTreeSet(ObjectsClassComparator.INSTANCE);
 registrars.addAll(Lists.newArrayList(
 ServiceLoader.load(CoderProviderRegistrar.class, 
ReflectHelpers.findClassLoader(;
+
+// DefaultCoder should have the highest precedence and SerializableCoder 
the lowest
+codersToRegister.addAll(new 
DefaultCoder.DefaultCoderProviderRegistrar().getCoderProviders());
 for (CoderProviderRegistrar registrar : registrars) {
 codersToRegister.addAll(registrar.getCoderProviders());
 }
+codersToRegister.add(SerializableCoder.getCoderProvider());
 
 REGISTERED_CODER_FACTORIES = ImmutableList.copyOf(codersToRegister);
   }

http://git-wip-us.apache.org/repos/asf/beam/blob/809a7f86/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/DefaultCoder.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/DefaultCoder.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/DefaultCoder.java
index 6eff9e9..7eb2ecb 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/DefaultCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/DefaultCoder.java
@@ -17,7 +17,6 @@
  */
 package org.apache.beam.sdk.coders;
 
-import com.google.auto.service.AutoService;
 import com.google.common.collect.ImmutableList;
 import java.lang.annotation.Documented;
 import java.lang.annotation.ElementType;
@@ -57,7 +56,6 @@ public @interface DefaultCoder {
* the {@code @DefaultCoder} annotation to provide {@link CoderProvider 
coder providers} that
* creates {@link Coder}s.
*/
-  @AutoService(CoderProviderRegistrar.class)
   class DefaultCoderProviderRegistrar implements CoderProviderRegistrar {
 
 @Override

http://git-wip-us.apache.org/repos/asf/beam/blob/809a7f86/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/SerializableCoder.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/SerializableCoder.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/SerializableCoder.java
index 6691876..9204942 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/SerializableCoder.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/SerializableCoder.java
@@ -17,7 +17,6 @@
  */
 package org.apache.beam.sdk.coders;
 
-import com.google.auto.service.AutoService;
 import com.google.common.collect.ImmutableList;
 import java.io.IOException;
 import java.io.InputStream;
@@ -80,7 +79,6 @@ public class SerializableCoder 
extends CustomCoder {
* A {@link CoderProviderRegistrar} which registers a {@link CoderProvider} 
which can handle
* serializable types.
*/
-  @AutoService(CoderProviderRegistrar.class)
   public static class SerializableCoderProviderRegistrar implements 
CoderProviderRegistrar {
 
 @Override

http://git-wip-us.apache.org/repos/asf/beam/blob/809a7f86/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/CoderRegistryTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/CoderRegistryTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/coder

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3653

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2708

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4457

[Apache Jenkins Server]

See 


--
[...truncated 273.89 KB...]
2017-07-26T12:02:00.124 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/asm/asm/3.3.1/asm-3.3.1.pom (266 B at 10.0 
KB/sec)
2017-07-26T12:02:00.126 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/asm/asm-parent/3.3.1/asm-parent-3.3.1.pom
2017-07-26T12:02:00.152 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/asm/asm-parent/3.3.1/asm-parent-3.3.1.pom 
(5 KB at 162.7 KB/sec)
2017-07-26T12:02:00.154 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-beanutils/commons-beanutils/1.8.3/commons-beanutils-1.8.3.pom
2017-07-26T12:02:00.181 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-beanutils/commons-beanutils/1.8.3/commons-beanutils-1.8.3.pom
 (11 KB at 405.0 KB/sec)
2017-07-26T12:02:00.182 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/14/commons-parent-14.pom
2017-07-26T12:02:00.210 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/14/commons-parent-14.pom
 (31 KB at 1094.8 KB/sec)
2017-07-26T12:02:00.214 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugin-tools/maven-plugin-annotations/3.5/maven-plugin-annotations-3.5.pom
2017-07-26T12:02:00.240 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugin-tools/maven-plugin-annotations/3.5/maven-plugin-annotations-3.5.pom
 (2 KB at 60.4 KB/sec)
2017-07-26T12:02:00.242 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugin-tools/maven-plugin-tools/3.5/maven-plugin-tools-3.5.pom
2017-07-26T12:02:00.270 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugin-tools/maven-plugin-tools/3.5/maven-plugin-tools-3.5.pom
 (16 KB at 564.7 KB/sec)
2017-07-26T12:02:00.275 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-resources/1.0.1/plexus-resources-1.0.1.pom
2017-07-26T12:02:00.305 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-resources/1.0.1/plexus-resources-1.0.1.pom
 (2 KB at 42.9 KB/sec)
2017-07-26T12:02:00.308 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.8/plexus-utils-3.0.8.pom
2017-07-26T12:02:00.335 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.8/plexus-utils-3.0.8.pom
 (4 KB at 113.6 KB/sec)
2017-07-26T12:02:00.341 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/20.0/guava-20.0.jar
2017-07-26T12:02:00.342 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/net/revelc/code/formatter/jdt-core/2.0.1/jdt-core-2.0.1.jar
2017-07-26T12:02:00.343 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/net/revelc/code/formatter/jsdt-core/2.0.1/jsdt-core-2.0.1.jar
2017-07-26T12:02:00.344 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-digester3/3.2/commons-digester3-3.2.jar
2017-07-26T12:02:00.344 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/cglib/cglib/2.2.2/cglib-2.2.2.jar
2017-07-26T12:02:00.437 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-digester3/3.2/commons-digester3-3.2.jar
 (236 KB at 2483.7 KB/sec)
2017-07-26T12:02:00.437 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/asm/asm/3.3.1/asm-3.3.1.jar
2017-07-26T12:02:00.439 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/cglib/cglib/2.2.2/cglib-2.2.2.jar (281 KB 
at 2952.2 KB/sec)
2017-07-26T12:02:00.439 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-beanutils/commons-beanutils/1.8.3/commons-beanutils-1.8.3.jar
2017-07-26T12:02:00.530 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/asm/asm/3.3.1/asm-3.3.1.jar (43 KB at 
228.8 KB/sec)
2017-07-26T12:02:00.530 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugin-tools/maven-plugin-annotations/3.5/maven-plugin-annotations-3.5.jar
2017-07-26T12:02:00.534 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-beanutils/commons-beanutils/1.8.3/commons-beanutils-1.8.3.jar
 (227 KB at 1192.5 KB/sec)
2017-07-26T12:02:00.534 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-resources/1.0.1/plexus-resources-1.0.1.jar
2017-07-26T12:02:00.562 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-resources/1.0.1/plexus-resources-1.0.1.jar
 (22 KB at 98.0 KB/sec)
2017-07-26T12:02:00.585 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugin-tools/maven-plugin-annotations/3.5/maven-plugin-annotations-3.5.jar
 (14 KB at 58.0 KB/sec)
2017-07-26T12:02:00.807 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/20.0/guava-20.0.jar 
(2386 KB at 5118.8 KB/sec)
2017-07-26T12:02:03.096

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3652

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #4456

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3651

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #4454

[Apache Jenkins Server]

See