[jira] [Commented] (BEAM-3011) Pin Runner harness container image in Python SDK

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198142#comment-16198142
 ] 

Valentyn Tymofieiev commented on BEAM-3011:
---

https://github.com/apache/beam/pull/3966 that fixes this is in review.

> Pin Runner harness container image in Python SDK
> 
>
> Key: BEAM-3011
> URL: https://issues.apache.org/jira/browse/BEAM-3011
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #4133

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3968: Only fail nightly release at end, for more detailed...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3968

Only fail nightly release at end, for more detailed signal

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @aviemzur (since you mentioned how broken things are, sending things to 
you)

I think the nightly build - misnamed, since it isn't night for everyone - 
should gather all the failures. It hit a flake in some IO and then we lost any 
other signal.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam nightly-fail-at-end

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3968.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3968


commit 15b0e47b1e35dbcfd1149379bad6e7170552ec0f
Author: Kenneth Knowles 
Date:   2017-10-10T03:16:24Z

Only fail nightly release at end, for more detailed signal




---

[jira] [Resolved] (BEAM-3037) Java DirectRunner jar includes non-relocated runner api protos

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-3037.
---
Resolution: Fixed

> Java DirectRunner jar includes non-relocated runner api protos
> --
>
> Key: BEAM-3037
> URL: https://issues.apache.org/jira/browse/BEAM-3037
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: 2.2.0
>
>
> The DirectRunner jar includes classes like 
> {{org.apache.beam.portability.v1.Endpoints}} that actually extend a shaded 
> proto library 
> {{org.apache.beam.runner.direct.repackaged.portability.v1.Endpoints}} so you 
> get very inscrutable messages.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3037) Java DirectRunner jar includes non-relocated runner api protos

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3037:
--
Fix Version/s: 2.2.0

> Java DirectRunner jar includes non-relocated runner api protos
> --
>
> Key: BEAM-3037
> URL: https://issues.apache.org/jira/browse/BEAM-3037
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: 2.2.0
>
>
> The DirectRunner jar includes classes like 
> {{org.apache.beam.portability.v1.Endpoints}} that actually extend a shaded 
> proto library 
> {{org.apache.beam.runner.direct.repackaged.portability.v1.Endpoints}} so you 
> get very inscrutable messages.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4132

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #3257

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #428

[Apache Jenkins Server]

See 


Changes:

[klk] Allow precommits to coexist with phrase triggering

[klk] Restore Python precommit

[klk] Enable phrase triggering of Java PreCommit

[klk] Enable phrase triggering of Go PreCommit

[klk] Add runners/direct-java to Java precommit

[kirpichov] Fixes TextIO and AvroIO tests of watchForNewFiles

[kirpichov] Add @FunctionalInterface annotation to several interfaces

[klk] Ensure metric names are not null or empty

[klk] Relocate sdks.common.v1 and portability.v1 in Java direct runner

[chamikara] Fixes a bug in query splitting.

[relax] [maven-release-plugin] prepare branch release-2.2.0

[relax] [maven-release-plugin] prepare for next development iteration

[relax] Bump Python version for next release.

--
[...truncated 1.21 MB...]
cc1: warning: command line option -std=c++11 is valid for C++/ObjC++ but 
not for C [enabled by default]
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 
-Wall -Wstrict-prototypes -fPIC -DOPENSSL_NO_ASM=1 -D_WIN32_WINNT=1536 
-DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 
-DPyMODINIT_FUNC=__attribute__((visibility ("default"))) void 
-Isrc/python/grpcio -Iinclude -I. -Ithird_party/boringssl/include 
-Ithird_party/zlib -Ithird_party/cares -Ithird_party/cares/cares 
-Ithird_party/cares/config_linux -I/usr/include/python2.7 -c 
third_party/boringssl/crypto/asn1/a_type.c -o 
python_build/temp.linux-x86_64-2.7/third_party/boringssl/crypto/asn1/a_type.o 
-std=c++11 -std=gnu99 -fvisibility=hidden -fno-wrapv -fno-exceptions -pthread
cc1: warning: command line option -std=c++11 is valid for C++/ObjC++ but 
not for C [enabled by default]
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 
-Wall -Wstrict-prototypes -fPIC -DOPENSSL_NO_ASM=1 -D_WIN32_WINNT=1536 
-DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 
-DPyMODINIT_FUNC=__attribute__((visibility ("default"))) void 
-Isrc/python/grpcio -Iinclude -I. -Ithird_party/boringssl/include 
-Ithird_party/zlib -Ithird_party/cares -Ithird_party/cares/cares 
-Ithird_party/cares/config_linux -I/usr/include/python2.7 -c 
third_party/boringssl/crypto/asn1/a_utctm.c -o 
python_build/temp.linux-x86_64-2.7/third_party/boringssl/crypto/asn1/a_utctm.o 
-std=c++11 -std=gnu99 -fvisibility=hidden -fno-wrapv -fno-exceptions -pthread
cc1: warning: command line option -std=c++11 is valid for C++/ObjC++ but 
not for C [enabled by default]
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 
-Wall -Wstrict-prototypes -fPIC -DOPENSSL_NO_ASM=1 -D_WIN32_WINNT=1536 
-DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 
-DPyMODINIT_FUNC=__attribute__((visibility ("default"))) void 
-Isrc/python/grpcio -Iinclude -I. -Ithird_party/boringssl/include 
-Ithird_party/zlib -Ithird_party/cares -Ithird_party/cares/cares 
-Ithird_party/cares/config_linux -I/usr/include/python2.7 -c 
third_party/boringssl/crypto/asn1/a_utf8.c -o 
python_build/temp.linux-x86_64-2.7/third_party/boringssl/crypto/asn1/a_utf8.o 
-std=c++11 -std=gnu99 -fvisibility=hidden -fno-wrapv -fno-exceptions -pthread
cc1: warning: command line option -std=c++11 is valid for C++/ObjC++ but 
not for C [enabled by default]
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 
-Wall -Wstrict-prototypes -fPIC -DOPENSSL_NO_ASM=1 -D_WIN32_WINNT=1536 
-DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 
-DPyMODINIT_FUNC=__attribute__((visibility ("default"))) void 
-Isrc/python/grpcio -Iinclude -I. -Ithird_party/boringssl/include 
-Ithird_party/zlib -Ithird_party/cares -Ithird_party/cares/cares 
-Ithird_party/cares/config_linux -I/usr/include/python2.7 -c 
third_party/boringssl/crypto/asn1/asn1_lib.c -o 
python_build/temp.linux-x86_64-2.7/third_party/boringssl/crypto/asn1/asn1_lib.o 
-std=c++11 -std=gnu99 -fvisibility=hidden -fno-wrapv -fno-exceptions -pthread
cc1: warning: command line option -std=c++11 is valid for C++/ObjC++ but 
not for C [enabled by default]
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 
-Wall -Wstrict-prototypes -fPIC -DOPENSSL_NO_ASM=1 -D_WIN32_WINNT=1536 
-DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 
-DPyMODINIT_FUNC=__attribute__((visibility ("default"))) void 
-Isrc/python/grpcio -Iinclude -I. -Ithird_party/boringssl/include 
-Ithird_party/zlib -Ithird_party/cares -Ithird_party/cares/cares 
-Ithird_party/cares/config_linux -I/usr/include/python2.7 -c 
third_party/boringssl/crypto/asn1/asn1_par.c -o 
python_build/temp.linux-x86_64-2.7/third_party/boringssl/crypto/asn1/asn1_par.o 
-std=c++11 -std=gnu99 -fvisibility=hidden -fno-wrapv -fno-exceptions -pthread
cc1: warning: command line option -std=c++11 is valid for C++/ObjC++ but 
not for C [enabled by default]
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 
-Wall 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4992

[Apache Jenkins Server]

See 


--
[...truncated 326.13 KB...]
2017-10-10T00:04:05.638 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/findbugs/3.0.1/findbugs-3.0.1.pom
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection reset
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection reset
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection reset
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
2017-10-10T00:04:05.708 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.9.4/ant-1.9.4.pom
2017-10-10T00:04:05.738 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.9.4/ant-1.9.4.pom (10 
KB at 313.3 KB/sec)
2017-10-10T00:04:05.741 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.9.4/ant-parent-1.9.4.pom
2017-10-10T00:04:05.775 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.9.4/ant-parent-1.9.4.pom
 (6 KB at 160.9 KB/sec)
2017-10-10T00:04:05.782 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-launcher/1.9.4/ant-launcher-1.9.4.pom
2017-10-10T00:04:05.813 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-launcher/1.9.4/ant-launcher-1.9.4.pom
 (3 KB at 73.6 KB/sec)
2017-10-10T00:04:05.815 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy/2.4.7/groovy-2.4.7.pom
2017-10-10T00:04:05.844 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy/2.4.7/groovy-2.4.7.pom
 (17 KB at 569.2 KB/sec)
2017-10-10T00:04:05.849 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-ant/2.4.7/groovy-ant-2.4.7.pom
2017-10-10T00:04:05.879 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-ant/2.4.7/groovy-ant-2.4.7.pom
 (17 KB at 538.0 KB/sec)
2017-10-10T00:04:05.882 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-antlr/1.9.4/ant-antlr-1.9.4.pom
2017-10-10T00:04:05.910 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-antlr/1.9.4/ant-antlr-1.9.4.pom
 (4 KB at 107.3 KB/sec)
2017-10-10T00:04:05.915 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-groovydoc/2.4.7/groovy-groovydoc-2.4.7.pom
2017-10-10T00:04:05.958 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-groovydoc/2.4.7/groovy-groovydoc-2.4.7.pom
 (16 KB at 356.0 KB/sec)
2017-10-10T00:04:05.961 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-templates/2.4.7/groovy-templates-2.4.7.pom
2017-10-10T00:04:05.993 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-templates/2.4.7/groovy-templates-2.4.7.pom
 (16 KB at 463.7 KB/sec)
2017-10-10T00:04:05.996 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-xml/2.4.7/groovy-xml-2.4.7.pom
2017-10-10T00:04:06.031 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-xml/2.4.7/groovy-xml-2.4.7.pom
 (16 KB at 444.7 KB/sec)
2017-10-10T00:04:06.034 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-junit/1.9.4/ant-junit-1.9.4.pom
2017-10-10T00:04:06.084 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-junit/1.9.4/ant-junit-1.9.4.pom
 (4 KB at 79.2 KB/sec)
2017-10-10T00:04:06.087 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-impl/2.1/maven-reporting-impl-2.1.pom
2017-10-10T00:04:06.120 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-impl/2.1/maven-reporting-impl-2.1.pom
 (5 KB at 133.9 KB/sec)
2017-10-10T00:04:06.123 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.0.10/maven-project-2.0.10.pom
2017-10-10T00:04:06.155 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.0.10/maven-project-2.0.10.pom
 (3 KB at 86.9 KB/sec)
2017-10-10T00:04:06.157 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/2.0.10/maven-2.0.10.pom
2017-10-10T00:04:06.190 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/2.0.10/maven-2.0.10.pom
 (24 KB at 718.6 KB/sec)
2017-10-10T00:04:06.197 [INFO] Downloading: 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #3320

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4990

[Apache Jenkins Server]

See 


Changes:

[relax] [maven-release-plugin] prepare for next development iteration

--
[...truncated 1.66 MB...]
2017-10-09T23:37:58.834 [INFO] 2017-10-09\T\23:37:58.828 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.6/maven-resources-plugin-2.6.jar
2017-10-09T23:37:59.218 [INFO] 2017-10-09\T\23:37:59.215 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.6/maven-resources-plugin-2.6.jar
 (29 KB at 74.3 KB/sec)
2017-10-09T23:37:59.586 [INFO] 2017-10-09\T\23:37:59.568 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/2.12.4/maven-surefire-plugin-2.12.4.pom
2017-10-09T23:37:59.623 [INFO] 2017-10-09\T\23:37:59.623 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/2.12.4/maven-surefire-plugin-2.12.4.pom
 (11 KB at 189.4 KB/sec)
2017-10-09T23:37:59.630 [INFO] 2017-10-09\T\23:37:59.629 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire/2.12.4/surefire-2.12.4.pom
2017-10-09T23:37:59.674 [INFO] 2017-10-09\T\23:37:59.672 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire/2.12.4/surefire-2.12.4.pom
 (14 KB at 292.8 KB/sec)
2017-10-09T23:37:59.714 [INFO] 2017-10-09\T\23:37:59.702 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/2.12.4/maven-surefire-plugin-2.12.4.jar
2017-10-09T23:37:59.762 [INFO] 2017-10-09\T\23:37:59.761 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/2.12.4/maven-surefire-plugin-2.12.4.jar
 (30 KB at 504.2 KB/sec)
2017-10-09T23:37:59.844 [INFO] 2017-10-09\T\23:37:59.842 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-jar-plugin/2.4/maven-jar-plugin-2.4.pom
2017-10-09T23:37:59.878 [INFO] 2017-10-09\T\23:37:59.874 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-jar-plugin/2.4/maven-jar-plugin-2.4.pom
 (6 KB at 178.0 KB/sec)
2017-10-09T23:37:59.878 [INFO] 2017-10-09\T\23:37:59.877 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-plugins/22/maven-plugins-22.pom
2017-10-09T23:37:59.918 [INFO] 2017-10-09\T\23:37:59.916 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-plugins/22/maven-plugins-22.pom
 (13 KB at 326.5 KB/sec)
2017-10-09T23:37:59.945 [INFO] 2017-10-09\T\23:37:59.943 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-jar-plugin/2.4/maven-jar-plugin-2.4.jar
2017-10-09T23:38:00.007 [INFO] 2017-10-09\T\23:38:00.003 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-jar-plugin/2.4/maven-jar-plugin-2.4.jar
 (34 KB at 553.7 KB/sec)
2017-10-09T23:38:00.498 [INFO] 2017-10-09\T\23:38:00.496 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.8.0/jackson-annotations-2.8.0.pom
2017-10-09T23:38:00.570 [INFO] 2017-10-09\T\23:38:00.568 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.8.0/jackson-annotations-2.8.0.pom
 (2 KB at 25.4 KB/sec)
2017-10-09T23:38:00.951 [INFO] 2017-10-09\T\23:38:00.948 [INFO] Downloading: 
http://repository.apache.org/snapshots/org/apache/beam/beam-runners-parent/2.3.0-SNAPSHOT/maven-metadata.xml
2017-10-09T23:38:01.078 [INFO] 2017-10-09\T\23:38:01.077 [INFO] Downloading: 
http://repository.apache.org/snapshots/org/apache/beam/beam-runners-parent/2.3.0-SNAPSHOT/beam-runners-parent-2.3.0-SNAPSHOT.pom
2017-10-09T23:38:01.305 [INFO] 2017-10-09\T\23:38:01.304 [INFO] 

2017-10-09T23:38:01.306 [INFO] 2017-10-09\T\23:38:01.304 [INFO] BUILD FAILURE
2017-10-09T23:38:01.307 [INFO] 2017-10-09\T\23:38:01.306 [INFO] 

2017-10-09T23:38:01.307 [INFO] 2017-10-09\T\23:38:01.307 [INFO] Total time: 
6.685 s
2017-10-09T23:38:01.308 [INFO] 2017-10-09\T\23:38:01.308 [INFO] Finished at: 
2017-10-09T23:38:01+00:00
2017-10-09T23:38:02.018 [INFO] 2017-10-09\T\23:38:02.008 [INFO] Final Memory: 
10M/286M
2017-10-09T23:38:02.018 [INFO] 2017-10-09\T\23:38:02.008 [INFO] 

2017-10-09T23:38:02.056 [INFO] 2017-10-09\T\23:38:02.023 [ERROR] Failed to 
execute goal on project basic: Could not resolve dependencies for project 
archetype.it:basic:jar:0.1: Failed to collect dependencies at 
org.apache.beam:beam-runners-direct-java:jar:2.3.0-SNAPSHOT: Failed to read 
artifact descriptor for 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4991

[Apache Jenkins Server]

See 


Changes:

[relax] Bump Python version for next release.

--
[...truncated 1.25 MB...]
2017-10-09T23:30:15.426 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-protocol/1.2.6/hbase-protocol-1.2.6.pom
 (9 KB at 245.3 KB/sec)
2017-10-09T23:30:15.435 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-annotations/1.2.6/hbase-annotations-1.2.6.pom
2017-10-09T23:30:15.472 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-annotations/1.2.6/hbase-annotations-1.2.6.pom
 (3 KB at 61.2 KB/sec)
2017-10-09T23:30:15.486 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-procedure/1.2.6/hbase-procedure-1.2.6.pom
2017-10-09T23:30:15.523 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-procedure/1.2.6/hbase-procedure-1.2.6.pom
 (6 KB at 149.1 KB/sec)
2017-10-09T23:30:15.545 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-client/1.2.6/hbase-client-1.2.6.pom
2017-10-09T23:30:15.578 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-client/1.2.6/hbase-client-1.2.6.pom
 (13 KB at 365.9 KB/sec)
2017-10-09T23:30:15.585 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jruby/jcodings/jcodings/1.0.8/jcodings-1.0.8.pom
2017-10-09T23:30:15.615 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/jruby/jcodings/jcodings/1.0.8/jcodings-1.0.8.pom
 (5 KB at 140.4 KB/sec)
2017-10-09T23:30:15.618 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jruby/joni/joni/2.1.2/joni-2.1.2.pom
2017-10-09T23:30:15.645 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/jruby/joni/joni/2.1.2/joni-2.1.2.pom 
(5 KB at 159.5 KB/sec)
2017-10-09T23:30:15.648 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/yammer/metrics/metrics-core/2.2.0/metrics-core-2.2.0.pom
2017-10-09T23:30:15.678 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/yammer/metrics/metrics-core/2.2.0/metrics-core-2.2.0.pom
 (2 KB at 40.8 KB/sec)
2017-10-09T23:30:15.679 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/yammer/metrics/metrics-parent/2.2.0/metrics-parent-2.2.0.pom
2017-10-09T23:30:15.711 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/yammer/metrics/metrics-parent/2.2.0/metrics-parent-2.2.0.pom
 (10 KB at 308.6 KB/sec)
2017-10-09T23:30:15.714 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-auth/2.5.1/hadoop-auth-2.5.1.pom
2017-10-09T23:30:15.742 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-auth/2.5.1/hadoop-auth-2.5.1.pom
 (7 KB at 227.7 KB/sec)
2017-10-09T23:30:15.744 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-project/2.5.1/hadoop-project-2.5.1.pom
2017-10-09T23:30:15.779 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-project/2.5.1/hadoop-project-2.5.1.pom
 (39 KB at 1087.3 KB/sec)
2017-10-09T23:30:15.782 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-main/2.5.1/hadoop-main-2.5.1.pom
2017-10-09T23:30:15.819 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-main/2.5.1/hadoop-main-2.5.1.pom
 (18 KB at 477.2 KB/sec)
2017-10-09T23:30:15.832 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-prefix-tree/1.2.6/hbase-prefix-tree-1.2.6.pom
2017-10-09T23:30:15.860 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-prefix-tree/1.2.6/hbase-prefix-tree-1.2.6.pom
 (7 KB at 247.6 KB/sec)
2017-10-09T23:30:15.867 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-hadoop-compat/1.2.6/hbase-hadoop-compat-1.2.6.pom
2017-10-09T23:30:15.900 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-hadoop-compat/1.2.6/hbase-hadoop-compat-1.2.6.pom
 (6 KB at 154.8 KB/sec)
2017-10-09T23:30:15.906 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-math/2.2/commons-math-2.2.pom
2017-10-09T23:30:15.936 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-math/2.2/commons-math-2.2.pom
 (11 KB at 363.9 KB/sec)
2017-10-09T23:30:15.938 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/18/commons-parent-18.pom
2017-10-09T23:30:15.971 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/18/commons-parent-18.pom
 (33 KB at 991.8 KB/sec)
2017-10-09T23:30:15.977 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-hadoop2-compat/1.2.6/hbase-hadoop2-compat-1.2.6.pom
2017-10-09T23:30:16.003 [INFO] Downloaded: 

[jira] [Commented] (BEAM-2899) Universal Local Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197852#comment-16197852
 ] 

ASF GitHub Bot commented on BEAM-2899:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3967

[BEAM-2899] Move the Job Server into a Submodule

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---
This permits separation of the Job Server and the Java Runner which
submits to it, while both being part of the 'ReferenceRunner'



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam reference_runner_module

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3967.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3967


commit ff8ff926c5c911c4bba78286aba2000439bda767
Author: Thomas Groh 
Date:   2017-10-09T23:13:28Z

Move the Job Server into a Submodule

This permits separation of the Job Server and the Java Runner which
submits to it, while both being part of the 'ReferenceRunner'




> Universal Local Runner
> --
>
> Key: BEAM-2899
> URL: https://issues.apache.org/jira/browse/BEAM-2899
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>  Labels: portability
>
> To make the portability effort tractable, we should implement a Universal 
> Local Runner (ULR) in Java that runs in a single server process plus docker 
> containers for the SDK harness containers. It would serve multiple purposes:
>   (1) A reference implementation for other runners. Ideally, any new feature 
> should be implemented in the ULR first.
>   (2) A fully-featured test runner for SDKs who participate in the 
> portability framework. It thus complements the direct runners.
>   (3) A test runner for user code that depends on or customizes the runtime 
> environment. For example, a DoFn that shells out has a dependency that may be 
> satisfied on the user's desktop (and thus works fine on the direct runner), 
> but perhaps not by the container harness image. The ULR allows for an easy 
> way to find out.
> The Java direct runner presumably has lots of pieces that can be reused.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3967: [BEAM-2899] Move the Job Server into a Submodule

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3967

[BEAM-2899] Move the Job Server into a Submodule

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---
This permits separation of the Job Server and the Java Runner which
submits to it, while both being part of the 'ReferenceRunner'



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam reference_runner_module

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3967.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3967


commit ff8ff926c5c911c4bba78286aba2000439bda767
Author: Thomas Groh 
Date:   2017-10-09T23:13:28Z

Move the Job Server into a Submodule

This permits separation of the Job Server and the Java Runner which
submits to it, while both being part of the 'ReferenceRunner'




---

[GitHub] beam pull request #3966: Pin runner harness minimal

[tvalentyn]

GitHub user tvalentyn opened a pull request:

https://github.com/apache/beam/pull/3966

Pin runner harness minimal

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tvalentyn/beam pin_runner_harness_minimal

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3966.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3966


commit 3db8f80857e041732c24de2f382a394c9f3ee599
Author: Valentyn Tymofieiev 
Date:   2017-10-09T22:07:56Z

Pin runner harness

commit 90dbb72e90e2f731b7490c6899c859f9c1e5cdae
Author: Valentyn Tymofieiev 
Date:   2017-10-09T22:55:00Z

Pin runner harness container image for Dataflow distributions of BEAM 
Python SDK




---

[jira] [Created] (BEAM-3038) Add support for Azure Data Lake Storage as a Apache Beam FileSystem

[Romit Girdhar (JIRA)]

Romit Girdhar created BEAM-3038:
---

 Summary: Add support for Azure Data Lake Storage as a Apache Beam 
FileSystem
 Key: BEAM-3038
 URL: https://issues.apache.org/jira/browse/BEAM-3038
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Reporter: Romit Girdhar
Assignee: Reuven Lax
Priority: Minor


This is for providing direct integration with Azure Data Lake Store as an 
Apache Beam File system.

There is already support for Azure Data Lake for using it as HDFS: 
https://hadoop.apache.org/docs/current/hadoop-azure-datalake/index.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3319

[Apache Jenkins Server]

See 


Changes:

[klk] Relocate sdks.common.v1 and portability.v1 in Java direct runner

--
[...truncated 927.00 KB...]
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s1"
}, 
"serialized_fn": "", 
"user_name": "months with tornadoes"
  }
}, 
{
  "kind": "GroupByKey", 
  "name": "s3", 
  "properties": {
"display_data": [], 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:stream", 
  "component_encodings": [
{
  "@type": 
"VarIntCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxhiUWeeSXOIA5XIYNmYyFjbSFTkh4A89cR+g==",
 
  "component_encodings": []
}
  ], 
  "is_stream_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "monthly count/GroupByKey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s2"
}, 
"serialized_fn": 
"%0AZ%22X%0A%1Dref_Coder_GlobalWindowCoder_1%127%0A5%0A3%0A1urn%3Abeam%3Acoders%3Aurn%3Abeam%3Acoders%3Aglobal_window%3A0.1jR%0A%25%0A%23%0A%21beam%3Awindowfn%3Aglobal_windows%3Av0.1%10%01%1A%1Dref_Coder_GlobalWindowCoder_1%22%02%3A%00%28%010%018%01",
 
"user_name": "monthly count/GroupByKey"
  }
}, 
{
  "kind": "CombineValues", 
  "name": "s4", 
  "properties": {
"display_data": [], 
"encoding": {
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": 

beam git commit: Update container version.

[reuvenlax]

Repository: beam
Updated Branches:
  refs/heads/release-2.2.0 bfa8136d7 -> 0a4774d88


Update container version.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0a4774d8
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0a4774d8
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0a4774d8

Branch: refs/heads/release-2.2.0
Commit: 0a4774d88aeed1bbf8bb7af60924d06902504ebd
Parents: bfa8136
Author: Reuven Lax 
Authored: Mon Oct 9 15:50:16 2017 -0700
Committer: Reuven Lax 
Committed: Mon Oct 9 15:50:16 2017 -0700

--
 runners/google-cloud-dataflow-java/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/0a4774d8/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index 36ccb5a..90ab9f6 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -33,7 +33,7 @@
   jar
 
   
-
beam-master-20170926
+beam-2.2.0
 
1
 
6
   

beam git commit: Bump Python version for next release.

[reuvenlax]

Repository: beam
Updated Branches:
  refs/heads/master 1d675b03f -> 9f69997ba


Bump Python version for next release.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9f69997b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9f69997b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9f69997b

Branch: refs/heads/master
Commit: 9f69997ba48c25bb969cf9b399931756cb643f65
Parents: 1d675b0
Author: Reuven Lax 
Authored: Mon Oct 9 15:43:54 2017 -0700
Committer: Reuven Lax 
Committed: Mon Oct 9 15:43:54 2017 -0700

--
 sdks/python/apache_beam/version.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/9f69997b/sdks/python/apache_beam/version.py
--
diff --git a/sdks/python/apache_beam/version.py 
b/sdks/python/apache_beam/version.py
index 8b0a430..b956661 100644
--- a/sdks/python/apache_beam/version.py
+++ b/sdks/python/apache_beam/version.py
@@ -18,4 +18,4 @@
 """Apache Beam SDK version information and utilities."""
 
 
-__version__ = '2.2.0.dev'
+__version__ = '2.3.0.dev'

beam git commit: Update python version.

[reuvenlax]

Repository: beam
Updated Branches:
  refs/heads/release-2.2.0 db0d47b03 -> bfa8136d7


Update python version.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/bfa8136d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/bfa8136d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/bfa8136d

Branch: refs/heads/release-2.2.0
Commit: bfa8136d73bace833c3235f7b71567f7a69e1d7a
Parents: db0d47b
Author: Reuven Lax 
Authored: Mon Oct 9 15:42:17 2017 -0700
Committer: Reuven Lax 
Committed: Mon Oct 9 15:42:17 2017 -0700

--
 sdks/python/apache_beam/version.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/bfa8136d/sdks/python/apache_beam/version.py
--
diff --git a/sdks/python/apache_beam/version.py 
b/sdks/python/apache_beam/version.py
index 8b0a430..e8eac9d 100644
--- a/sdks/python/apache_beam/version.py
+++ b/sdks/python/apache_beam/version.py
@@ -18,4 +18,4 @@
 """Apache Beam SDK version information and utilities."""
 
 
-__version__ = '2.2.0.dev'
+__version__ = '2.2.0'

[beam] Git Push Summary

[reuvenlax]

Repository: beam
Updated Branches:
  refs/heads/release-2.2.0 [created] db0d47b03

beam git commit: [maven-release-plugin] prepare for next development iteration

[reuvenlax]

Repository: beam
Updated Branches:
  refs/heads/master db0d47b03 -> 1d675b03f


[maven-release-plugin] prepare for next development iteration


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1d675b03
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1d675b03
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1d675b03

Branch: refs/heads/master
Commit: 1d675b03f0102849eba0ba2f5b8017eaa91f1c1b
Parents: db0d47b
Author: Reuven Lax 
Authored: Mon Oct 9 15:39:57 2017 -0700
Committer: Reuven Lax 
Committed: Mon Oct 9 15:39:57 2017 -0700

--
 examples/java/pom.xml| 2 +-
 examples/java8/pom.xml   | 2 +-
 examples/pom.xml | 2 +-
 pom.xml  | 4 ++--
 runners/apex/pom.xml | 2 +-
 runners/core-construction-java/pom.xml   | 2 +-
 runners/core-java/pom.xml| 2 +-
 runners/direct-java/pom.xml  | 2 +-
 runners/flink/pom.xml| 2 +-
 runners/gcp/gcemd/pom.xml| 2 +-
 runners/gcp/gcsproxy/pom.xml | 2 +-
 runners/gcp/pom.xml  | 2 +-
 runners/gearpump/pom.xml | 2 +-
 runners/google-cloud-dataflow-java/pom.xml   | 2 +-
 runners/local-artifact-service-java/pom.xml  | 2 +-
 runners/pom.xml  | 2 +-
 runners/reference/pom.xml| 2 +-
 runners/spark/pom.xml| 2 +-
 sdks/common/fn-api/pom.xml   | 2 +-
 sdks/common/pom.xml  | 2 +-
 sdks/common/runner-api/pom.xml   | 2 +-
 sdks/go/pom.xml  | 2 +-
 sdks/java/build-tools/pom.xml| 2 +-
 sdks/java/container/pom.xml  | 2 +-
 sdks/java/core/pom.xml   | 2 +-
 sdks/java/extensions/google-cloud-platform-core/pom.xml  | 2 +-
 sdks/java/extensions/jackson/pom.xml | 2 +-
 sdks/java/extensions/join-library/pom.xml| 2 +-
 sdks/java/extensions/pom.xml | 2 +-
 sdks/java/extensions/protobuf/pom.xml| 2 +-
 sdks/java/extensions/sorter/pom.xml  | 2 +-
 sdks/java/extensions/sql/pom.xml | 2 +-
 sdks/java/harness/pom.xml| 2 +-
 sdks/java/io/amqp/pom.xml| 2 +-
 sdks/java/io/cassandra/pom.xml   | 2 +-
 sdks/java/io/common/pom.xml  | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/pom.xml   | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/pom.xml   | 2 +-
 .../io/elasticsearch-tests/elasticsearch-tests-common/pom.xml| 2 +-
 sdks/java/io/elasticsearch-tests/pom.xml | 2 +-
 sdks/java/io/elasticsearch/pom.xml   | 2 +-
 sdks/java/io/google-cloud-platform/pom.xml   | 2 +-
 sdks/java/io/hadoop-common/pom.xml   | 2 +-
 sdks/java/io/hadoop-file-system/pom.xml  | 2 +-
 sdks/java/io/hadoop/input-format/pom.xml | 2 +-
 sdks/java/io/hadoop/jdk1.8-tests/pom.xml | 2 +-
 sdks/java/io/hadoop/pom.xml  | 2 +-
 sdks/java/io/hbase/pom.xml   | 2 +-
 sdks/java/io/hcatalog/pom.xml| 2 +-
 sdks/java/io/jdbc/pom.xml| 2 +-
 sdks/java/io/jms/pom.xml | 2 +-
 sdks/java/io/kafka/pom.xml   | 2 +-
 sdks/java/io/kinesis/pom.xml | 2 +-
 sdks/java/io/mongodb/pom.xml | 2 +-
 sdks/java/io/mqtt/pom.xml| 2 +-
 sdks/java/io/pom.xml | 2 +-
 sdks/java/io/solr/pom.xml| 2 +-
 sdks/java/io/tika/pom.xml   

[jira] [Commented] (BEAM-2600) Python SDK harness container

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197818#comment-16197818
 ] 

ASF GitHub Bot commented on BEAM-2600:
--

GitHub user herohde opened a pull request:

https://github.com/apache/beam/pull/3965

[BEAM-2600] Add minimal python SDK harness container

The container image does not support requirements files, main sessions, etc 
yet.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/herohde/beam containers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3965.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3965


commit 56da326f04020ee3bae197177d1f541dc35c6f2d
Author: Henning Rohde 
Date:   2017-10-09T22:28:25Z

[BEAM-2600] Add minimal python SDK harness container




> Python SDK harness container
> 
>
> Key: BEAM-2600
> URL: https://issues.apache.org/jira/browse/BEAM-2600
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: Henning Rohde
>  Labels: portability
>
> In order to build a pipeline that invokes a Python UDF, we need to be able to 
> construct something like this:
> {code}
> SdkFunctionSpec {
>   environment = ,
>   spec = {
> urn = ,
> data = 
>   }
> }
> {code}
> I could be out of date, but based on a couple of conversations I do not know 
> that there exists anything we can put for "" today. For 
> prototyping, it could be just a symbol that runners have to know. But 
> eventually it should be something that runners can instantiate without 
> knowing anything about the SDK that put it there. I imagine it may encompass 
> "custom containers" eventually, though that doesn't block anything 
> immediately.
> Update 9/11/2017: this issue will track the first portable SDK harness as per 
> https://s.apache.org/beam-fn-api-container-contract, incl shared boot code 
> and maven integration to build Go and Docker container images.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

beam git commit: [maven-release-plugin] prepare branch release-2.2.0

[reuvenlax]

Repository: beam
Updated Branches:
  refs/heads/master f446853ee -> db0d47b03


[maven-release-plugin] prepare branch release-2.2.0


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/db0d47b0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/db0d47b0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/db0d47b0

Branch: refs/heads/master
Commit: db0d47b03f0148de8ab647e05df8fbacf00039ea
Parents: f446853
Author: Reuven Lax 
Authored: Mon Oct 9 15:39:18 2017 -0700
Committer: Reuven Lax 
Committed: Mon Oct 9 15:39:18 2017 -0700

--
 pom.xml| 2 +-
 runners/gearpump/pom.xml   | 3 +--
 sdks/java/extensions/sql/pom.xml   | 4 +---
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/pom.xml | 4 +---
 .../io/elasticsearch-tests/elasticsearch-tests-common/pom.xml  | 4 +---
 sdks/java/io/elasticsearch/pom.xml | 4 +---
 sdks/java/io/pom.xml   | 6 +++---
 sdks/java/io/solr/pom.xml  | 4 +---
 sdks/java/io/tika/pom.xml  | 3 +--
 sdks/java/nexmark/pom.xml  | 4 +---
 10 files changed, 12 insertions(+), 26 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/db0d47b0/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 42671e8..c313ce6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -48,7 +48,7 @@
 
scm:git:https://git-wip-us.apache.org/repos/asf/beam.git
 
scm:git:https://git-wip-us.apache.org/repos/asf/beam.git
 https://git-wip-us.apache.org/repos/asf?p=beam.git;a=summary
-HEAD
+release-2.2.0
   
 
   

http://git-wip-us.apache.org/repos/asf/beam/blob/db0d47b0/runners/gearpump/pom.xml
--
diff --git a/runners/gearpump/pom.xml b/runners/gearpump/pom.xml
index 2b460e7..8a473ad 100644
--- a/runners/gearpump/pom.xml
+++ b/runners/gearpump/pom.xml
@@ -15,8 +15,7 @@
 specific language governing permissions and limitations
 under the License.
 -->
-http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;
- xmlns="http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
 
   4.0.0
 

http://git-wip-us.apache.org/repos/asf/beam/blob/db0d47b0/sdks/java/extensions/sql/pom.xml
--
diff --git a/sdks/java/extensions/sql/pom.xml b/sdks/java/extensions/sql/pom.xml
index 0c105ba..df1c237 100644
--- a/sdks/java/extensions/sql/pom.xml
+++ b/sdks/java/extensions/sql/pom.xml
@@ -15,9 +15,7 @@
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;
-  xmlns="http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
 
   4.0.0
 

http://git-wip-us.apache.org/repos/asf/beam/blob/db0d47b0/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/pom.xml
--
diff --git a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/pom.xml 
b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/pom.xml
index 48d75ce..55006d1 100644
--- a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/pom.xml
+++ b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/pom.xml
@@ -15,9 +15,7 @@
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-http://maven.apache.org/POM/4.0.0;
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
 
 4.0.0
 

http://git-wip-us.apache.org/repos/asf/beam/blob/db0d47b0/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/pom.xml

[GitHub] beam pull request #3965: [BEAM-2600] Add minimal python SDK harness containe...

[herohde]

GitHub user herohde opened a pull request:

https://github.com/apache/beam/pull/3965

[BEAM-2600] Add minimal python SDK harness container

The container image does not support requirements files, main sessions, etc 
yet.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/herohde/beam containers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3965.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3965


commit 56da326f04020ee3bae197177d1f541dc35c6f2d
Author: Henning Rohde 
Date:   2017-10-09T22:28:25Z

[BEAM-2600] Add minimal python SDK harness container




---

[jira] [Commented] (BEAM-3037) Java DirectRunner jar includes non-relocated runner api protos

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197800#comment-16197800
 ] 

ASF GitHub Bot commented on BEAM-3037:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3964


> Java DirectRunner jar includes non-relocated runner api protos
> --
>
> Key: BEAM-3037
> URL: https://issues.apache.org/jira/browse/BEAM-3037
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> The DirectRunner jar includes classes like 
> {{org.apache.beam.portability.v1.Endpoints}} that actually extend a shaded 
> proto library 
> {{org.apache.beam.runner.direct.repackaged.portability.v1.Endpoints}} so you 
> get very inscrutable messages.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3964: [BEAM-3037] Relocate sdks.common.v1 and portability...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3964


---

[2/2] beam git commit: This closes #3964: [BEAM-3037] Relocate sdks.common.v1 and portability.v1 in Java direct runner

[kenn]

This closes #3964: [BEAM-3037] Relocate sdks.common.v1 and portability.v1 in 
Java direct runner


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f446853e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f446853e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f446853e

Branch: refs/heads/master
Commit: f446853eefc05e961ad5383400bba40dc7b083c1
Parents: 75bab74 1b59e4d
Author: Kenneth Knowles 
Authored: Mon Oct 9 15:21:31 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 9 15:21:31 2017 -0700

--
 runners/direct-java/pom.xml | 14 ++
 1 file changed, 14 insertions(+)
--

[1/2] beam git commit: Relocate sdks.common.v1 and portability.v1 in Java direct runner

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 75bab74eb -> f446853ee


Relocate sdks.common.v1 and portability.v1 in Java direct runner

Currently these two namespaces are mistakenly left as-is, even though they link
to other relocated proto libraries that make them incompatible when used
outside the direct runner.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1b59e4d4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1b59e4d4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1b59e4d4

Branch: refs/heads/master
Commit: 1b59e4d443a346e10a6bbb03492f6b83d207cf85
Parents: eac561f
Author: Kenneth Knowles 
Authored: Mon Oct 9 13:06:36 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 9 13:06:36 2017 -0700

--
 runners/direct-java/pom.xml | 14 ++
 1 file changed, 14 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/1b59e4d4/runners/direct-java/pom.xml
--
diff --git a/runners/direct-java/pom.xml b/runners/direct-java/pom.xml
index e14e813..4a094f8 100644
--- a/runners/direct-java/pom.xml
+++ b/runners/direct-java/pom.xml
@@ -88,6 +88,20 @@
 
   
   
+
+org.apache.beam.sdks.common
+
+  org.apache.beam.runners.direct.repackaged.sdks.common
+
+  
+  
+
+org.apache.beam.portability
+
+  org.apache.beam.runners.direct.repackaged.portability
+
+  
+  
 com.google.common
 
   

[jira] [Resolved] (BEAM-3028) DatastoreIO integration tests failing on Dataflow

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath resolved BEAM-3028.
--
Resolution: Fixed

> DatastoreIO integration tests failing on Dataflow
> -
>
> Key: BEAM-3028
> URL: https://issues.apache.org/jira/browse/BEAM-3028
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.2.0
>Reporter: Kenneth Knowles
>Assignee: Chamikara Jayalath
>Priority: Blocker
> Fix For: 2.2.0
>
>
> Our testing signal has been bad enough that we seemed to have ignored or 
> missed this when it happened.
> From these builds: (while the logs last, you can check them)
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/4963/org.apache.beam$beam-runners-google-cloud-dataflow-java/testReport/junit/org.apache.beam.sdk.io.gcp.datastore/V1ReadIT/testE2EV1Read/
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/4963/org.apache.beam$beam-runners-google-cloud-dataflow-java/testReport/junit/org.apache.beam.sdk.io.gcp.datastore/V1ReadIT/testE2EV1ReadWithGQLQueryWithNoLimit/
> Quoting:
> {code}
> java.lang.AssertionError: Combine.globally(Count)/Flatten.PCollections.out: 
> Expected: <1000L>
>  but: was <12000L>
>  at 
> org.apache.beam.sdk.testing.PAssert$PAssertionSite.capture(PAssert.java:165)
>  at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:397)
>  at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:387)
>  at 
> org.apache.beam.sdk.io.gcp.datastore.V1ReadIT.testE2EV1Read(V1ReadIT.java:95)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3028) DatastoreIO integration tests failing on Dataflow

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197796#comment-16197796
 ] 

Chamikara Jayalath commented on BEAM-3028:
--

Fix is in. Seems like post commits are failing currently for unrelated reasons. 
But this test passed for me locally with the fix. So closing this JIRA.

> DatastoreIO integration tests failing on Dataflow
> -
>
> Key: BEAM-3028
> URL: https://issues.apache.org/jira/browse/BEAM-3028
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.2.0
>Reporter: Kenneth Knowles
>Assignee: Chamikara Jayalath
>Priority: Blocker
> Fix For: 2.2.0
>
>
> Our testing signal has been bad enough that we seemed to have ignored or 
> missed this when it happened.
> From these builds: (while the logs last, you can check them)
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/4963/org.apache.beam$beam-runners-google-cloud-dataflow-java/testReport/junit/org.apache.beam.sdk.io.gcp.datastore/V1ReadIT/testE2EV1Read/
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/4963/org.apache.beam$beam-runners-google-cloud-dataflow-java/testReport/junit/org.apache.beam.sdk.io.gcp.datastore/V1ReadIT/testE2EV1ReadWithGQLQueryWithNoLimit/
> Quoting:
> {code}
> java.lang.AssertionError: Combine.globally(Count)/Flatten.PCollections.out: 
> Expected: <1000L>
>  but: was <12000L>
>  at 
> org.apache.beam.sdk.testing.PAssert$PAssertionSite.capture(PAssert.java:165)
>  at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:397)
>  at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:387)
>  at 
> org.apache.beam.sdk.io.gcp.datastore.V1ReadIT.testE2EV1Read(V1ReadIT.java:95)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

svn commit: r22188 - /dev/beam/KEYS

[kenn]

Author: kenn
Date: Mon Oct  9 22:05:43 2017
New Revision: 22188

Log:
Add public key for Reuven Lax

Modified:
dev/beam/KEYS

Modified: dev/beam/KEYS
==
--- dev/beam/KEYS (original)
+++ dev/beam/KEYS Mon Oct  9 22:05:43 2017
@@ -311,3 +311,58 @@ gNWtoMhOK+ZfVj/buLwl9yzFpF0fzpNm4yiTC3RQ
 Tk/FVT3ZFFSU1UigaXZY
 =/AhW
 -END PGP PUBLIC KEY BLOCK-
+pub   4096R/A3714FE8 2017-08-09
+uid  reuvenlax 
+sig 3A3714FE8 2017-08-09  reuvenlax 
+sub   4096R/994E3F61 2017-08-09
+sig  A3714FE8 2017-08-09  reuvenlax 
+-BEGIN PGP PUBLIC KEY BLOCK-
+
+mQINBFmKVxQBEADBvYMZBEZFty9k4+PrnCl01aycebfs6Ni7Ciqr54Fk4Lblx9lt
+a49QJtM7/iky49b17cl3YebJFLUxEDu7Vh8th7kKPbfEnKmtxh4k4LfEJ1z2QYbv
+d8e7DucHZRdG+4+mpPNdKAg6V3ctOo/rjjBrZOoSj1E8t1AglZnthueZkqTgaJca
+fyN4mljuA33q5ObD650vKftrlzZLO4icfMh1Ku3Ynq/7jZ5RI42kdMw459xO9VYn
+SSP7QbDA9io9+JiDeJYrqT0CIsC2z50+1phcCLKvQIoN8QT7+m7FIgWV2pZzYXjb
+4+u1IqOogDeqloPGfh74myUQMsIcRWr1uQup7L9es+SgFTRbMRK1LYur1QcoKW+b
+HlNRHP7VFEtsLF7iR7abvI06XCZk3Bh/CnfHjZCF0YSXNdXNBmd0Oh3a0yNTx2l6
+48KsN2IYQ6/eU7jWmDt8A4GabEwJr3PNtnJWjrnAPANAYP6JrqzP6roCKqVgxJlO
+eh4uLiOdetAu/QpWwhrAXplXOdXQLfmjxpKso9cFpaK8I63qIsSWQM5QKyzcRSPx
+HVPFfC6e4PesAxuvT9/CXYH122VXaaVoW2BdMQ7vtUjtKWSuPozXbyDfHU/VVBKF
+ky4broxVcovUg8I1ip3Rn41My1eX1hu+gB33maIXTZej0BwS4iXGAXkL4QARAQAB
+tCByZXV2ZW5sYXggPHJsYXhAYWx1bW5pLnJpY2UuZWR1PokCOAQTAQIAIgUCWYpX
+FAIbAwYLCQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQAkB/SqNxT+j0vQ/+Jzxm
+bCecSUI+smvBvOPeu0qyBvx05AwwmCl+kfi8wdr3WIApYB0cOiKRlKSwqbaeOcvj
+bz0RwrOg+u9WWlG6S0ardu25jTEVf6GmOA1HWPSrXrHLPP0qJFythNE07R9KbjAA
+oO4VOghXPf782ZCWmdXd2cw9dfa3JIdYCI5CKrukjohtnT/pZcNS6FjrlCrg1Rbs
+5310K1q5w9HixnaVwaa3u251MsNqVNwSIIJZObC0KZQDC0ILDg054kngC9xx40KY
+QgaHDPbYXxFrNspcCTuXjhMT26Z0Z17G2CHDHISau3ctcLXvE8GJYOFCFRO/TOWe
+PdVgBbrWBlmxpYPCUsp7OYO7qS6FTeBGoApY8zRWcHgKpwxf/FmaOSFoTXAQDOM7
+YsO87mekhqMJC+ZFk6QF87JIE1mel4mNE6zUU9pvTMbyagKHktLfxw1AOJTp+f4h
+qmXHQYKNbaE5Q+7PRIIs8irNtYNYzxLKcVjJFVNbH5gIiZV/FMnfwSyqGNrUmFYo
+c0XDdwMFHa/7CnaedbQCxYUWs7ocWt5gnECvMMVXuS2SuGkeNl5w687K64RoS12f
+QSsy+eE4R3p4spcDkXHDNPD0YNk8MVvf7JRzyLUwEmgQj0Jh5tJDHYrF1ejCimC7
+UKP1Y1e/8kIpZXVrCyObnvNpP7z1Ugw0brNus5S5Ag0EWYpXFAEQAN4GbJapptNh
+nKSWn97/QP5i+eVGL3F9yzO4UUYjZuM2c3R2NPca8OQSGGy9VDsRndps6AuerzHW
+LKqoTs0thg9KOHOQ3IJ80J6b56zbNnEAAzZ2Q4uOzK1JTKfp7dtg5ihH6QD4hwTq
+QtKmsmXVLFfhFeO1h+hCuosZlGwECfnx+kfJzgm/VbkRVeapsRTQRd0eIN6OfT+L
+VRs6LS0PiS2nrbRp0brd2DJY+DLDTGk21heL8XaPadEQc6dpXjsH+b8prT9IApKL
+jqTngfVq5oJXyem0RP2WgK606kvez3z+fwVJQ9DIWXxQIpvuX0paDl8oJG1EpKNJ
+iH7R5TMTFtYjG6kwxTyAu7wPmZ8q7UfsyLOKl+6C5wWT64+eWh3GASCXu8BNTlsO
+Qqr0nsx/UJ/U1m59qh4gFsTx/TBbVgVF9VOTTt361kbQYjms3YuVGhANt0/puBS1
+Xp4rTmVjCDqe4B5CX6JYuvZYaoIXj7cnhdsblLSY8WlocTrIL+SdzCTQF2LA2pRu
+SlXoHDbGbtdh6zhgCYU8PJCSGFREl/BoOlAf/xluJjtcnkPY/o2d4jx14n3tNyGw
+Oy1HB5rCQO4CnsWR3LPF2Yexn9eHgOb9ShfuBMVnBdatYMdYFb7AUU34bhf7+Oyx
+WOZ7WdBFzbcPlmC+K+MikTsA3hpDrzPJABEBAAGJAh8EGAECAAkFAlmKVxQCGwwA
+CgkQAkB/SqNxT+gLEA//RXC31cv5vzoGhHFg4w/KgH3/zvYgzHc+ui5PkS/htiBp
+pEO29xVl6msudtNqs4XYqpXjHDRCCWRdd46UkJAq6zrZDmlIGoacvTLBq8uKIk2g
+IpxvEVys2QIOohEGTxty0t3/LnrXyfRyrKlWGndRoQasE5+fF2CpzCZvS5aojKG8
+l+3zivZQlslqDQilOwqsk1AePkQJH1RRFe39KhF0ZGTPM2Wy7GQ0tWr/xoGokhu7
+kLIYki2RVQokqS0wdrxsBOg/8ld40Qb6kNVTAW5tH2HklEvcL/8FQyRLygN6DbBm
+RoCgyfd339EpGPOpax5R/N6wiB1usu/YC0qPCuA64F1WMx8Z65nQaqF+mfGpTcfR
+uq8ZuhIaCrK9/y7yyXDt1GcoZaX9YXzQnlS+wmMZ2Kf5y+UiSgmydBOCdG1Ke0EI
+2rYMwQARijbG3/KGQYmd/UNLxXXvb+AegFXKL6hAZjtgBzid8fxps+J0Ius+6dK9
+gEIPAdQ3z07y0W19gj0CJQzGv9929vBLNSsvNsNOCkSEzyGczBCuiIJa5nHtuhIx
+ihzdeN4MN35i3Gtxz4vGxvDNpADwSZMTmFT5wW5BsOMbL4kfz+WE/kX6X07cn6iM
+WIp7YH3KH93zhbhpTgq0vvKT/U6DivZbwJzG6pMpRzOUtpfdx6I7+GhBAc3YKZU=
+=nw7v
+-END PGP PUBLIC KEY BLOCK-

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #4986

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #4130

[Apache Jenkins Server]

See 


Changes:

[klk] Fix typo in nightly release script

[kirpichov] Fixes TextIO and AvroIO tests of watchForNewFiles

[kirpichov] Add @FunctionalInterface annotation to several interfaces

--
[...truncated 21.06 MB...]
[INFO] 2017-10-09T21:46:36.557Z: (81796e4cf2df7253): Fusing unzipped copy of 
PAssert$327/GroupGlobally/GroupDummyAndContents/Reify, through flatten , into 
producer PAssert$327/GroupGlobally/KeyForDummy/AddKeys/Map
[INFO] 2017-10-09T21:46:36.560Z: (81796e4cf2df7589): Fusing consumer 
PAssert$327/GroupGlobally/GroupDummyAndContents/GroupByWindow into 
PAssert$327/GroupGlobally/GroupDummyAndContents/Read
[INFO] 2017-10-09T21:46:36.562Z: (81796e4cf2df78bf): Unzipping flatten s37-u117 
for input s39-reify-value59-c115
[INFO] 2017-10-09T21:46:36.564Z: (81796e4cf2df7bf5): Fusing unzipped copy of 
PAssert$325/GroupGlobally/GroupDummyAndContents/Write, through flatten , into 
producer PAssert$325/GroupGlobally/GroupDummyAndContents/Reify
[INFO] 2017-10-09T21:46:36.566Z: (81796e4cf2df7f2b): Unzipping flatten s77-u122 
for input s79-reify-value104-c120
[INFO] 2017-10-09T21:46:36.569Z: (81796e4cf2df7261): Fusing unzipped copy of 
PAssert$327/GroupGlobally/GroupDummyAndContents/Write, through flatten , into 
producer PAssert$327/GroupGlobally/GroupDummyAndContents/Reify
[INFO] 2017-10-09T21:46:36.571Z: (81796e4cf2df7597): Unzipping flatten s57 for 
input s52.52
[INFO] 2017-10-09T21:46:36.574Z: (81796e4cf2df78cd): Fusing unzipped copy of 
PAssert$326/GroupGlobally/GroupDummyAndContents/Reify, through flatten , into 
producer PAssert$326/GroupGlobally/KeyForDummy/AddKeys/Map
[INFO] 2017-10-09T21:46:36.576Z: (81796e4cf2df7c03): Fusing consumer 
PAssert$326/GetPane/Map into PAssert$326/GroupGlobally/ParDo(Concat)
[INFO] 2017-10-09T21:46:36.579Z: (81796e4cf2df7f39): Fusing consumer 
PAssert$326/RunChecks into PAssert$326/GetPane/Map
[INFO] 2017-10-09T21:46:36.582Z: (81796e4cf2df726f): Unzipping flatten s57-u131 
for input s59-reify-value77-c129
[INFO] 2017-10-09T21:46:36.585Z: (81796e4cf2df75a5): Fusing unzipped copy of 
PAssert$326/GroupGlobally/GroupDummyAndContents/Write, through flatten , into 
producer PAssert$326/GroupGlobally/GroupDummyAndContents/Reify
[INFO] 2017-10-09T21:46:36.587Z: (81796e4cf2df78db): Fusing consumer 
PAssert$326/VerifyAssertions/ParDo(DefaultConclude) into PAssert$326/RunChecks
[INFO] 2017-10-09T21:46:36.590Z: (81796e4cf2df7c11): Fusing consumer 
PAssert$326/GroupGlobally/GroupDummyAndContents/GroupByWindow into 
PAssert$326/GroupGlobally/GroupDummyAndContents/Read
[INFO] 2017-10-09T21:46:36.593Z: (81796e4cf2df7f47): Fusing consumer 
PAssert$326/GroupGlobally/Values/Values/Map into 
PAssert$326/GroupGlobally/GroupDummyAndContents/GroupByWindow
[INFO] 2017-10-09T21:46:36.596Z: (81796e4cf2df727d): Fusing consumer 
PAssert$326/GroupGlobally/ParDo(Concat) into 
PAssert$326/GroupGlobally/Values/Values/Map
[INFO] 2017-10-09T21:46:36.598Z: (81796e4cf2df75b3): Fusing consumer 
Sum/Combine.perKey(SumInts)/GroupByKey/Write into 
Sum/Combine.perKey(SumInts)/GroupByKey/Reify
[INFO] 2017-10-09T21:46:36.600Z: (81796e4cf2df78e9): Fusing consumer 
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into 
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
[INFO] 2017-10-09T21:46:36.604Z: (81796e4cf2df7c1f): Fusing consumer 
Combine.globally(TestCombineFnWithContext)/Combine.perKey(TestCombineFnWithContext)/Combine.GroupedValues
 into 
Combine.globally(TestCombineFnWithContext)/Combine.perKey(TestCombineFnWithContext)/GroupByKey/GroupByWindow
[INFO] 2017-10-09T21:46:36.606Z: (81796e4cf2df7f55): Fusing consumer 
PAssert$326/GroupGlobally/KeyForDummy/AddKeys/Map into 
PAssert$326/GroupGlobally/RewindowActuals/Window.Assign
[INFO] 2017-10-09T21:46:36.609Z: (81796e4cf2df728b): Fusing consumer 
PAssert$326/GroupGlobally/RewindowActuals/Window.Assign into 
PAssert$326/GroupGlobally/GatherAllOutputs/Values/Values/Map
[INFO] 2017-10-09T21:46:36.611Z: (81796e4cf2df75c1): Fusing consumer 
PAssert$327/GroupGlobally/GatherAllOutputs/Values/Values/Map into 
PAssert$327/GroupGlobally/GatherAllOutputs/GroupByKey/GroupByWindow
[INFO] 2017-10-09T21:46:36.614Z: (81796e4cf2df78f7): Fusing consumer 
PAssert$327/GroupGlobally/GatherAllOutputs/GroupByKey/GroupByWindow into 
PAssert$327/GroupGlobally/GatherAllOutputs/GroupByKey/Read
[INFO] 2017-10-09T21:46:36.616Z: (81796e4cf2df7c2d): Fusing consumer 
PAssert$327/GroupGlobally/GatherAllOutputs/GroupByKey/Write into 
PAssert$327/GroupGlobally/GatherAllOutputs/GroupByKey/Reify
[INFO] 2017-10-09T21:46:36.618Z: (81796e4cf2df7f63): Fusing consumer 
PAssert$327/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map into 

[jira] [Commented] (BEAM-3028) DatastoreIO integration tests failing on Dataflow

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197693#comment-16197693
 ] 

ASF GitHub Bot commented on BEAM-3028:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3962


> DatastoreIO integration tests failing on Dataflow
> -
>
> Key: BEAM-3028
> URL: https://issues.apache.org/jira/browse/BEAM-3028
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.2.0
>Reporter: Kenneth Knowles
>Assignee: Chamikara Jayalath
>Priority: Blocker
> Fix For: 2.2.0
>
>
> Our testing signal has been bad enough that we seemed to have ignored or 
> missed this when it happened.
> From these builds: (while the logs last, you can check them)
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/4963/org.apache.beam$beam-runners-google-cloud-dataflow-java/testReport/junit/org.apache.beam.sdk.io.gcp.datastore/V1ReadIT/testE2EV1Read/
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/4963/org.apache.beam$beam-runners-google-cloud-dataflow-java/testReport/junit/org.apache.beam.sdk.io.gcp.datastore/V1ReadIT/testE2EV1ReadWithGQLQueryWithNoLimit/
> Quoting:
> {code}
> java.lang.AssertionError: Combine.globally(Count)/Flatten.PCollections.out: 
> Expected: <1000L>
>  but: was <12000L>
>  at 
> org.apache.beam.sdk.testing.PAssert$PAssertionSite.capture(PAssert.java:165)
>  at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:397)
>  at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:387)
>  at 
> org.apache.beam.sdk.io.gcp.datastore.V1ReadIT.testE2EV1Read(V1ReadIT.java:95)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3962: [BEAM-3028] Fixes a bug in DatastoreIO query splitt...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3962


---

[2/2] beam git commit: This closes #3962

[chamikara]

This closes #3962


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/75bab74e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/75bab74e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/75bab74e

Branch: refs/heads/master
Commit: 75bab74ebc9ed01303646f4c9e3fde192112c032
Parents: 296f7fa 999e957
Author: Chamikara Jayalath 
Authored: Mon Oct 9 14:01:30 2017 -0700
Committer: Chamikara Jayalath 
Committed: Mon Oct 9 14:01:30 2017 -0700

--
 .../beam/sdk/io/gcp/datastore/DatastoreV1.java   |  2 +-
 .../beam/sdk/io/gcp/datastore/DatastoreV1Test.java   | 15 ++-
 2 files changed, 15 insertions(+), 2 deletions(-)
--

[1/2] beam git commit: Fixes a bug in query splitting.

[chamikara]

Repository: beam
Updated Branches:
  refs/heads/master 296f7fa96 -> 75bab74eb


Fixes a bug in query splitting.

We were returning original query instead of the sub-queries resulting in data 
duplication when reading.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/999e957d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/999e957d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/999e957d

Branch: refs/heads/master
Commit: 999e9579bf826e2df912719d2f0e3ad8f5c5
Parents: 296f7fa
Author: chamik...@google.com 
Authored: Sun Oct 8 17:02:43 2017 -0700
Committer: Chamikara Jayalath 
Committed: Mon Oct 9 14:01:16 2017 -0700

--
 .../beam/sdk/io/gcp/datastore/DatastoreV1.java   |  2 +-
 .../beam/sdk/io/gcp/datastore/DatastoreV1Test.java   | 15 ++-
 2 files changed, 15 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/999e957d/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
index eb0c26f..9b20c0d 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
@@ -807,7 +807,7 @@ public class DatastoreV1 {
 
 // assign unique keys to query splits.
 for (Query subquery : querySplits) {
-  c.output(query);
+  c.output(subquery);
 }
   }
 

http://git-wip-us.apache.org/repos/asf/beam/blob/999e957d/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
index 58bab21..550b6b9 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
@@ -40,6 +40,7 @@ import static org.hamcrest.Matchers.hasItem;
 import static org.hamcrest.Matchers.lessThanOrEqualTo;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotEquals;
 import static org.junit.Assert.assertThat;
 import static org.junit.Assert.assertTrue;
 import static org.mockito.Matchers.any;
@@ -622,6 +623,12 @@ public class DatastoreV1Test {
 List queries = doFnTester.processBundle(QUERY);
 
 assertEquals(queries.size(), numSplits);
+
+// Confirms that sub-queries are not equal to original when there is more 
than one split.
+for (Query subQuery : queries) {
+  assertNotEquals(subQuery, QUERY);
+}
+
 verify(mockQuerySplitter, times(1)).getSplits(
 eq(QUERY), any(PartitionId.class), eq(numSplits), 
any(Datastore.class));
 verifyZeroInteractions(mockDatastore);
@@ -991,8 +998,14 @@ public class DatastoreV1Test {
   /** Generate dummy query splits. */
   private List splitQuery(Query query, int numSplits) {
 List queries = new LinkedList<>();
+int offsetOfOriginal = query.getOffset();
 for (int i = 0; i < numSplits; i++) {
-  queries.add(query.toBuilder().build());
+  Query.Builder q = Query.newBuilder();
+  q.addKindBuilder().setName(KIND);
+  // Making sub-queries unique (and not equal to the original query) by 
setting different
+  // offsets.
+  q.setOffset(++offsetOfOriginal);
+  queries.add(q.build());
 }
 return queries;
   }

[GitHub] beam pull request #3956: Restore Python precommit; enable phrase triggering ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3956


---

[6/6] beam git commit: This closes #3956: Restore Python precommit; enable phrase triggering for precommits in general

[kenn]

This closes #3956: Restore Python precommit; enable phrase triggering for 
precommits in general

  Add runners/direct-java to Java precommit
  Enable phrase triggering of Go PreCommit
  Enable phrase triggering of Java PreCommit
  Restore Python precommit
  Allow precommits to coexist with phrase triggering


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/296f7fa9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/296f7fa9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/296f7fa9

Branch: refs/heads/master
Commit: 296f7fa96742147bc799afec1c7e90f0c61540be
Parents: 61777b4 cdefb11
Author: Kenneth Knowles 
Authored: Mon Oct 9 13:11:30 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 9 13:11:30 2017 -0700

--
 .test-infra/jenkins/common_job_properties.groovy | 15 ++-
 .../job_beam_PreCommit_Go_MavenInstall.groovy|  2 +-
 .../job_beam_PreCommit_Java_MavenInstall.groovy  |  4 ++--
 .../job_beam_PreCommit_Python_MavenInstall.groovy|  8 +---
 4 files changed, 14 insertions(+), 15 deletions(-)
--

[2/6] beam git commit: Allow precommits to coexist with phrase triggering

[kenn]

Allow precommits to coexist with phrase triggering


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6416deb0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6416deb0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6416deb0

Branch: refs/heads/master
Commit: 6416deb0f1a706c7ef899f8a227e225ea5a3c2d9
Parents: ec5e724
Author: Kenneth Knowles 
Authored: Fri Oct 6 14:14:29 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 9 09:10:06 2017 -0700

--
 .test-infra/jenkins/common_job_properties.groovy | 15 ++-
 1 file changed, 10 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/6416deb0/.test-infra/jenkins/common_job_properties.groovy
--
diff --git a/.test-infra/jenkins/common_job_properties.groovy 
b/.test-infra/jenkins/common_job_properties.groovy
index 87763a2..c6a8b27 100644
--- a/.test-infra/jenkins/common_job_properties.groovy
+++ b/.test-infra/jenkins/common_job_properties.groovy
@@ -122,8 +122,9 @@ class common_job_properties {
   // below to insulate callers from internal parameter defaults.
   private static void setPullRequestBuildTrigger(context,
  String commitStatusContext,
- String successComment = 
'--none--',
- String prTriggerPhrase = '') {
+ String prTriggerPhrase = '',
+ boolean 
onlyTriggerPhraseToggle = true,
+ String successComment = 
'--none--') {
 context.triggers {
   githubPullRequest {
 admins(['asfbot'])
@@ -138,6 +139,8 @@ class common_job_properties {
 // required to start it.
 if (prTriggerPhrase) {
   triggerPhrase(prTriggerPhrase)
+}
+if (onlyTriggerPhraseToggle) {
   onlyTriggerPhrase()
 }
 
@@ -182,9 +185,10 @@ class common_job_properties {
   // Sets common config for PreCommit jobs.
   static void setPreCommit(context,
String commitStatusName,
+   String prTriggerPhrase = '',
String successComment = '--none--') {
 // Set pull request build trigger.
-setPullRequestBuildTrigger(context, commitStatusName, successComment)
+setPullRequestBuildTrigger(context, commitStatusName, prTriggerPhrase, 
false, successComment)
   }
 
   // Enable triggering postcommit runs against pull requests. Users can 
comment the trigger phrase
@@ -196,8 +200,9 @@ class common_job_properties {
 setPullRequestBuildTrigger(
   context,
   commitStatusName,
-  '--none--',
-  prTriggerPhrase)
+  prTriggerPhrase,
+  true,
+  '--none--')
   }
 
   // Sets common config for PostCommit jobs.

[4/6] beam git commit: Restore Python precommit

[kenn]

Restore Python precommit


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b47c372b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b47c372b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b47c372b

Branch: refs/heads/master
Commit: b47c372b35680a15226824dfe245156465d75de2
Parents: 6416deb
Author: Kenneth Knowles 
Authored: Fri Oct 6 14:14:43 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 9 09:10:06 2017 -0700

--
 .../jenkins/job_beam_PreCommit_Python_MavenInstall.groovy| 8 +---
 1 file changed, 1 insertion(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b47c372b/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
--
diff --git a/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy 
b/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
index 5ce1391..eae129b 100644
--- a/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
+++ b/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
@@ -38,13 +38,7 @@ mavenJob('beam_PreCommit_Python_MavenInstall') {
   common_job_properties.setMavenConfig(delegate)
 
   // Sets that this is a PreCommit job.
-  common_job_properties.setPreCommit(delegate, 'mvn clean install -pl 
sdks/python -am -amd')
-  
-  // Enable Triggering these tests with a comment in the pull request
-  common_job_properties.enablePhraseTriggeringFromPullRequest(
-delegate,
-'Python SDK PreCommit Tests',
-'Run Python PreCommit')
+  common_job_properties.setPreCommit(delegate, 'mvn clean install -pl 
sdks/python -am -amd', 'Run Python PreCommit')
 
   // Maven goals for this job: The Python SDK, its dependencies, and things 
that depend on it.
   goals('''\

[5/6] beam git commit: Add runners/direct-java to Java precommit

[kenn]

Add runners/direct-java to Java precommit

Previously `-pl sdks/java/core -am -amd` was our Maven shorthand for getting
"everything" into the Maven reactor.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/cdefb111
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/cdefb111
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/cdefb111

Branch: refs/heads/master
Commit: cdefb111b542b70667e015d0370f2bee22531284
Parents: 687828f
Author: Kenneth Knowles 
Authored: Mon Oct 9 09:11:06 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 9 09:14:25 2017 -0700

--
 .test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/cdefb111/.test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy
--
diff --git a/.test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy 
b/.test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy
index e8f87b9..8721468 100644
--- a/.test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy
+++ b/.test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy
@@ -38,14 +38,14 @@ mavenJob('beam_PreCommit_Java_MavenInstall') {
   common_job_properties.setMavenConfig(delegate)
 
   // Sets that this is a PreCommit job.
-  common_job_properties.setPreCommit(delegate, 'mvn clean install -pl 
sdks/java/core -am -amd', 'Run Java PreCommit')
+  common_job_properties.setPreCommit(delegate, 'mvn clean install -pl 
sdks/java/core,runners/direct-java -am -amd', 'Run Java PreCommit')
 
   // Maven goals for this job: The Java SDK, its dependencies, and things that 
depend on it.
   goals('''\
 --batch-mode \
 --errors \
 --activate-profiles 
release,jenkins-precommit,direct-runner,dataflow-runner,spark-runner,flink-runner,apex-runner
 \
---projects sdks/java/core \
+--projects sdks/java/core,runners/direct-java \
 --also-make \
 --also-make-dependents \
 -D repoToken=$COVERALLS_REPO_TOKEN \

[1/6] beam git commit: Enable phrase triggering of Go PreCommit

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 61777b433 -> 296f7fa96


Enable phrase triggering of Go PreCommit


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/687828f0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/687828f0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/687828f0

Branch: refs/heads/master
Commit: 687828f0b75b1b8e6fbc29bc6058d58368624a90
Parents: 834494a
Author: Kenneth Knowles 
Authored: Fri Oct 6 14:19:43 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 9 09:10:06 2017 -0700

--
 .test-infra/jenkins/job_beam_PreCommit_Go_MavenInstall.groovy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/687828f0/.test-infra/jenkins/job_beam_PreCommit_Go_MavenInstall.groovy
--
diff --git a/.test-infra/jenkins/job_beam_PreCommit_Go_MavenInstall.groovy 
b/.test-infra/jenkins/job_beam_PreCommit_Go_MavenInstall.groovy
index d078c89..c616edc 100644
--- a/.test-infra/jenkins/job_beam_PreCommit_Go_MavenInstall.groovy
+++ b/.test-infra/jenkins/job_beam_PreCommit_Go_MavenInstall.groovy
@@ -38,7 +38,7 @@ mavenJob('beam_PreCommit_Go_MavenInstall') {
   common_job_properties.setMavenConfig(delegate)
 
   // Sets that this is a PreCommit job.
-  common_job_properties.setPreCommit(delegate, 'mvn clean install -pl sdks/go 
-am -amd')
+  common_job_properties.setPreCommit(delegate, 'mvn clean install -pl sdks/go 
-am -amd', 'Run Go PreCommit')
 
   // Maven goals for this job: The Go SDK, its dependencies, and things that 
depend on it.
   goals('''\

[3/6] beam git commit: Enable phrase triggering of Java PreCommit

[kenn]

Enable phrase triggering of Java PreCommit


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/834494a9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/834494a9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/834494a9

Branch: refs/heads/master
Commit: 834494a9b2cd7027ca8b7d7e05abac608b56a54a
Parents: b47c372
Author: Kenneth Knowles 
Authored: Fri Oct 6 14:17:28 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 9 09:10:06 2017 -0700

--
 .test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/834494a9/.test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy
--
diff --git a/.test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy 
b/.test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy
index dad3726..e8f87b9 100644
--- a/.test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy
+++ b/.test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy
@@ -38,7 +38,7 @@ mavenJob('beam_PreCommit_Java_MavenInstall') {
   common_job_properties.setMavenConfig(delegate)
 
   // Sets that this is a PreCommit job.
-  common_job_properties.setPreCommit(delegate, 'mvn clean install -pl 
sdks/java/core -am -amd')
+  common_job_properties.setPreCommit(delegate, 'mvn clean install -pl 
sdks/java/core -am -amd', 'Run Java PreCommit')
 
   // Maven goals for this job: The Java SDK, its dependencies, and things that 
depend on it.
   goals('''\

[GitHub] beam pull request #3914: [BEAM-2996] Ensure metric names are not null or emp...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3914


---

[1/2] beam git commit: Ensure metric names are not null or empty

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 63de41ac2 -> 61777b433


Ensure metric names are not null or empty


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b6c838e1
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b6c838e1
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b6c838e1

Branch: refs/heads/master
Commit: b6c838e18f56f4b29eee74d2cc7735a154bf44ae
Parents: 63de41a
Author: bchambers 
Authored: Wed Sep 27 10:44:48 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 9 12:47:16 2017 -0700

--
 .../org/apache/beam/sdk/metrics/MetricName.java |  7 +
 .../apache/beam/sdk/metrics/MetricsTest.java| 28 
 sdks/python/apache_beam/metrics/metric_test.py  | 16 +++
 sdks/python/apache_beam/metrics/metricbase.py   |  4 +++
 4 files changed, 55 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b6c838e1/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricName.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricName.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricName.java
index 3c77043..6c88fa2 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricName.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricName.java
@@ -17,7 +17,10 @@
  */
 package org.apache.beam.sdk.metrics;
 
+import static com.google.common.base.Preconditions.checkArgument;
+
 import com.google.auto.value.AutoValue;
+import com.google.common.base.Strings;
 import java.io.Serializable;
 import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.annotations.Experimental.Kind;
@@ -38,10 +41,14 @@ public abstract class MetricName implements Serializable {
   public abstract String name();
 
   public static MetricName named(String namespace, String name) {
+checkArgument(!Strings.isNullOrEmpty(namespace), "Metric namespace must be 
non-empty");
+checkArgument(!Strings.isNullOrEmpty(name), "Metric name must be 
non-empty");
 return new AutoValue_MetricName(namespace, name);
   }
 
   public static MetricName named(Class namespace, String name) {
+checkArgument(namespace != null, "Metric namespace must be non-null");
+checkArgument(!Strings.isNullOrEmpty(name), "Metric name must be 
non-empty");
 return new AutoValue_MetricName(namespace.getName(), name);
   }
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/b6c838e1/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricsTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricsTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricsTest.java
index bc768f8..bdcf892 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricsTest.java
@@ -48,6 +48,7 @@ import org.junit.After;
 import org.junit.Rule;
 import org.junit.Test;
 import org.junit.experimental.categories.Category;
+import org.junit.rules.ExpectedException;
 import org.mockito.Mockito;
 
 /**
@@ -71,6 +72,9 @@ public class MetricsTest implements Serializable {
   @Rule
   public final transient TestPipeline pipeline = TestPipeline.create();
 
+  @Rule
+  public final transient ExpectedException thrown = ExpectedException.none();
+
   @After
   public void tearDown() {
 MetricsEnvironment.setCurrentContainer(null);
@@ -95,6 +99,30 @@ public class MetricsTest implements Serializable {
   }
 
   @Test
+  public void testCounterWithEmptyName() {
+thrown.expect(IllegalArgumentException.class);
+Metrics.counter(NS, "");
+  }
+
+  @Test
+  public void testCounterWithEmptyNamespace() {
+thrown.expect(IllegalArgumentException.class);
+Metrics.counter("", NAME);
+  }
+
+  @Test
+  public void testDistributionWithEmptyName() {
+thrown.expect(IllegalArgumentException.class);
+Metrics.distribution(NS, "");
+  }
+
+  @Test
+  public void testDistributionWithEmptyNamespace() {
+thrown.expect(IllegalArgumentException.class);
+Metrics.distribution("", NAME);
+  }
+
+  @Test
   public void testDistributionToCell() {
 MetricsContainer mockContainer = Mockito.mock(MetricsContainer.class);
 Distribution mockDistribution = Mockito.mock(Distribution.class);

http://git-wip-us.apache.org/repos/asf/beam/blob/b6c838e1/sdks/python/apache_beam/metrics/metric_test.py
--
diff --git a/sdks/python/apache_beam/metrics/metric_test.py 
b/sdks/python/apache_beam/metrics/metric_test.py
index 

[2/2] beam git commit: This closes #3914: [BEAM-2996] Ensure metric names are not null or empty

[kenn]

This closes #3914: [BEAM-2996] Ensure metric names are not null or empty


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/61777b43
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/61777b43
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/61777b43

Branch: refs/heads/master
Commit: 61777b4338733d99f4f858be0d7d0313ec138a06
Parents: 63de41a b6c838e
Author: Kenneth Knowles 
Authored: Mon Oct 9 12:47:57 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 9 12:47:57 2017 -0700

--
 .../org/apache/beam/sdk/metrics/MetricName.java |  7 +
 .../apache/beam/sdk/metrics/MetricsTest.java| 28 
 sdks/python/apache_beam/metrics/metric_test.py  | 16 +++
 sdks/python/apache_beam/metrics/metricbase.py   |  4 +++
 4 files changed, 55 insertions(+)
--

[jira] [Commented] (BEAM-3037) Java DirectRunner jar includes non-relocated runner api protos

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197622#comment-16197622
 ] 

ASF GitHub Bot commented on BEAM-3037:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3964

[BEAM-3037] Relocate sdks.common.v1 and portability.v1 in Java direct runner

Currently these two namespaces are mistakenly left as-is, even though they 
link
to other relocated proto libraries that make them incompatible when used
outside the direct runner.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @tgroh 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam DirectRunner-shading

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3964.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3964


commit 1b59e4d443a346e10a6bbb03492f6b83d207cf85
Author: Kenneth Knowles 
Date:   2017-10-09T20:06:36Z

Relocate sdks.common.v1 and portability.v1 in Java direct runner

Currently these two namespaces are mistakenly left as-is, even though they 
link
to other relocated proto libraries that make them incompatible when used
outside the direct runner.




> Java DirectRunner jar includes non-relocated runner api protos
> --
>
> Key: BEAM-3037
> URL: https://issues.apache.org/jira/browse/BEAM-3037
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> The DirectRunner jar includes classes like 
> {{org.apache.beam.portability.v1.Endpoints}} that actually extend a shaded 
> proto library 
> {{org.apache.beam.runner.direct.repackaged.portability.v1.Endpoints}} so you 
> get very inscrutable messages.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3964: [BEAM-3037] Relocate sdks.common.v1 and portability...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3964

[BEAM-3037] Relocate sdks.common.v1 and portability.v1 in Java direct runner

Currently these two namespaces are mistakenly left as-is, even though they 
link
to other relocated proto libraries that make them incompatible when used
outside the direct runner.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @tgroh 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam DirectRunner-shading

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3964.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3964


commit 1b59e4d443a346e10a6bbb03492f6b83d207cf85
Author: Kenneth Knowles 
Date:   2017-10-09T20:06:36Z

Relocate sdks.common.v1 and portability.v1 in Java direct runner

Currently these two namespaces are mistakenly left as-is, even though they 
link
to other relocated proto libraries that make them incompatible when used
outside the direct runner.




---

[jira] [Created] (BEAM-3037) Java DirectRunner jar includes non-relocated runner api protos

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3037:
-

 Summary: Java DirectRunner jar includes non-relocated runner api 
protos
 Key: BEAM-3037
 URL: https://issues.apache.org/jira/browse/BEAM-3037
 Project: Beam
  Issue Type: Bug
  Components: runner-direct
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


The DirectRunner jar includes classes like 
{{org.apache.beam.portability.v1.Endpoints}} that actually extend a shaded 
proto library 
{{org.apache.beam.runner.direct.repackaged.portability.v1.Endpoints}} so you 
get very inscrutable messages.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #4984

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3934: Add @FunctionalInterface annotation to several Jdbc...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3934


---

[1/2] beam git commit: Add @FunctionalInterface annotation to several interfaces

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master 5130cab4a -> 63de41ac2


Add @FunctionalInterface annotation to several interfaces

By adding the @FunctionalInterface to RowMapper, StatementPreperator and
PreparedStatementSetter it's possible to write a JdbcIO source in a
more concise and functional style.

This change is backwards compatible with existing JdbcIO sources.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c1bfe00f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c1bfe00f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c1bfe00f

Branch: refs/heads/master
Commit: c1bfe00f77bb3726fd7d8282cd1cf41ef2ddd99e
Parents: 5130cab
Author: Ward Van Assche 
Authored: Tue Oct 3 15:13:22 2017 +0200
Committer: Eugene Kirpichov 
Committed: Mon Oct 9 11:42:00 2017 -0700

--
 .../io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 3 +++
 1 file changed, 3 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c1bfe00f/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
--
diff --git 
a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java 
b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
index 6ac6341..b134ec0 100644
--- a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
+++ b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
@@ -169,6 +169,7 @@ public class JdbcIO {
* An interface used by {@link JdbcIO.Read} for converting each row of the 
{@link ResultSet} into
* an element of the resulting {@link PCollection}.
*/
+  @FunctionalInterface
   public interface RowMapper extends Serializable {
 T mapRow(ResultSet resultSet) throws Exception;
   }
@@ -268,6 +269,7 @@ public class JdbcIO {
* An interface used by the JdbcIO Write to set the parameters of the {@link 
PreparedStatement}
* used to setParameters into the database.
*/
+  @FunctionalInterface
   public interface StatementPreparator extends Serializable {
 void setParameters(PreparedStatement preparedStatement) throws Exception;
   }
@@ -499,6 +501,7 @@ public class JdbcIO {
* An interface used by the JdbcIO Write to set the parameters of the {@link 
PreparedStatement}
* used to setParameters into the database.
*/
+  @FunctionalInterface
   public interface PreparedStatementSetter extends Serializable {
 void setParameters(T element, PreparedStatement preparedStatement) throws 
Exception;
   }

[jira] [Commented] (BEAM-3030) watchForNewFiles() can emit a file multiple times if it's growing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197478#comment-16197478
 ] 

ASF GitHub Bot commented on BEAM-3030:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3957


> watchForNewFiles() can emit a file multiple times if it's growing
> -
>
> Key: BEAM-3030
> URL: https://issues.apache.org/jira/browse/BEAM-3030
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
> Fix For: 2.3.0
>
>
> TextIO and AvroIO watchForNewFiles(), as well as 
> FileIO.match().continuously(), use Watch transform under the hood, and watch 
> the set of Metadata matching a filepattern.
> Two Metadata's with the same filename but different size are not considered 
> equal, so if these transforms observe the same file multiple times with 
> different sizes, they'll read the file multiple times.
> This is likely not yet a problem for production users, because these features 
> require SDF, it's supported only in Dataflow runner, and users of the 
> Dataflow runner are likely to use only files on GCS which doesn't support 
> appends. However, this needs to be fixed still.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[1/2] beam git commit: Fixes TextIO and AvroIO tests of watchForNewFiles

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master ec1782c18 -> 5130cab4a


Fixes TextIO and AvroIO tests of watchForNewFiles

* AvroIO: Need to specify a trigger to make sure that files are really
generated continuously and testing of watchForNewFiles is non-vacuous.

* TextIO: files were generated by manual code,
and sometimes writing of a file could race with TextIO reading it, and
it might see the same file with two different sizes, and count it as two
different files (two Metadata objects for the same filename with
different sizes are not equal) and read the file twice.

It makes sense to address that separately: e.g. in the Watch transform
allow specifying a key extractor - but it's outside the scope of this
PR.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/35183c7d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/35183c7d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/35183c7d

Branch: refs/heads/master
Commit: 35183c7d77614f07ed2e643690e07f1b0741efcc
Parents: ec1782c
Author: Eugene Kirpichov 
Authored: Fri Oct 6 13:29:10 2017 -0700
Committer: Eugene Kirpichov 
Committed: Mon Oct 9 11:32:51 2017 -0700

--
 .../java/org/apache/beam/sdk/io/AvroIOTest.java | 21 +---
 .../org/apache/beam/sdk/io/TextIOReadTest.java  | 56 +++-
 2 files changed, 34 insertions(+), 43 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/35183c7d/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java
index 695e196..3976392 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java
@@ -59,7 +59,6 @@ import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.reflect.Nullable;
 import org.apache.avro.reflect.ReflectData;
 import org.apache.avro.reflect.ReflectDatumReader;
-import org.apache.beam.sdk.PipelineResult;
 import org.apache.beam.sdk.coders.AvroCoder;
 import org.apache.beam.sdk.coders.DefaultCoder;
 import org.apache.beam.sdk.coders.StringUtf8Coder;
@@ -81,10 +80,12 @@ import org.apache.beam.sdk.transforms.View;
 import org.apache.beam.sdk.transforms.Watch;
 import org.apache.beam.sdk.transforms.display.DisplayData;
 import org.apache.beam.sdk.transforms.display.DisplayDataEvaluator;
+import org.apache.beam.sdk.transforms.windowing.AfterPane;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.FixedWindows;
 import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
 import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.util.SerializableUtils;
 import org.apache.beam.sdk.values.PCollection;
@@ -307,25 +308,32 @@ public class AvroIOTest {
 for (int i = 0; i < 7; ++i) {
   (i < 3 ? firstValues : secondValues).add(mapFn.apply((long) i));
 }
-writePipeline.apply(
+// Configure windowing of the input so that it fires every time a new 
element is generated,
+// so that files are written continuously.
+Window window = 
Window.into(FixedWindows.of(Duration.millis(100)))
+.withAllowedLateness(Duration.ZERO)
+.triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1)))
+.discardingFiredPanes();
+readPipeline.apply(
 "Sequence first",
 GenerateSequence.from(0).to(3).withRate(1, Duration.millis(300)))
+.apply("Window first", window)
 .apply("Map first", MapElements.via(mapFn))
 .apply(
 "Write first",
 AvroIO.write(GenericClass.class)
 .to(tmpFolder.getRoot().getAbsolutePath() + "/first")
-.withNumShards(2));
-writePipeline.apply(
+.withNumShards(2).withWindowedWrites());
+readPipeline.apply(
 "Sequence second",
 GenerateSequence.from(3).to(7).withRate(1, Duration.millis(300)))
+.apply("Window second", window)
 .apply("Map second", MapElements.via(mapFn))
 .apply(
 "Write second",
 AvroIO.write(GenericClass.class)
 .to(tmpFolder.getRoot().getAbsolutePath() + "/second")
-.withNumShards(3));
-PipelineResult writeRes = writePipeline.run();
+.withNumShards(3).withWindowedWrites());
 
 // Test read(), readAll(), parse(), and parseAllGenericRecords() with 
watchForNewFiles().
 

[2/2] beam git commit: This closes #3957: Fixes TextIO and AvroIO tests of watchForNewFiles

[jkff]

This closes #3957: Fixes TextIO and AvroIO tests of watchForNewFiles


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/5130cab4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/5130cab4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/5130cab4

Branch: refs/heads/master
Commit: 5130cab4a1d39300a266915cac4efebc1ffd658e
Parents: ec1782c 35183c7
Author: Eugene Kirpichov 
Authored: Mon Oct 9 11:33:07 2017 -0700
Committer: Eugene Kirpichov 
Committed: Mon Oct 9 11:33:07 2017 -0700

--
 .../java/org/apache/beam/sdk/io/AvroIOTest.java | 21 +---
 .../org/apache/beam/sdk/io/TextIOReadTest.java  | 56 +++-
 2 files changed, 34 insertions(+), 43 deletions(-)
--

beam git commit: Fix typo in nightly release script

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master eac561f59 -> ec1782c18


Fix typo in nightly release script


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ec1782c1
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ec1782c1
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ec1782c1

Branch: refs/heads/master
Commit: ec1782c1897fbdff4efc536485aac5eb858b9918
Parents: eac561f
Author: Kenneth Knowles 
Authored: Mon Oct 9 10:41:29 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 9 10:41:29 2017 -0700

--
 .test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ec1782c1/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy
--
diff --git a/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy 
b/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy
index 394a0c4..2e3b142 100644
--- a/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy
+++ b/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy
@@ -46,7 +46,7 @@ mavenJob('beam_Release_NightlySnapshot') {
 
   // Maven goals for this job.
   goals('''\
-  clean deploy
+  clean deploy \
   --batch-mode \
   --errors \
   -P release,dataflow-runner \

[GitHub] beam pull request #3963: Extend nightly build timeout to 4 hours

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3963


---

[1/2] beam git commit: Extend nightly build timeout to 4 hours

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master ec5e72403 -> eac561f59


Extend nightly build timeout to 4 hours


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/742a0823
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/742a0823
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/742a0823

Branch: refs/heads/master
Commit: 742a082302d328ed0c6a20c611d7ddf8da64c272
Parents: ec5e724
Author: Kenneth Knowles 
Authored: Mon Oct 9 09:25:55 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 9 09:25:55 2017 -0700

--
 .../job_beam_Release_NightlySnapshot.groovy | 21 +---
 1 file changed, 18 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/742a0823/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy
--
diff --git a/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy 
b/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy
index 7284acd..394a0c4 100644
--- a/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy
+++ b/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy
@@ -27,8 +27,12 @@ mavenJob('beam_Release_NightlySnapshot') {
   // Execute concurrent builds if necessary.
   concurrentBuild()
 
-  // Set common parameters.
-  common_job_properties.setTopLevelMainJobProperties(delegate)
+  // Set common parameters. Huge timeout because we really do need to
+  // run all the ITs and release the artifacts.
+  common_job_properties.setTopLevelMainJobProperties(
+  delegate,
+  'master',
+  240)
 
   // Set maven paramaters.
   common_job_properties.setMavenConfig(delegate)
@@ -41,5 +45,16 @@ mavenJob('beam_Release_NightlySnapshot') {
   'd...@beam.apache.org')
 
   // Maven goals for this job.
-  goals('-B -e clean deploy -P release,dataflow-runner -DskipITs=false 
-DintegrationTestPipelineOptions=\'[ "--project=apache-beam-testing", 
"--tempRoot=gs://temp-storage-for-end-to-end-tests", 
"--runner=TestDataflowRunner" ]\'')
+  goals('''\
+  clean deploy
+  --batch-mode \
+  --errors \
+  -P release,dataflow-runner \
+  -D skipITs=false \
+  -D integrationTestPipelineOptions=\'[ \
+"--project=apache-beam-testing", \
+"--tempRoot=gs://temp-storage-for-end-to-end-tests", \
+"--runner=TestDataflowRunner" \
+  ]\'\
+  ''')
 }

[2/2] beam git commit: This closes #3963: Extend nightly build timeout to 4 hours

[kenn]

This closes #3963: Extend nightly build timeout to 4 hours


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/eac561f5
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/eac561f5
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/eac561f5

Branch: refs/heads/master
Commit: eac561f59b9de67baa67aea492b4cf2004102782
Parents: ec5e724 742a082
Author: Kenneth Knowles 
Authored: Mon Oct 9 10:34:09 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 9 10:34:09 2017 -0700

--
 .../job_beam_Release_NightlySnapshot.groovy | 21 +---
 1 file changed, 18 insertions(+), 3 deletions(-)
--

[GitHub] beam pull request #3963: Extend nightly build timeout to 4 hours

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3963

Extend nightly build timeout to 4 hours

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

The last successful nightly build was Sept 17, and it took 89 minutes. 
Since then, we have had test failures but we also know that the build time has 
increased to exceed the 100 minute timeout. For the nightly build, we should 
have very liberal timeouts since it builds everything and does additional work. 
And anyhow if it takes a long time to push a working snapshot, that is OK.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam super-long-nightly

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3963.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3963


commit 742a082302d328ed0c6a20c611d7ddf8da64c272
Author: Kenneth Knowles 
Date:   2017-10-09T16:25:55Z

Extend nightly build timeout to 4 hours




---

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4981

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4980

[Apache Jenkins Server]

See 

[jira] [Closed] (BEAM-3027) Output type of SourceID-stripper is not correctly set

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed BEAM-3027.
--
Resolution: Fixed

> Output type of SourceID-stripper is not correctly set
> -
>
> Key: BEAM-3027
> URL: https://issues.apache.org/jira/browse/BEAM-3027
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 2.2.0
>
>
> This code: 
> https://github.com/apache/beam/blob/be9fb29901cf4a1ae7b4a9d8e9f25f4ea78359fd/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L193
> Does not set the output type of the flatMap to {{outputTypeInfo}}, which is 
> the correct {{TypeInfo}} that uses the Beam coder. This means that Flink will 
> use Kryo as a fallback, which can be problematic in cases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3027) Output type of SourceID-stripper is not correctly set

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196917#comment-16196917
 ] 

ASF GitHub Bot commented on BEAM-3027:
--

Github user aljoscha closed the pull request at:

https://github.com/apache/beam/pull/3953


> Output type of SourceID-stripper is not correctly set
> -
>
> Key: BEAM-3027
> URL: https://issues.apache.org/jira/browse/BEAM-3027
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 2.2.0
>
>
> This code: 
> https://github.com/apache/beam/blob/be9fb29901cf4a1ae7b4a9d8e9f25f4ea78359fd/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L193
> Does not set the output type of the flatMap to {{outputTypeInfo}}, which is 
> the correct {{TypeInfo}} that uses the Beam coder. This means that Flink will 
> use Kryo as a fallback, which can be problematic in cases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3953: [BEAM-3027] Correctly set output type on SourceId-s...

[aljoscha]

Github user aljoscha closed the pull request at:

https://github.com/apache/beam/pull/3953


---

[1/2] beam git commit: [BEAM-3027] Correctly set output type on SourceId-stripper

[aljoscha]

Repository: beam
Updated Branches:
  refs/heads/master b44056fc6 -> ec5e72403


[BEAM-3027] Correctly set output type on SourceId-stripper


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ca20e69c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ca20e69c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ca20e69c

Branch: refs/heads/master
Commit: ca20e69ce1a817f63600e38a9d4450c2ac3bf949
Parents: b44056f
Author: Aljoscha Krettek 
Authored: Fri Oct 6 14:55:10 2017 +0200
Committer: Aljoscha Krettek 
Committed: Mon Oct 9 14:57:21 2017 +0200

--
 .../beam/runners/flink/FlinkStreamingTransformTranslators.java   | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ca20e69c/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
--
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
index 7cedb56..4d2166c 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
@@ -188,9 +188,9 @@ class FlinkStreamingTransformTranslators {
 if (transform.getSource().requiresDeduping()) {
   source = nonDedupSource.keyBy(
   new ValueWithRecordIdKeySelector())
-  .transform("debuping", outputTypeInfo, new 
DedupingOperator());
+  .transform("deduping", outputTypeInfo, new 
DedupingOperator());
 } else {
-  source = nonDedupSource.flatMap(new StripIdsMap());
+  source = nonDedupSource.flatMap(new 
StripIdsMap()).returns(outputTypeInfo);
 }
   } catch (Exception e) {
 throw new RuntimeException(

[2/2] beam git commit: This closes #3953

[aljoscha]

This closes #3953


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ec5e7240
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ec5e7240
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ec5e7240

Branch: refs/heads/master
Commit: ec5e7240309154be4a94f501bd7dffb326d85a40
Parents: b44056f ca20e69
Author: Aljoscha Krettek 
Authored: Mon Oct 9 14:57:59 2017 +0200
Committer: Aljoscha Krettek 
Committed: Mon Oct 9 14:57:59 2017 +0200

--
 .../beam/runners/flink/FlinkStreamingTransformTranslators.java   | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4979

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_Python #425

[Apache Jenkins Server]

See