Build failed in Jenkins: beam_PerformanceTests_JDBC #175

[Apache Jenkins Server]

See 


--
[...truncated 25.44 KB...]
[INFO] Replacing original test artifact with shaded test artifact.
[INFO] Replacing 

 with 

[INFO] Dependency-reduced POM written at: 

[INFO] 
[INFO] --- maven-failsafe-plugin:2.20:integration-test (integration-test) @ 
beam-runners-google-cloud-dataflow-java ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-dependency-plugin:3.0.1:analyze-only (default) @ 
beam-runners-google-cloud-dataflow-java ---
[INFO] No dependency problems found
[INFO] 
[INFO] --- maven-failsafe-plugin:2.20:verify (integration-test) @ 
beam-runners-google-cloud-dataflow-java ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ 
beam-runners-google-cloud-dataflow-java ---
[INFO] Installing 

 to 
/home/jenkins/.m2/repository/org/apache/beam/beam-runners-google-cloud-dataflow-java/2.2.0-SNAPSHOT/beam-runners-google-cloud-dataflow-java-2.2.0-SNAPSHOT.jar
[INFO] Installing 

 to 
/home/jenkins/.m2/repository/org/apache/beam/beam-runners-google-cloud-dataflow-java/2.2.0-SNAPSHOT/beam-runners-google-cloud-dataflow-java-2.2.0-SNAPSHOT.pom
[INFO] Installing 

 to 
/home/jenkins/.m2/repository/org/apache/beam/beam-runners-google-cloud-dataflow-java/2.2.0-SNAPSHOT/beam-runners-google-cloud-dataflow-java-2.2.0-SNAPSHOT-tests.jar
[INFO] Installing 

 to 
/home/jenkins/.m2/repository/org/apache/beam/beam-runners-google-cloud-dataflow-java/2.2.0-SNAPSHOT/beam-runners-google-cloud-dataflow-java-2.2.0-SNAPSHOT-tests.jar
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 46.189 s
[INFO] Finished at: 2017-07-21T07:00:59Z
[INFO] Final Memory: 90M/1351M
[INFO] 
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins7105811243781283786.sh
+ /home/jenkins/tools/maven/latest/bin/mvn -B -e verify -pl sdks/java/io/jdbc 
-Dio-it-suite 
-DpkbLocation=
 '-DintegrationTestPipelineOptions=[ "--project=apache-beam-testing", 
"--tempRoot=gs://temp-storage-for-end-to-end-tests" ]'
[INFO] Error stacktraces are turned on.
[INFO] Scanning for projects...
[INFO] 
[INFO] Detecting the operating system and CPU architecture
[INFO] 
[INFO] os.detected.name: linux
[INFO] os.detected.arch: x86_64
[INFO] os.detected.version: 3.19
[INFO] os.detected.version.major: 3
[INFO] os.detected.version.minor: 19
[INFO] os.detected.release: ubuntu
[INFO] os.detected.release.version: 14.04
[INFO] os.detected.release.like.ubuntu: true
[INFO] os.detected.release.like.debian: true
[INFO] os.detected.classifier: linux-x86_64
[INFO] 
[INFO] 
[INFO] Building Apache Beam :: SDKs :: Java :: IO :: JDBC 2.2.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce) @ 
beam-sdks-java-io-jdbc ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-banned-dependencies) @ 
beam-sdks-java-io-jdbc ---
[INFO] 
[INFO] --- groovy-maven-plugin:2.0:execute (find-supported-python-for-compile) 
@ beam-sdks-java-io-jdbc ---
[INFO] Using python interpreter binary 'python2.7' with pip 'pip2.7'
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) 
@ beam-sdks-java-io-jdbc ---
[INFO] 
[INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ 
beam-sdks-java-io-jdbc ---

Build failed in Jenkins: beam_PerformanceTests_Python #125

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 1d9160fa3337704b9fbdc423796925be78b0087e (origin/master)
Commit message: "This closes #3590: [BEAM-2628] Makes AvroSource not open files 
while splitting"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 1d9160fa3337704b9fbdc423796925be78b0087e
 > git rev-list 1d9160fa3337704b9fbdc423796925be78b0087e # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2024243883326141565.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1362475219178783464.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5991545095789938608.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): python-gflags==3.1.1 
in /home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
/usr/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8561525467019050251.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://

:66:
 UserWarning: You are using version 1.5.4 of pip. However, version 7.0.0 is 
recommended.
  _PIP_VERSION, REQUIRED_PIP_VERSION
no previously-included directories found matching 'doc/.build'

Installed 


warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3454

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-2653) Python SDK missing OrderByKey

[Shahar Frank (JIRA)]

Shahar Frank created BEAM-2653:
--

 Summary: Python SDK missing OrderByKey
 Key: BEAM-2653
 URL: https://issues.apache.org/jira/browse/BEAM-2653
 Project: Beam
  Issue Type: New Feature
  Components: beam-model, sdk-py
 Environment: Python SDK
Reporter: Shahar Frank
Assignee: Frances Perry
Priority: Critical


Trying to sort a collection by key.

The Java SDK has this implementation: 
https://beam.apache.org/documentation/sdks/javadoc/2.0.0/org/apache/beam/sdk/values/KV.OrderByKey.html

But it seems to be missing in the Python SDK.






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2653) Python SDK missing OrderByKey

[Shahar Frank (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095899#comment-16095899
 ] 

Shahar Frank commented on BEAM-2653:


Link to Stackoverflow for tracking:
https://stackoverflow.com/questions/45165625/apache-beam-2-0-orderbykey


> Python SDK missing OrderByKey
> -
>
> Key: BEAM-2653
> URL: https://issues.apache.org/jira/browse/BEAM-2653
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, sdk-py
> Environment: Python SDK
>Reporter: Shahar Frank
>Assignee: Frances Perry
>Priority: Critical
>
> Trying to sort a collection by key.
> The Java SDK has this implementation: 
> https://beam.apache.org/documentation/sdks/javadoc/2.0.0/org/apache/beam/sdk/values/KV.OrderByKey.html
> But it seems to be missing in the Python SDK.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2676

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #4423

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3624

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-2499) Support Custom Windows in Spark runner

[Etienne Chauchot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Etienne Chauchot updated BEAM-2499:
---
Summary: Support Custom Windows in Spark runner  (was: Fail to merge Custom 
Windows in Spark runner)

> Support Custom Windows in Spark runner
> --
>
> Key: BEAM-2499
> URL: https://issues.apache.org/jira/browse/BEAM-2499
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>
> If we extend {{IntervalWindow}} and we try to merge these custom windows like 
> in this PR:
> https://github.com/apache/beam/pull/3286
> Then spark runner fails with 
> {{org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
> java.lang.ClassCastException: 
> org.apache.beam.sdk.transforms.windowing.IntervalWindow cannot be cast to 
> org.apache.beam.sdk.transforms.windowing.MergingCustomWindowsTest$CustomWindow}}
> It seems to be because of the cast to {{IntervalWindow}} there: 
> https://github.com/apache/beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGlobalCombineFn.java#L111



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2499) Support Custom Windows in Spark runner

[Etienne Chauchot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096196#comment-16096196
 ] 

Etienne Chauchot commented on BEAM-2499:


I'm renaming this ticket and closing the above PR because supporting Custom 
Windows in Spark end up being a lot more than making {{SparkGlobalCombineFn}} 
generic. Indeed, it requires a complete re-coding of the window merging in a 
way similar to what is done in {{HashingFlinkCombineRunner}}.

> Support Custom Windows in Spark runner
> --
>
> Key: BEAM-2499
> URL: https://issues.apache.org/jira/browse/BEAM-2499
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>
> If we extend {{IntervalWindow}} and we try to merge these custom windows like 
> in this PR:
> https://github.com/apache/beam/pull/3286
> Then spark runner fails with 
> {{org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
> java.lang.ClassCastException: 
> org.apache.beam.sdk.transforms.windowing.IntervalWindow cannot be cast to 
> org.apache.beam.sdk.transforms.windowing.MergingCustomWindowsTest$CustomWindow}}
> It seems to be because of the cast to {{IntervalWindow}} there: 
> https://github.com/apache/beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGlobalCombineFn.java#L111



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3517: [BEAM-2499] Make SparkAbstractCombineFn generic to ...

[echauchot]

Github user echauchot closed the pull request at:

https://github.com/apache/beam/pull/3517


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2499) Support Custom Windows in Spark runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096199#comment-16096199
 ] 

ASF GitHub Bot commented on BEAM-2499:
--

Github user echauchot closed the pull request at:

https://github.com/apache/beam/pull/3517


> Support Custom Windows in Spark runner
> --
>
> Key: BEAM-2499
> URL: https://issues.apache.org/jira/browse/BEAM-2499
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>
> If we extend {{IntervalWindow}} and we try to merge these custom windows like 
> in this PR:
> https://github.com/apache/beam/pull/3286
> Then spark runner fails with 
> {{org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
> java.lang.ClassCastException: 
> org.apache.beam.sdk.transforms.windowing.IntervalWindow cannot be cast to 
> org.apache.beam.sdk.transforms.windowing.MergingCustomWindowsTest$CustomWindow}}
> It seems to be because of the cast to {{IntervalWindow}} there: 
> https://github.com/apache/beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGlobalCombineFn.java#L111



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3533: [BEAM-2230] ApiSurface Refactoring

[evindj]

GitHub user evindj reopened a pull request:

https://github.com/apache/beam/pull/3533

[BEAM-2230] ApiSurface Refactoring

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/evindj/beam BEAM-2230

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3533.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3533


commit a0a90f298a5c3131a887193cccb278fbf45e61b5
Author: Innocent Djiofack 
Date:   2017-07-10T04:30:46Z

Made ApiSurface abstract and created implementing subclasses

commit c800fb95b7f9a622f4a9dc2fa061764610739298
Author: Innocent Djiofack 
Date:   2017-07-10T05:34:18Z

Created the direct runner api surface sub class

commit a532663cf2650d0f5dfbc6ea7838e3c5da77a1bb
Author: Innocent Djiofack 
Date:   2017-07-10T05:52:22Z

Modified GcpApiSurface to reflect the correct Api Surface

commit d696b2323bcf89d9272bed94c860dac091bffe3d
Author: Innocent Djiofack 
Date:   2017-07-10T06:02:03Z

Fixed Core Google Cloud API surface.

commit 4a321dc4e07087d0fc4148268f83552fa5b65be6
Author: Innocent Djiofack 
Date:   2017-07-10T11:25:08Z

Fixed a bug in GcpApiSurface

commit 25c6a62bd887066a3c5532a0843355814385b94a
Author: Innocent Djiofack 
Date:   2017-07-10T12:09:52Z

Added ASF Licence in DirectRunnerApiSurface

commit 5977469f6304d7cf2595297ec941f3671ab2b2a7
Author: Innocent Djiofack 
Date:   2017-07-21T04:31:06Z

Merge branch 'master' of https://github.com/apache/beam into BEAM-2230

commit e70a64da7e6899d8db1630d99a361e77f552fa9b
Author: Innocent Djiofack 
Date:   2017-07-21T05:29:36Z

Modified code by review comments.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3533: [BEAM-2230] ApiSurface Refactoring

[evindj]

Github user evindj closed the pull request at:

https://github.com/apache/beam/pull/3533


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2230) Core SDK ApiSurface should be only org.apache.beam.sdk and should be defined outside of the general ApiSurface class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096209#comment-16096209
 ] 

ASF GitHub Bot commented on BEAM-2230:
--

GitHub user evindj reopened a pull request:

https://github.com/apache/beam/pull/3533

[BEAM-2230] ApiSurface Refactoring

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/evindj/beam BEAM-2230

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3533.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3533


commit a0a90f298a5c3131a887193cccb278fbf45e61b5
Author: Innocent Djiofack 
Date:   2017-07-10T04:30:46Z

Made ApiSurface abstract and created implementing subclasses

commit c800fb95b7f9a622f4a9dc2fa061764610739298
Author: Innocent Djiofack 
Date:   2017-07-10T05:34:18Z

Created the direct runner api surface sub class

commit a532663cf2650d0f5dfbc6ea7838e3c5da77a1bb
Author: Innocent Djiofack 
Date:   2017-07-10T05:52:22Z

Modified GcpApiSurface to reflect the correct Api Surface

commit d696b2323bcf89d9272bed94c860dac091bffe3d
Author: Innocent Djiofack 
Date:   2017-07-10T06:02:03Z

Fixed Core Google Cloud API surface.

commit 4a321dc4e07087d0fc4148268f83552fa5b65be6
Author: Innocent Djiofack 
Date:   2017-07-10T11:25:08Z

Fixed a bug in GcpApiSurface

commit 25c6a62bd887066a3c5532a0843355814385b94a
Author: Innocent Djiofack 
Date:   2017-07-10T12:09:52Z

Added ASF Licence in DirectRunnerApiSurface

commit 5977469f6304d7cf2595297ec941f3671ab2b2a7
Author: Innocent Djiofack 
Date:   2017-07-21T04:31:06Z

Merge branch 'master' of https://github.com/apache/beam into BEAM-2230

commit e70a64da7e6899d8db1630d99a361e77f552fa9b
Author: Innocent Djiofack 
Date:   2017-07-21T05:29:36Z

Modified code by review comments.




> Core SDK ApiSurface should be only org.apache.beam.sdk and should be defined 
> outside of the general ApiSurface class
> 
>
> Key: BEAM-2230
> URL: https://issues.apache.org/jira/browse/BEAM-2230
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Innocent
>
> Currenlty, ApiSurface.getSdkApiSurface() is highly specialized and also not 
> correct.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2230) Core SDK ApiSurface should be only org.apache.beam.sdk and should be defined outside of the general ApiSurface class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096208#comment-16096208
 ] 

ASF GitHub Bot commented on BEAM-2230:
--

Github user evindj closed the pull request at:

https://github.com/apache/beam/pull/3533


> Core SDK ApiSurface should be only org.apache.beam.sdk and should be defined 
> outside of the general ApiSurface class
> 
>
> Key: BEAM-2230
> URL: https://issues.apache.org/jira/browse/BEAM-2230
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Innocent
>
> Currenlty, ApiSurface.getSdkApiSurface() is highly specialized and also not 
> correct.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PerformanceTests_Python #126

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 1d9160fa3337704b9fbdc423796925be78b0087e (origin/master)
Commit message: "This closes #3590: [BEAM-2628] Makes AvroSource not open files 
while splitting"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 1d9160fa3337704b9fbdc423796925be78b0087e
 > git rev-list 1d9160fa3337704b9fbdc423796925be78b0087e # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins137452776068905766.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2812935434230867626.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3149452666964721213.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): python-gflags==3.1.1 
in /home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
/usr/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2627207906880064479.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://

:66:
 UserWarning: You are using version 1.5.4 of pip. However, version 7.0.0 is 
recommended.
  _PIP_VERSION, REQUIRED_PIP_VERSION
no previously-included directories found matching 'doc/.build'

Installed 


warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
 

Build failed in Jenkins: beam_PerformanceTests_JDBC #176

[Apache Jenkins Server]

See 


--
[...truncated 25.44 KB...]
[INFO] Replacing original test artifact with shaded test artifact.
[INFO] Replacing 

 with 

[INFO] Dependency-reduced POM written at: 

[INFO] 
[INFO] --- maven-failsafe-plugin:2.20:integration-test (integration-test) @ 
beam-runners-google-cloud-dataflow-java ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-dependency-plugin:3.0.1:analyze-only (default) @ 
beam-runners-google-cloud-dataflow-java ---
[INFO] No dependency problems found
[INFO] 
[INFO] --- maven-failsafe-plugin:2.20:verify (integration-test) @ 
beam-runners-google-cloud-dataflow-java ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ 
beam-runners-google-cloud-dataflow-java ---
[INFO] Installing 

 to 
/home/jenkins/.m2/repository/org/apache/beam/beam-runners-google-cloud-dataflow-java/2.2.0-SNAPSHOT/beam-runners-google-cloud-dataflow-java-2.2.0-SNAPSHOT.jar
[INFO] Installing 

 to 
/home/jenkins/.m2/repository/org/apache/beam/beam-runners-google-cloud-dataflow-java/2.2.0-SNAPSHOT/beam-runners-google-cloud-dataflow-java-2.2.0-SNAPSHOT.pom
[INFO] Installing 

 to 
/home/jenkins/.m2/repository/org/apache/beam/beam-runners-google-cloud-dataflow-java/2.2.0-SNAPSHOT/beam-runners-google-cloud-dataflow-java-2.2.0-SNAPSHOT-tests.jar
[INFO] Installing 

 to 
/home/jenkins/.m2/repository/org/apache/beam/beam-runners-google-cloud-dataflow-java/2.2.0-SNAPSHOT/beam-runners-google-cloud-dataflow-java-2.2.0-SNAPSHOT-tests.jar
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 42.250 s
[INFO] Finished at: 2017-07-21T13:00:57Z
[INFO] Final Memory: 88M/1380M
[INFO] 
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins1958874333985923902.sh
+ /home/jenkins/tools/maven/latest/bin/mvn -B -e verify -pl sdks/java/io/jdbc 
-Dio-it-suite 
-DpkbLocation=
 '-DintegrationTestPipelineOptions=[ "--project=apache-beam-testing", 
"--tempRoot=gs://temp-storage-for-end-to-end-tests" ]'
[INFO] Error stacktraces are turned on.
[INFO] Scanning for projects...
[INFO] 
[INFO] Detecting the operating system and CPU architecture
[INFO] 
[INFO] os.detected.name: linux
[INFO] os.detected.arch: x86_64
[INFO] os.detected.version: 3.19
[INFO] os.detected.version.major: 3
[INFO] os.detected.version.minor: 19
[INFO] os.detected.release: ubuntu
[INFO] os.detected.release.version: 14.04
[INFO] os.detected.release.like.ubuntu: true
[INFO] os.detected.release.like.debian: true
[INFO] os.detected.classifier: linux-x86_64
[INFO] 
[INFO] 
[INFO] Building Apache Beam :: SDKs :: Java :: IO :: JDBC 2.2.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce) @ 
beam-sdks-java-io-jdbc ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-banned-dependencies) @ 
beam-sdks-java-io-jdbc ---
[INFO] 
[INFO] --- groovy-maven-plugin:2.0:execute (find-supported-python-for-compile) 
@ beam-sdks-java-io-jdbc ---
[INFO] Using python interpreter binary 'python2.7' with pip 'pip2.7'
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) 
@ beam-sdks-java-io-jdbc ---
[INFO] 
[INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ 
beam-sdks-java-io-jdbc ---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3455

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2677

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4424

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3625

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-2653) Python SDK missing OrderByKey

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reassigned BEAM-2653:
-

Assignee: (was: Frances Perry)

> Python SDK missing OrderByKey
> -
>
> Key: BEAM-2653
> URL: https://issues.apache.org/jira/browse/BEAM-2653
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, sdk-py
> Environment: Python SDK
>Reporter: Shahar Frank
>Priority: Critical
>
> Trying to sort a collection by key.
> The Java SDK has this implementation: 
> https://beam.apache.org/documentation/sdks/javadoc/2.0.0/org/apache/beam/sdk/values/KV.OrderByKey.html
> But it seems to be missing in the Python SDK.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2654) io-it-suite profile: support ability to use other runners

[Stephen Sisk (JIRA)]

Stephen Sisk created BEAM-2654:
--

 Summary: io-it-suite profile: support ability to use other runners
 Key: BEAM-2654
 URL: https://issues.apache.org/jira/browse/BEAM-2654
 Project: Beam
  Issue Type: Improvement
  Components: testing
Reporter: Stephen Sisk


Currently, we do not have the ability to run integration tests on clusters of > 
1 node using any runners other than dataflow. So we currently do not support 
the ability to invoke them in io-it-suite.

Once we can run the integration tests on clusters of > 1 node, we'll want to be 
able to run the IO ITs on them. That involves:
1.  In the io-it-suite profiles, we should have a runner flag or something like 
it to allow choosing a runner
2.  Add support for them in dpb_service in PerfKit Benchmarker - flesh out 
dpb_service and beam_integration_benchmark for the newly supported runners.

While we are building this, we should improve the mechanism by which we invoke 
the direct runner in io-it-suite. Currently, we use forceDirectRunner, which is 
a maven property that overrides the runner options/profiles to be added to the 
mvn invocation of io-it.  We'll need to talk to the pkb team about how we 
should better support this since we'll need to make a modification on their 
side.

cc [~iemejia] [~jasonkuster]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2653) Python SDK missing OrderByKey

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096437#comment-16096437
 ] 

Ahmet Altay commented on BEAM-2653:
---

[~srfrnk] Could you describe what are you trying to accomplish?

{{OrderByKey}} in Java is a comparator. In Python we do not have a special 
{{KV}} class, instead there is {{(key, values)}} tuples. It is possible to 
write a simple function that compares two of such tuples by the key. Within the 
context of combiners, combiners can take a function like as argument for 
comparison.

> Python SDK missing OrderByKey
> -
>
> Key: BEAM-2653
> URL: https://issues.apache.org/jira/browse/BEAM-2653
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, sdk-py
> Environment: Python SDK
>Reporter: Shahar Frank
>Priority: Critical
>
> Trying to sort a collection by key.
> The Java SDK has this implementation: 
> https://beam.apache.org/documentation/sdks/javadoc/2.0.0/org/apache/beam/sdk/values/KV.OrderByKey.html
> But it seems to be missing in the Python SDK.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2653) Python SDK missing OrderByKey

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-2653:
--
Priority: Minor  (was: Critical)

> Python SDK missing OrderByKey
> -
>
> Key: BEAM-2653
> URL: https://issues.apache.org/jira/browse/BEAM-2653
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, sdk-py
> Environment: Python SDK
>Reporter: Shahar Frank
>Priority: Minor
>
> Trying to sort a collection by key.
> The Java SDK has this implementation: 
> https://beam.apache.org/documentation/sdks/javadoc/2.0.0/org/apache/beam/sdk/values/KV.OrderByKey.html
> But it seems to be missing in the Python SDK.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2653) Python SDK missing OrderByKey

[Shahar Frank (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096509#comment-16096509
 ] 

Shahar Frank commented on BEAM-2653:


Sorry my bad -  I thought it was a PTransform.
I'll go over it again.

> Python SDK missing OrderByKey
> -
>
> Key: BEAM-2653
> URL: https://issues.apache.org/jira/browse/BEAM-2653
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, sdk-py
> Environment: Python SDK
>Reporter: Shahar Frank
>Priority: Minor
> Fix For: Not applicable
>
>
> Trying to sort a collection by key.
> The Java SDK has this implementation: 
> https://beam.apache.org/documentation/sdks/javadoc/2.0.0/org/apache/beam/sdk/values/KV.OrderByKey.html
> But it seems to be missing in the Python SDK.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2653) Python SDK missing OrderByKey

[Shahar Frank (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shahar Frank closed BEAM-2653.
--
   Resolution: Not A Problem
Fix Version/s: Not applicable

> Python SDK missing OrderByKey
> -
>
> Key: BEAM-2653
> URL: https://issues.apache.org/jira/browse/BEAM-2653
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, sdk-py
> Environment: Python SDK
>Reporter: Shahar Frank
>Priority: Minor
> Fix For: Not applicable
>
>
> Trying to sort a collection by key.
> The Java SDK has this implementation: 
> https://beam.apache.org/documentation/sdks/javadoc/2.0.0/org/apache/beam/sdk/values/KV.OrderByKey.html
> But it seems to be missing in the Python SDK.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[2/2] beam git commit: Register a PTransformTranslator for Combine

[tgroh]

Register a PTransformTranslator for Combine

Include the Combine Payload in the Runner API Graph.

Add getCombineFn(AppliedPTransform) to extract the CombineFn from an
arbitrary transform.

Update Pipeline Translation tests to include accumulator coders.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c2110c97
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c2110c97
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c2110c97

Branch: refs/heads/master
Commit: c2110c97d530b8e90387bf99b3ac7d36201d85c7
Parents: 1d9160f
Author: Thomas Groh 
Authored: Wed Jul 19 10:55:33 2017 -0700
Committer: Thomas Groh 
Committed: Fri Jul 21 10:03:30 2017 -0700

--
 .../core/construction/CombineTranslation.java   |  83 -
 .../construction/PTransformTranslation.java |   3 +
 .../construction/CombineTranslationTest.java| 171 +++
 .../core/construction/SdkComponentsTest.java|  14 +-
 4 files changed, 229 insertions(+), 42 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c2110c97/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CombineTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CombineTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CombineTranslation.java
index 472b6f8..2e5b02c 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CombineTranslation.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CombineTranslation.java
@@ -19,19 +19,24 @@
 package org.apache.beam.runners.core.construction;
 
 import static com.google.common.base.Preconditions.checkArgument;
+import static 
org.apache.beam.runners.core.construction.PTransformTranslation.COMBINE_TRANSFORM_URN;
 
+import com.google.auto.service.AutoService;
 import com.google.common.collect.Iterables;
 import com.google.protobuf.Any;
 import com.google.protobuf.ByteString;
 import com.google.protobuf.BytesValue;
 import java.io.IOException;
+import java.util.Collections;
 import java.util.HashMap;
 import java.util.Map;
+import 
org.apache.beam.runners.core.construction.PTransformTranslation.TransformPayloadTranslator;
 import org.apache.beam.sdk.coders.CannotProvideCoderException;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.KvCoder;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi.CombinePayload;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.Components;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi.FunctionSpec;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi.SdkFunctionSpec;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi.SideInput;
@@ -49,7 +54,47 @@ import org.apache.beam.sdk.values.PCollection;
  * RunnerApi.CombinePayload} protos.
  */
 public class CombineTranslation {
-  public static final String JAVA_SERIALIZED_COMBINE_FN_URN = 
"urn:beam:java:combinefn:v1";
+  public static final String JAVA_SERIALIZED_COMBINE_FN_URN = 
"urn:beam:combinefn:javasdk:v1";
+
+   /**
+   * A {@link TransformPayloadTranslator} for {@link Combine.PerKey}.
+   */
+  public static class CombinePayloadTranslator
+  implements 
PTransformTranslation.TransformPayloadTranslator> {
+public static TransformPayloadTranslator create() {
+  return new CombinePayloadTranslator();
+}
+
+private CombinePayloadTranslator() {}
+
+@Override
+public String getUrn(Combine.PerKey transform) {
+  return COMBINE_TRANSFORM_URN;
+}
+
+@Override
+public FunctionSpec translate(
+AppliedPTransform> transform, 
SdkComponents components)
+throws IOException {
+  CombinePayload payload = toProto(transform, components);
+  return RunnerApi.FunctionSpec.newBuilder()
+  .setUrn(COMBINE_TRANSFORM_URN)
+  .setParameter(Any.pack(payload))
+  .build();
+}
+
+/**
+ * Registers {@link CombinePayloadTranslator}.
+ */
+@AutoService(TransformPayloadTranslatorRegistrar.class)
+public static class Registrar implements 
TransformPayloadTranslatorRegistrar {
+  @Override
+  public Map, ? extends 
TransformPayloadTranslator>
+  getTransformPayloadTranslators() {
+return Collections.singletonMap(Combine.PerKey.class, new 
CombinePayloadTranslator());
+  }
+}
+  }
 
   public static CombinePayload toProto(
   AppliedPTransform> combine, SdkComponents 
sdkComponents)
@@ -72,10 +117,11 @@ public class CombineTranslation {
   GlobalCombineFn combineFn,
   Applied

[1/2] beam git commit: This closes #3597

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 1d9160fa3 -> b6f126dc6


This closes #3597


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b6f126dc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b6f126dc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b6f126dc

Branch: refs/heads/master
Commit: b6f126dc6000ceba410d5074fd65af141b7ab3c6
Parents: 1d9160f c2110c9
Author: Thomas Groh 
Authored: Fri Jul 21 10:03:30 2017 -0700
Committer: Thomas Groh 
Committed: Fri Jul 21 10:03:30 2017 -0700

--
 .../core/construction/CombineTranslation.java   |  83 -
 .../construction/PTransformTranslation.java |   3 +
 .../construction/CombineTranslationTest.java| 171 +++
 .../core/construction/SdkComponentsTest.java|  14 +-
 4 files changed, 229 insertions(+), 42 deletions(-)
--

[GitHub] beam pull request #3597: [BEAM-2333] Register a PTransformTranslator for Com...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3597


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2333) Rehydrate Pipeline from Runner API proto

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096529#comment-16096529
 ] 

ASF GitHub Bot commented on BEAM-2333:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3597


> Rehydrate Pipeline from Runner API proto
> 
>
> Key: BEAM-2333
> URL: https://issues.apache.org/jira/browse/BEAM-2333
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: beam-python-everywhere
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2655) Improve Python API reference generation

[David Cavazos (JIRA)]

David Cavazos created BEAM-2655:
---

 Summary: Improve Python API reference generation
 Key: BEAM-2655
 URL: https://issues.apache.org/jira/browse/BEAM-2655
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py
Reporter: David Cavazos
Assignee: Ahmet Altay
Priority: Minor


Changes the theme to something more readable and pleasing to the eyes. Makes 
sphinx do strict checking on pydoc cross references and fix the broken links.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex #2035

[Apache Jenkins Server]

See 


Changes:

[tgroh] Register a PTransformTranslator for Combine

--
[...truncated 53.91 KB...]
2017-07-21T17:06:14.763 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-logging-api/1.1/doxia-logging-api-1.1.pom
 (2 KB at 57.0 KB/sec)
2017-07-21T17:06:14.766 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-container-default/1.0-alpha-30/plexus-container-default-1.0-alpha-30.pom
2017-07-21T17:06:14.793 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-container-default/1.0-alpha-30/plexus-container-default-1.0-alpha-30.pom
 (4 KB at 125.8 KB/sec)
2017-07-21T17:06:14.795 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-containers/1.0-alpha-30/plexus-containers-1.0-alpha-30.pom
2017-07-21T17:06:14.834 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-containers/1.0-alpha-30/plexus-containers-1.0-alpha-30.pom
 (2 KB at 46.2 KB/sec)
2017-07-21T17:06:14.835 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/1.0.11/plexus-1.0.11.pom
2017-07-21T17:06:14.862 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/1.0.11/plexus-1.0.11.pom
 (9 KB at 324.4 KB/sec)
2017-07-21T17:06:14.864 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/1.2-alpha-9/plexus-classworlds-1.2-alpha-9.pom
2017-07-21T17:06:14.901 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/1.2-alpha-9/plexus-classworlds-1.2-alpha-9.pom
 (4 KB at 85.1 KB/sec)
2017-07-21T17:06:14.903 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/1.0.10/plexus-1.0.10.pom
2017-07-21T17:06:14.931 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/1.0.10/plexus-1.0.10.pom
 (9 KB at 287.7 KB/sec)
2017-07-21T17:06:14.934 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-error-diagnostics/2.2.1/maven-error-diagnostics-2.2.1.pom
2017-07-21T17:06:14.961 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-error-diagnostics/2.2.1/maven-error-diagnostics-2.2.1.pom
 (2 KB at 61.8 KB/sec)
2017-07-21T17:06:14.965 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-cli/commons-cli/1.2/commons-cli-1.2.pom
2017-07-21T17:06:14.994 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-cli/commons-cli/1.2/commons-cli-1.2.pom
 (8 KB at 268.7 KB/sec)
2017-07-21T17:06:14.996 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/11/commons-parent-11.pom
2017-07-21T17:06:15.024 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/11/commons-parent-11.pom
 (25 KB at 859.7 KB/sec)
2017-07-21T17:06:15.027 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/4/apache-4.pom
2017-07-21T17:06:15.054 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/4/apache-4.pom (5 KB at 
162.6 KB/sec)
2017-07-21T17:06:15.057 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-descriptor/2.2.1/maven-plugin-descriptor-2.2.1.pom
2017-07-21T17:06:15.083 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-descriptor/2.2.1/maven-plugin-descriptor-2.2.1.pom
 (3 KB at 77.5 KB/sec)
2017-07-21T17:06:15.086 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interactivity-api/1.0-alpha-4/plexus-interactivity-api-1.0-alpha-4.pom
2017-07-21T17:06:15.137 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interactivity-api/1.0-alpha-4/plexus-interactivity-api-1.0-alpha-4.pom
 (7 KB at 135.8 KB/sec)
2017-07-21T17:06:15.140 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-monitor/2.2.1/maven-monitor-2.2.1.pom
2017-07-21T17:06:15.167 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-monitor/2.2.1/maven-monitor-2.2.1.pom
 (2 KB at 45.5 KB/sec)
2017-07-21T17:06:15.171 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/classworlds/classworlds/1.1/classworlds-1.1.pom
2017-07-21T17:06:15.199 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/classworlds/classworlds/1.1/classworlds-1.1.pom
 (4 KB at 115.9 KB/sec)
2017-07-21T17:06:15.201 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.pom
2017-07-21T17:06:15.228 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.pom
 (3 KB at 107.1 KB/sec)
2017-07-21T17:06:15.230 [INFO] Downloadi

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #3626

[Apache Jenkins Server]

See 


Changes:

[tgroh] Register a PTransformTranslator for Combine

--
[...truncated 91.02 KB...]
2017-07-21T17:06:15.756 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.0.8/maven-plugin-registry-2.0.8.pom
 (2 KB at 69.7 KB/sec)
2017-07-21T17:06:15.759 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/2.0.8/maven-plugin-api-2.0.8.pom
2017-07-21T17:06:15.787 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/2.0.8/maven-plugin-api-2.0.8.pom
 (2 KB at 52.1 KB/sec)
2017-07-21T17:06:15.790 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/junit/junit/3.8.2/junit-3.8.2.pom
2017-07-21T17:06:15.818 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/junit/junit/3.8.2/junit-3.8.2.pom (747 B 
at 26.1 KB/sec)
2017-07-21T17:06:15.820 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-filtering/1.1/maven-filtering-1.1.pom
2017-07-21T17:06:15.847 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-filtering/1.1/maven-filtering-1.1.pom
 (6 KB at 209.4 KB/sec)
2017-07-21T17:06:15.850 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.0.6/maven-project-2.0.6.pom
2017-07-21T17:06:15.879 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.0.6/maven-project-2.0.6.pom
 (3 KB at 88.8 KB/sec)
2017-07-21T17:06:15.881 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/2.0.6/maven-2.0.6.pom
2017-07-21T17:06:15.910 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/2.0.6/maven-2.0.6.pom
 (9 KB at 304.7 KB/sec)
2017-07-21T17:06:15.911 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/5/maven-parent-5.pom
2017-07-21T17:06:15.939 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/5/maven-parent-5.pom
 (15 KB at 531.4 KB/sec)
2017-07-21T17:06:15.941 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/3/apache-3.pom
2017-07-21T17:06:15.968 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/3/apache-3.pom (4 KB at 
124.1 KB/sec)
2017-07-21T17:06:15.971 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-settings/2.0.6/maven-settings-2.0.6.pom
2017-07-21T17:06:15.999 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-settings/2.0.6/maven-settings-2.0.6.pom
 (2 KB at 69.8 KB/sec)
2017-07-21T17:06:16.003 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/2.0.6/maven-model-2.0.6.pom
2017-07-21T17:06:16.030 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/2.0.6/maven-model-2.0.6.pom
 (3 KB at 110.2 KB/sec)
2017-07-21T17:06:16.033 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/1.4.1/plexus-utils-1.4.1.pom
2017-07-21T17:06:16.062 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/1.4.1/plexus-utils-1.4.1.pom
 (2 KB at 64.3 KB/sec)
2017-07-21T17:06:16.065 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-profile/2.0.6/maven-profile-2.0.6.pom
2017-07-21T17:06:16.094 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-profile/2.0.6/maven-profile-2.0.6.pom
 (2 KB at 66.7 KB/sec)
2017-07-21T17:06:16.097 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.0.6/maven-artifact-manager-2.0.6.pom
2017-07-21T17:06:16.125 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.0.6/maven-artifact-manager-2.0.6.pom
 (3 KB at 91.4 KB/sec)
2017-07-21T17:06:16.128 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.0.6/maven-repository-metadata-2.0.6.pom
2017-07-21T17:06:16.155 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.0.6/maven-repository-metadata-2.0.6.pom
 (2 KB at 66.9 KB/sec)
2017-07-21T17:06:16.158 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/2.0.6/maven-artifact-2.0.6.pom
2017-07-21T17:06:16.188 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/2.0.6/maven-artifact-2.0.6.pom
 (2 KB at 51.3 KB/sec)
2017-07-21T17:06:16.191 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.0.6/maven-plugin-registry-2.0.6.pom
2017-07-21T17:06:16.220 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.0.6/maven-plugin

[GitHub] beam pull request #3613: [BEAM-2655] Makes Python API reference generation m...

[davidcavazos]

GitHub user davidcavazos opened a pull request:

https://github.com/apache/beam/pull/3613

[BEAM-2655] Makes Python API reference generation more strict

This changes the Python API reference theme to something more readable. It 
also makes cross reference checking more strict and fixes all the broken cross 
references. Pages are also now separated by files and the affected docs were 
reformatted to respect the reStructuredText format that Sphinx expects.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/davidcavazos/beam pydoc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3613.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3613


commit 04fbd1de89ce401c3f3c265b3a8476233aaa8d22
Author: David Cavazos 
Date:   2017-07-21T16:58:11Z

Changed sphinx API reference theme

commit b3275af572aced372bd027763b1eb89230926053
Author: David Cavazos 
Date:   2017-07-21T16:59:30Z

Fixed broken type cross references




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2655) Improve Python API reference generation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096544#comment-16096544
 ] 

ASF GitHub Bot commented on BEAM-2655:
--

GitHub user davidcavazos opened a pull request:

https://github.com/apache/beam/pull/3613

[BEAM-2655] Makes Python API reference generation more strict

This changes the Python API reference theme to something more readable. It 
also makes cross reference checking more strict and fixes all the broken cross 
references. Pages are also now separated by files and the affected docs were 
reformatted to respect the reStructuredText format that Sphinx expects.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/davidcavazos/beam pydoc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3613.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3613


commit 04fbd1de89ce401c3f3c265b3a8476233aaa8d22
Author: David Cavazos 
Date:   2017-07-21T16:58:11Z

Changed sphinx API reference theme

commit b3275af572aced372bd027763b1eb89230926053
Author: David Cavazos 
Date:   2017-07-21T16:59:30Z

Fixed broken type cross references




> Improve Python API reference generation
> ---
>
> Key: BEAM-2655
> URL: https://issues.apache.org/jira/browse/BEAM-2655
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: David Cavazos
>Assignee: Ahmet Altay
>Priority: Minor
>
> Changes the theme to something more readable and pleasing to the eyes. Makes 
> sphinx do strict checking on pydoc cross references and fix the broken links.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3456

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2678

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3614: [BEAM-2652] cleanup pom.xml

[XuMingmin]

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/3614

[BEAM-2652] cleanup pom.xml

R: @kennknowles @rangadi 

@rangadi I also update pom.xml in kafka-io as now kafka-client is shared in 
multiple modules.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam BEAM-2652

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3614.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3614


commit 5dca20f88c1d78ebb65b75c9e777e7af5bcee587
Author: mingmxu 
Date:   2017-07-21T17:40:43Z

update pom.xml




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2652) cleanup pom.xml

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096606#comment-16096606
 ] 

ASF GitHub Bot commented on BEAM-2652:
--

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/3614

[BEAM-2652] cleanup pom.xml

R: @kennknowles @rangadi 

@rangadi I also update pom.xml in kafka-io as now kafka-client is shared in 
multiple modules.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam BEAM-2652

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3614.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3614


commit 5dca20f88c1d78ebb65b75c9e777e7af5bcee587
Author: mingmxu 
Date:   2017-07-21T17:40:43Z

update pom.xml




> cleanup pom.xml
> ---
>
> Key: BEAM-2652
> URL: https://issues.apache.org/jira/browse/BEAM-2652
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_review
>
> Update pom.xml as mentioned in 
> https://github.com/apache/beam/pull/3606#pullrequestreview-51373781 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4425

[Apache Jenkins Server]

See 

[1/2] beam git commit: Use RehydratedComponents for memoized rehydration

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master b6f126dc6 -> f870bf516


Use RehydratedComponents for memoized rehydration


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/01492e69
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/01492e69
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/01492e69

Branch: refs/heads/master
Commit: 01492e69d1b1eedb99931f7071a9dae9d60d25fe
Parents: b6f126d
Author: Kenneth Knowles 
Authored: Wed Jul 19 20:58:36 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jul 21 11:05:41 2017 -0700

--
 .../core/construction/CoderTranslation.java |  14 +-
 .../core/construction/CombineTranslation.java   |   7 +-
 .../construction/PCollectionTranslation.java|  28 +---
 .../core/construction/ParDoTranslation.java |  48 ++
 .../core/construction/RehydratedComponents.java | 158 +++
 .../construction/TestStreamTranslation.java |  11 +-
 .../WindowingStrategyTranslation.java   |   6 +-
 .../core/construction/CoderTranslationTest.java |   4 +-
 .../construction/CombineTranslationTest.java|   9 +-
 .../PCollectionTranslationTest.java |  12 +-
 .../core/construction/ParDoTranslationTest.java |  61 +++
 .../construction/RehydratedComponentsTest.java  |  96 +++
 .../construction/TestStreamTranslationTest.java |  11 +-
 .../WindowingStrategyTranslationTest.java   |   5 +-
 .../beam/runners/dataflow/DataflowRunner.java   |  16 +-
 15 files changed, 355 insertions(+), 131 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/01492e69/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslation.java
index 470db6a..a6719ff 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslation.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslation.java
@@ -40,7 +40,6 @@ import org.apache.beam.sdk.coders.LengthPrefixCoder;
 import org.apache.beam.sdk.coders.StructuredCoder;
 import org.apache.beam.sdk.coders.VarLongCoder;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi;
-import org.apache.beam.sdk.common.runner.v1.RunnerApi.Components;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi.FunctionSpec;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi.SdkFunctionSpec;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
@@ -149,21 +148,22 @@ public class CoderTranslation {
 .build();
   }
 
-  public static Coder fromProto(RunnerApi.Coder protoCoder, Components 
components)
+  public static Coder fromProto(
+  RunnerApi.Coder protoCoder, RehydratedComponents components)
   throws IOException {
 String coderSpecUrn = protoCoder.getSpec().getSpec().getUrn();
 if (coderSpecUrn.equals(JAVA_SERIALIZED_CODER_URN)) {
-  return fromCustomCoder(protoCoder, components);
+  return fromCustomCoder(protoCoder);
 }
 return fromKnownCoder(protoCoder, components);
   }
 
-  private static Coder fromKnownCoder(RunnerApi.Coder coder, Components 
components)
+  private static Coder fromKnownCoder(RunnerApi.Coder coder, 
RehydratedComponents components)
   throws IOException {
 String coderUrn = coder.getSpec().getSpec().getUrn();
 List> coderComponents = new LinkedList<>();
 for (String componentId : coder.getComponentCoderIdsList()) {
-  Coder innerCoder = 
fromProto(components.getCodersOrThrow(componentId), components);
+  Coder innerCoder = components.getCoder(componentId);
   coderComponents.add(innerCoder);
 }
 Class coderType = 
KNOWN_CODER_URNS.inverse().get(coderUrn);
@@ -176,9 +176,7 @@ public class CoderTranslation {
 return translator.fromComponents(coderComponents);
   }
 
-  private static Coder fromCustomCoder(
-  RunnerApi.Coder protoCoder, @SuppressWarnings("unused") Components 
components)
-  throws IOException {
+  private static Coder fromCustomCoder(RunnerApi.Coder protoCoder) throws 
IOException {
 return (Coder)
 SerializableUtils.deserializeFromByteArray(
 protoCoder

http://git-wip-us.apache.org/repos/asf/beam/blob/01492e69/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CombineTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Combin

[2/2] beam git commit: This closes #3601: Use RehydratedComponents for memoized rehydration

[kenn]

This closes #3601: Use RehydratedComponents for memoized rehydration


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f870bf51
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f870bf51
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f870bf51

Branch: refs/heads/master
Commit: f870bf5164b00a1379f1e33a5ef27b3701a87c08
Parents: b6f126d 01492e6
Author: Kenneth Knowles 
Authored: Fri Jul 21 11:06:02 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jul 21 11:06:02 2017 -0700

--
 .../core/construction/CoderTranslation.java |  14 +-
 .../core/construction/CombineTranslation.java   |   7 +-
 .../construction/PCollectionTranslation.java|  28 +---
 .../core/construction/ParDoTranslation.java |  48 ++
 .../core/construction/RehydratedComponents.java | 158 +++
 .../construction/TestStreamTranslation.java |  11 +-
 .../WindowingStrategyTranslation.java   |   6 +-
 .../core/construction/CoderTranslationTest.java |   4 +-
 .../construction/CombineTranslationTest.java|   9 +-
 .../PCollectionTranslationTest.java |  12 +-
 .../core/construction/ParDoTranslationTest.java |  61 +++
 .../construction/RehydratedComponentsTest.java  |  96 +++
 .../construction/TestStreamTranslationTest.java |  11 +-
 .../WindowingStrategyTranslationTest.java   |   5 +-
 .../beam/runners/dataflow/DataflowRunner.java   |  16 +-
 15 files changed, 355 insertions(+), 131 deletions(-)
--

[GitHub] beam pull request #3601: [BEAM-2333] Use RehydratedComponents for memoized r...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3601


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2333) Rehydrate Pipeline from Runner API proto

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096675#comment-16096675
 ] 

ASF GitHub Bot commented on BEAM-2333:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3601


> Rehydrate Pipeline from Runner API proto
> 
>
> Key: BEAM-2333
> URL: https://issues.apache.org/jira/browse/BEAM-2333
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: beam-python-everywhere
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #2774

[Apache Jenkins Server]

See 


Changes:

[klk] Use RehydratedComponents for memoized rehydration

--
[...truncated 320.97 KB...]
test_write_no_write_results 
(apache_beam.transforms.write_ptransform_test.WriteTest) ... ok
test_write_with_empty_pcollection 
(apache_beam.transforms.write_ptransform_test.WriteTest) ... ok
testAlwaysReturnsEarly 
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) ... ok
testBinOp (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
testBuiltins 
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) ... ok
testCall (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
testClosure (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
testDict (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
testDictComprehension 
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) ... ok
testGenerator 
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) ... ok
testGetAttr (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
testGlobals (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
testIdentity 
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) ... ok
testListComprehension 
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) ... ok
testMethod (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
testNoneReturn 
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) ... ok
testTupleListComprehension 
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) ... ok
testTuples (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
testUnpack (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
test_custom_transform 
(apache_beam.typehints.typed_pipeline_test.CustomTransformTest) ... ok
test_flat_type_hint 
(apache_beam.typehints.typed_pipeline_test.CustomTransformTest) ... ok
test_keyword_type_hints 
(apache_beam.typehints.typed_pipeline_test.CustomTransformTest) ... ok
test_bad_main_input (apache_beam.typehints.typed_pipeline_test.MainInputTest) 
... ok
test_loose_bounds (apache_beam.typehints.typed_pipeline_test.MainInputTest) ... 
ok
test_non_function (apache_beam.typehints.typed_pipeline_test.MainInputTest) ... 
ok
test_typed_dofn_class (apache_beam.typehints.typed_pipeline_test.MainInputTest) 
... ok
test_typed_dofn_instance 
(apache_beam.typehints.typed_pipeline_test.MainInputTest) ... ok
test_basic_side_input_hint 
(apache_beam.typehints.typed_pipeline_test.SideInputTest) ... ok
test_default_typed_hint 
(apache_beam.typehints.typed_pipeline_test.SideInputTest) ... ok
test_default_untyped_hint 
(apache_beam.typehints.typed_pipeline_test.SideInputTest) ... ok
test_deferred_side_input_iterable 
(apache_beam.typehints.typed_pipeline_test.SideInputTest) ... ok
test_deferred_side_inputs 
(apache_beam.typehints.typed_pipeline_test.SideInputTest) ... ok
test_keyword_side_input_hint 
(apache_beam.typehints.typed_pipeline_test.SideInputTest) ... ok
test_any_compatibility 
(apache_beam.typehints.typehints_test.AnyTypeConstraintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.AnyTypeConstraintTestCase) ... 
ok
test_type_check 
(apache_beam.typehints.typehints_test.AnyTypeConstraintTestCase) ... ok
test_composite_takes_and_returns_hints 
(apache_beam.typehints.typehints_test.CombinedReturnsAndTakesTestCase) ... ok
test_enable_and_disable_type_checking_returns 
(apache_beam.typehints.typehints_test.CombinedReturnsAndTakesTestCase) ... ok
test_enable_and_disable_type_checking_takes 
(apache_beam.typehints.typehints_test.CombinedReturnsAndTakesTestCase) ... ok
test_simple_takes_and_returns_hints 
(apache_beam.typehints.typehints_test.CombinedReturnsAndTakesTestCase) ... ok
test_valid_mix_pos_and_keyword_with_both_orders 
(apache_beam.typehints.typehints_test.CombinedReturnsAndTakesTestCase) ... ok
test_getcallargs_forhints 
(apache_beam.typehints.typehints_test.DecoratorHelpers) ... ok
test_hint_helper (apache_beam.typehints.typehints_test.DecoratorHelpers) ... ok
test_positional_arg_hints 
(apache_beam.typehints.typehints_test.DecoratorHelpers) ... ok
test_compatibility (apache_beam.typehints.typehints_test.DictHintTestCase) ... 
ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... 
:497:
 DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  e.exception.message)
ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_key_type_must_be_valid_composite_param 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3457

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2648) beam_PerformanceTests_Python failing since 2017-07-17

[Mark Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096694#comment-16096694
 ] 

Mark Liu commented on BEAM-2648:


fix PR: https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1449 is 
created.

> beam_PerformanceTests_Python failing since 2017-07-17
> -
>
> Key: BEAM-2648
> URL: https://issues.apache.org/jira/browse/BEAM-2648
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Kenneth Knowles
>Assignee: Mark Liu
>
> See 
> https://builds.apache.org/blue/organizations/jenkins/beam_PerformanceTests_Python/activity



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[1/2] beam git commit: Use dehydration-insensitive APIs in WindowEvaluatorFactory

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master f870bf516 -> 8bd647596


Use dehydration-insensitive APIs in WindowEvaluatorFactory


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e38dc5fb
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e38dc5fb
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e38dc5fb

Branch: refs/heads/master
Commit: e38dc5fbe1f45fe282b4cd96b858631db73130e9
Parents: b6f126d
Author: Kenneth Knowles 
Authored: Wed Jun 7 13:58:11 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jul 21 11:36:21 2017 -0700

--
 .../construction/WindowIntoTranslation.java | 46 +--
 .../WindowingStrategyTranslation.java   | 83 
 .../construction/WindowIntoTranslationTest.java |  2 +-
 .../runners/direct/WindowEvaluatorFactory.java  |  5 +-
 4 files changed, 95 insertions(+), 41 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e38dc5fb/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowIntoTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowIntoTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowIntoTranslation.java
index aa17bc9..6aec908 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowIntoTranslation.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowIntoTranslation.java
@@ -18,15 +18,17 @@
 
 package org.apache.beam.runners.core.construction;
 
+import static com.google.common.base.Preconditions.checkArgument;
+
 import com.google.auto.service.AutoService;
 import com.google.protobuf.Any;
 import com.google.protobuf.InvalidProtocolBufferException;
+import java.io.IOException;
 import java.util.Collections;
 import java.util.Map;
 import 
org.apache.beam.runners.core.construction.PTransformTranslation.TransformPayloadTranslator;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi.FunctionSpec;
-import org.apache.beam.sdk.common.runner.v1.RunnerApi.SdkFunctionSpec;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi.WindowIntoPayload;
 import org.apache.beam.sdk.runners.AppliedPTransform;
 import org.apache.beam.sdk.transforms.PTransform;
@@ -64,10 +66,44 @@ public class WindowIntoTranslation {
 .build();
   }
 
-  public static WindowFn getWindowFn(WindowIntoPayload payload)
-  throws InvalidProtocolBufferException {
-SdkFunctionSpec spec = payload.getWindowFn();
-return WindowingStrategyTranslation.windowFnFromProto(spec);
+  public static WindowIntoPayload getWindowIntoPayload(AppliedPTransform application) {
+RunnerApi.PTransform transformProto;
+try {
+  transformProto =
+  PTransformTranslation.toProto(
+  application,
+  Collections.>emptyList(),
+  SdkComponents.create());
+} catch (IOException exc) {
+  throw new RuntimeException(exc);
+}
+
+checkArgument(
+
PTransformTranslation.WINDOW_TRANSFORM_URN.equals(transformProto.getSpec().getUrn()),
+"Illegal attempt to extract %s from transform %s with name \"%s\" and 
URN \"%s\"",
+Window.Assign.class.getSimpleName(),
+application.getTransform(),
+application.getFullName(),
+transformProto.getSpec().getUrn());
+
+WindowIntoPayload windowIntoPayload;
+try {
+  return 
transformProto.getSpec().getParameter().unpack(WindowIntoPayload.class);
+} catch (InvalidProtocolBufferException exc) {
+  throw new IllegalStateException(
+  String.format(
+  "%s translated %s with URN '%s' but payload was not a %s",
+  PTransformTranslation.class.getSimpleName(),
+  application,
+  PTransformTranslation.WINDOW_TRANSFORM_URN,
+  WindowIntoPayload.class.getSimpleName()),
+  exc);
+}
+  }
+
+  public static WindowFn getWindowFn(AppliedPTransform 
application) {
+return WindowingStrategyTranslation.windowFnFromProto(
+getWindowIntoPayload(application).getWindowFn());
   }
 
   /**

http://git-wip-us.apache.org/repos/asf/beam/blob/e38dc5fb/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/c

[2/2] beam git commit: This closes #3609: Use dehydration-insensitive APIs in WindowEvaluatorFactory

[kenn]

This closes #3609: Use dehydration-insensitive APIs in WindowEvaluatorFactory


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8bd64759
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8bd64759
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8bd64759

Branch: refs/heads/master
Commit: 8bd64759616a2f262f60ce2998b98f0dd1dd2000
Parents: f870bf5 e38dc5f
Author: Kenneth Knowles 
Authored: Fri Jul 21 11:36:46 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jul 21 11:36:46 2017 -0700

--
 .../construction/WindowIntoTranslation.java | 46 +--
 .../WindowingStrategyTranslation.java   | 83 
 .../construction/WindowIntoTranslationTest.java |  2 +-
 .../runners/direct/WindowEvaluatorFactory.java  |  5 +-
 4 files changed, 95 insertions(+), 41 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/8bd64759/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java
--

[GitHub] beam pull request #3609: [BEAM-2371] Use dehydration-insensitive APIs in Win...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3609


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096698#comment-16096698
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3609


> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: beam-python-everywhere
> Fix For: 2.2.0
>
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PerformanceTests_Python #127

[Apache Jenkins Server]

See 


Changes:

[tgroh] Register a PTransformTranslator for Combine

[klk] Use RehydratedComponents for memoized rehydration

[klk] Use dehydration-insensitive APIs in WindowEvaluatorFactory

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 8bd64759616a2f262f60ce2998b98f0dd1dd2000 (origin/master)
Commit message: "This closes #3609: Use dehydration-insensitive APIs in 
WindowEvaluatorFactory"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8bd64759616a2f262f60ce2998b98f0dd1dd2000
 > git rev-list 1d9160fa3337704b9fbdc423796925be78b0087e # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins9140571482146026102.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3189760049654706488.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7786261236385433510.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): python-gflags==3.1.1 
in /home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
/usr/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5021987718170214084.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://

:66:
 UserWarning: You are using version 1.5.4 of pip. However, version 7.0.0 is 
recommended.
  _PIP_VERSION, REQUIRED_PIP_VERSION
no previously-included directories found matching 'doc/.build'

Installed

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Apex #2036

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2679

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3458

[Apache Jenkins Server]

See 

[1/2] beam git commit: Use dehydration-insensitive APIs in ParDoEvaluatorFactory

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 8bd647596 -> 4b355844a


Use dehydration-insensitive APIs in ParDoEvaluatorFactory


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/fa3a5abb
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/fa3a5abb
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/fa3a5abb

Branch: refs/heads/master
Commit: fa3a5abbc94db629feae8d7d73a31e7dda06bf76
Parents: b6f126d
Author: Kenneth Knowles 
Authored: Wed Jun 7 14:35:09 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jul 21 11:38:23 2017 -0700

--
 .../runners/direct/ParDoEvaluatorFactory.java | 18 ++
 1 file changed, 6 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/fa3a5abb/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
index 516f798..408a7df 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
@@ -23,6 +23,7 @@ import com.google.common.cache.LoadingCache;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
+import org.apache.beam.runners.core.construction.ParDoTranslation;
 import org.apache.beam.runners.direct.DirectExecutionContext.DirectStepContext;
 import org.apache.beam.sdk.runners.AppliedPTransform;
 import org.apache.beam.sdk.transforms.DoFn;
@@ -63,15 +64,8 @@ final class ParDoEvaluatorFactory 
implements TransformEvaluator
   public  TransformEvaluator forApplication(
   AppliedPTransform application, CommittedBundle inputBundle) 
throws Exception {
 
-@SuppressWarnings("unchecked")
-AppliedPTransform, PCollectionTuple, 
ParDo.MultiOutput>
-parDoApplication =
-(AppliedPTransform<
-PCollection, PCollectionTuple, 
ParDo.MultiOutput>)
-application;
-
-ParDo.MultiOutput transform = 
parDoApplication.getTransform();
-final DoFn doFn = transform.getFn();
+final DoFn doFn =
+(DoFn) ParDoTranslation.getDoFn(application);
 
 @SuppressWarnings({"unchecked", "rawtypes"})
 TransformEvaluator evaluator =
@@ -81,9 +75,9 @@ final class ParDoEvaluatorFactory implements 
TransformEvaluator
 (PCollection) inputBundle.getPCollection(),
 inputBundle.getKey(),
 doFn,
-transform.getSideInputs(),
-transform.getMainOutputTag(),
-transform.getAdditionalOutputTags().getAll());
+ParDoTranslation.getSideInputs(application),
+(TupleTag) 
ParDoTranslation.getMainOutputTag(application),
+
ParDoTranslation.getAdditionalOutputTags(application).getAll());
 return evaluator;
   }
 

[2/2] beam git commit: This closes #3610: Use dehydration-insensitive APIs in ParDoEvaluatorFactory

[kenn]

This closes #3610: Use dehydration-insensitive APIs in ParDoEvaluatorFactory


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4b355844
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4b355844
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4b355844

Branch: refs/heads/master
Commit: 4b355844a4920bc9faba75f7cd61008bedebaf29
Parents: 8bd6475 fa3a5ab
Author: Kenneth Knowles 
Authored: Fri Jul 21 11:50:48 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jul 21 11:50:48 2017 -0700

--
 .../runners/direct/ParDoEvaluatorFactory.java | 18 ++
 1 file changed, 6 insertions(+), 12 deletions(-)
--

[GitHub] beam pull request #3610: [BEAM-2371] Use dehydration-insensitive APIs in Par...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3610


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096726#comment-16096726
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3610


> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: beam-python-everywhere
> Fix For: 2.2.0
>
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3459

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2680

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3460

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #2775

[Apache Jenkins Server]

See 

[2/2] beam git commit: This closes #3608: Uniquify application nodes in TextIOReadTest and only run pipeline once

[kenn]

This closes #3608: Uniquify application nodes in TextIOReadTest and only run 
pipeline once


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/81c2e90f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/81c2e90f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/81c2e90f

Branch: refs/heads/master
Commit: 81c2e90f3a9d109121b686c25b642cc9f16ced43
Parents: 4b35584 cfadecb
Author: Kenneth Knowles 
Authored: Fri Jul 21 12:21:05 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jul 21 12:21:05 2017 -0700

--
 .../org/apache/beam/sdk/io/TextIOReadTest.java  | 22 
 1 file changed, 18 insertions(+), 4 deletions(-)
--

[1/2] beam git commit: Uniquify application nodes in TextIOReadTest and only run pipeline once

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 4b355844a -> 81c2e90f3


Uniquify application nodes in TextIOReadTest and only run pipeline once

 - These nodes have never had stable unique names, but we have always mutated
   the pipeline before running it so they got lucky.
 - A pipeline can also be mutated when run() so it should be considered dead
   after run() is called.

This fixes both issues by uniquifying the names and running a given pipeline
only once.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/cfadecb5
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/cfadecb5
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/cfadecb5

Branch: refs/heads/master
Commit: cfadecb56c64aa155c8b5fd0d8a6654ceb918eba
Parents: 1d9160f
Author: Kenneth Knowles 
Authored: Mon Jul 17 22:03:09 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 20 20:06:54 2017 -0700

--
 .../org/apache/beam/sdk/io/TextIOReadTest.java  | 22 
 1 file changed, 18 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/cfadecb5/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOReadTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOReadTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOReadTest.java
index 8b53111..e733010 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOReadTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOReadTest.java
@@ -94,6 +94,7 @@ public class TextIOReadTest {
   private static final List TINY =
   Arrays.asList("Irritable eagle", "Optimistic jay", "Fanciful hawk");
   private static final List LARGE = makeLines(1000);
+  private static int uniquifier = 0;
 
   private static Path tempFolder;
   private static File emptyTxt;
@@ -296,17 +297,19 @@ public class TextIOReadTest {
   private void assertReadingCompressedFileMatchesExpected(
   File file, CompressionType compressionType, List expected) {
 
+int thisUniquifier = ++uniquifier;
+
 TextIO.Read read = 
TextIO.read().from(file.getPath()).withCompressionType(compressionType);
-PAssert.that(p.apply("Read_" + file + "_" + compressionType.toString(), 
read))
+PAssert.that(
+p.apply("Read_" + file + "_" + compressionType.toString() + "_" + 
thisUniquifier, read))
 .containsInAnyOrder(expected);
 
 TextIO.ReadAll readAll =
 
TextIO.readAll().withCompressionType(compressionType).withDesiredBundleSizeBytes(10);
 PAssert.that(
-p.apply("Create_" + file, Create.of(file.getPath()))
-.apply("Read_" + compressionType.toString(), readAll))
+p.apply("Create_" + file + "_" + thisUniquifier, 
Create.of(file.getPath()))
+.apply("Read_" + compressionType.toString() + "_" + 
thisUniquifier, readAll))
 .containsInAnyOrder(expected);
-p.run();
   }
 
   /** Helper to make an array of compressible strings. Returns ["word"i] for i 
in range(0,n). */
@@ -324,6 +327,7 @@ public class TextIOReadTest {
   public void testSmallCompressedGzipReadNoExtension() throws Exception {
 File smallGzNoExtension = writeToFile(TINY, "tiny_gz_no_extension", GZIP);
 assertReadingCompressedFileMatchesExpected(smallGzNoExtension, GZIP, TINY);
+p.run();
   }
 
   /**
@@ -340,6 +344,7 @@ public class TextIOReadTest {
 assertReadingCompressedFileMatchesExpected(smallGzNotCompressed, GZIP, 
TINY);
 // Should also work with AUTO mode set.
 assertReadingCompressedFileMatchesExpected(smallGzNotCompressed, AUTO, 
TINY);
+p.run();
   }
 
   /** Tests reading from a small, bzip2ed file with no .bz2 extension but 
BZIP2 compression set. */
@@ -348,6 +353,7 @@ public class TextIOReadTest {
   public void testSmallCompressedBzip2ReadNoExtension() throws Exception {
 File smallBz2NoExtension = writeToFile(TINY, "tiny_bz2_no_extension", 
BZIP2);
 assertReadingCompressedFileMatchesExpected(smallBz2NoExtension, BZIP2, 
TINY);
+p.run();
   }
 
   /**
@@ -393,6 +399,7 @@ public class TextIOReadTest {
   assertReadingCompressedFileMatchesExpected(tinyTxt, type, TINY);
   assertReadingCompressedFileMatchesExpected(largeTxt, type, LARGE);
 }
+p.run();
   }
 
   @Test
@@ -411,6 +418,7 @@ public class TextIOReadTest {
 // GZIP files with non-gz extension should work in GZIP mode.
 File gzFile = writeToFile(TINY, "tiny_gz_no_extension", GZIP);
 assertReadingCompressedFileMatchesExpected(gzFile, GZIP, TINY);
+p.run();
   }
 
   @Test
@@ -429,6 +437,7 @@ public class TextIOReadTest {
 // BZ2 files with non-bz2 extension should work in BZIP2 mode.
 File bz2File = writeToFile(TINY, "ti

[GitHub] beam pull request #3608: Uniquify application nodes in TextIOReadTest

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3608


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4426

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3461

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2681

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3627

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4427

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4428

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4429

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2682

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4430

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-2656) Introduce AvroIO.readAll()

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-2656:
--

 Summary: Introduce AvroIO.readAll()
 Key: BEAM-2656
 URL: https://issues.apache.org/jira/browse/BEAM-2656
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Eugene Kirpichov


TextIO.readAll() is nifty and performant when reading a large number of files.

We should similarly have AvroIO.readAll(). Maybe other connectors too in the 
future.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3615: [BEAM-2656] Introduces AvroIO.readAll()

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/3615

[BEAM-2656] Introduces AvroIO.readAll()

This includes a couple more improvements:

- Adding ValueProvider support to AvroIO.read()
- Creating a common utility transform for reading a collection of 
filepatterns via FileBasedSource. I did not make it public because it should 
bake some more with other use cases.

R: @jbonofre 
CC: @reuvenlax 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam avroio-readall

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3615.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3615


commit 0824e3982bd47f54b917da250497923049d23ca9
Author: Eugene Kirpichov 
Date:   2017-07-21T19:38:17Z

Adds ValueProvider support to AvroIO.Read

commit f988deb238178e28227443b5e8fc389c96f907cf
Author: Eugene Kirpichov 
Date:   2017-07-21T21:09:13Z

Extracts common logic from TextIO.ReadAll into a utility transform

commit 7345de77b2fa658970addfddc21fecaf57dc3775
Author: Eugene Kirpichov 
Date:   2017-07-21T21:09:35Z

Introduces AvroIO.readAll() and readAllGenericRecords()




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2656) Introduce AvroIO.readAll()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096882#comment-16096882
 ] 

ASF GitHub Bot commented on BEAM-2656:
--

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/3615

[BEAM-2656] Introduces AvroIO.readAll()

This includes a couple more improvements:

- Adding ValueProvider support to AvroIO.read()
- Creating a common utility transform for reading a collection of 
filepatterns via FileBasedSource. I did not make it public because it should 
bake some more with other use cases.

R: @jbonofre 
CC: @reuvenlax 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam avroio-readall

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3615.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3615


commit 0824e3982bd47f54b917da250497923049d23ca9
Author: Eugene Kirpichov 
Date:   2017-07-21T19:38:17Z

Adds ValueProvider support to AvroIO.Read

commit f988deb238178e28227443b5e8fc389c96f907cf
Author: Eugene Kirpichov 
Date:   2017-07-21T21:09:13Z

Extracts common logic from TextIO.ReadAll into a utility transform

commit 7345de77b2fa658970addfddc21fecaf57dc3775
Author: Eugene Kirpichov 
Date:   2017-07-21T21:09:35Z

Introduces AvroIO.readAll() and readAllGenericRecords()




> Introduce AvroIO.readAll()
> --
>
> Key: BEAM-2656
> URL: https://issues.apache.org/jira/browse/BEAM-2656
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>
> TextIO.readAll() is nifty and performant when reading a large number of files.
> We should similarly have AvroIO.readAll(). Maybe other connectors too in the 
> future.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2656) Introduce AvroIO.readAll()

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096888#comment-16096888
 ] 

Ahmet Altay commented on BEAM-2656:
---

[~chamikara] Should we have python equivalent for {{readAll}}?

> Introduce AvroIO.readAll()
> --
>
> Key: BEAM-2656
> URL: https://issues.apache.org/jira/browse/BEAM-2656
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>
> TextIO.readAll() is nifty and performant when reading a large number of files.
> We should similarly have AvroIO.readAll(). Maybe other connectors too in the 
> future.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2656) Introduce AvroIO.readAll()

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096903#comment-16096903
 ] 

Chamikara Jayalath commented on BEAM-2656:
--

I created https://issues.apache.org/jira/browse/BEAM-2643 for Python textio. 
Avro should follow.

> Introduce AvroIO.readAll()
> --
>
> Key: BEAM-2656
> URL: https://issues.apache.org/jira/browse/BEAM-2656
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>
> TextIO.readAll() is nifty and performant when reading a large number of files.
> We should similarly have AvroIO.readAll(). Maybe other connectors too in the 
> future.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3628

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #128

[Apache Jenkins Server]

See 


Changes:

[klk] Uniquify application nodes in TextIOReadTest and only run pipeline once

[klk] Use dehydration-insensitive APIs in ParDoEvaluatorFactory

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 81c2e90f3a9d109121b686c25b642cc9f16ced43 (origin/master)
Commit message: "This closes #3608: Uniquify application nodes in 
TextIOReadTest and only run pipeline once"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 81c2e90f3a9d109121b686c25b642cc9f16ced43
 > git rev-list 8bd64759616a2f262f60ce2998b98f0dd1dd2000 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1968346508332482697.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4386779086412966982.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8808316721299664421.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): python-gflags==3.1.1 
in /home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
/usr/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins906149996873207305.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://

:66:
 UserWarning: You are using version 1.5.4 of pip. However, version 7.0.0 is 
recommended.
  _PIP_VERSION, REQUIRED_PIP_VERSION
no previously-included directories found matching 'doc/.build'

Installed 

[GitHub] beam pull request #3616: Unbreaks AvroSource compatibility with Dataflow wor...

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/3616

Unbreaks AvroSource compatibility with Dataflow worker

R: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam unbreak-dataflow

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3616.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3616


commit 4f7d0b4ed70b40443aede1ce28d0939f08b1d04f
Author: Eugene Kirpichov 
Date:   2017-07-22T01:01:14Z

Unbreaks AvroSource compatibility with Dataflow worker




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3462

[Apache Jenkins Server]

See 

[2/2] beam git commit: This closes #3616: Unbreaks AvroSource compatibility with Dataflow worker

[jkff]

This closes #3616: Unbreaks AvroSource compatibility with Dataflow worker


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f3987486
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f3987486
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f3987486

Branch: refs/heads/master
Commit: f39874860027832fdeff957afdc5cc8326de6d30
Parents: 81c2e90 4f7d0b4
Author: Eugene Kirpichov 
Authored: Fri Jul 21 18:23:48 2017 -0700
Committer: Eugene Kirpichov 
Committed: Fri Jul 21 18:23:48 2017 -0700

--
 .../main/java/org/apache/beam/sdk/io/AvroSource.java| 12 
 1 file changed, 12 insertions(+)
--

[GitHub] beam pull request #3616: Unbreaks AvroSource compatibility with Dataflow wor...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3616


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Unbreaks AvroSource compatibility with Dataflow worker

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master 81c2e90f3 -> f39874860


Unbreaks AvroSource compatibility with Dataflow worker


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4f7d0b4e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4f7d0b4e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4f7d0b4e

Branch: refs/heads/master
Commit: 4f7d0b4ed70b40443aede1ce28d0939f08b1d04f
Parents: 81c2e90
Author: Eugene Kirpichov 
Authored: Fri Jul 21 18:01:14 2017 -0700
Committer: Eugene Kirpichov 
Committed: Fri Jul 21 18:01:14 2017 -0700

--
 .../main/java/org/apache/beam/sdk/io/AvroSource.java| 12 
 1 file changed, 12 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/4f7d0b4e/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java
index 30af344..7cd97a8 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java
@@ -201,6 +201,18 @@ public class AvroSource extends BlockBasedSource {
 super.validate();
   }
 
+  /**
+   * Used by the Dataflow worker. Do not introduce new usages. Do not delete 
without confirming that
+   * Dataflow ValidatesRunner tests pass.
+   *
+   * @deprecated Used by Dataflow worker
+   */
+  @Deprecated
+  public BlockBasedSource createForSubrangeOfFile(String fileName, long 
start, long end)
+  throws IOException {
+return createForSubrangeOfFile(FileSystems.matchSingleFileSpec(fileName), 
start, end);
+  }
+
   @Override
   public BlockBasedSource createForSubrangeOfFile(Metadata fileMetadata, 
long start, long end) {
 return new AvroSource<>(fileMetadata, getMinBundleSize(), start, end, 
readerSchemaString, type);

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2683

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3463

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2684

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4432

[Apache Jenkins Server]

See 


Changes:

[kirpichov] Unbreaks AvroSource compatibility with Dataflow worker

--
[...truncated 1.07 MB...]
2017-07-22T02:13:03.229 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/github/rholder/snowball-stemmer/1.3.0.581.1/snowball-stemmer-1.3.0.581.1.jar
2017-07-22T02:13:03.295 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/rholder/snowball-stemmer/1.3.0.581.1/snowball-stemmer-1.3.0.581.1.jar
 (91 KB at 10.2 KB/sec)
2017-07-22T02:13:03.295 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/googlecode/concurrent-trees/concurrent-trees/2.4.0/concurrent-trees-2.4.0.jar
2017-07-22T02:13:03.356 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/googlecode/concurrent-trees/concurrent-trees/2.4.0/concurrent-trees-2.4.0.jar
 (116 KB at 12.9 KB/sec)
2017-07-22T02:13:03.356 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/github/jbellis/jamm/0.3.0/jamm-0.3.0.jar
2017-07-22T02:13:03.379 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/cassandra/cassandra-thrift/3.9/cassandra-thrift-3.9.jar
 (1858 KB at 206.4 KB/sec)
2017-07-22T02:13:03.379 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-all/4.0.39.Final/netty-all-4.0.39.Final.jar
2017-07-22T02:13:03.387 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/jbellis/jamm/0.3.0/jamm-0.3.0.jar
 (21 KB at 2.3 KB/sec)
2017-07-22T02:13:03.387 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/fusesource/sigar/1.6.4/sigar-1.6.4.jar
2017-07-22T02:13:03.519 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/fusesource/sigar/1.6.4/sigar-1.6.4.jar 
(419 KB at 45.8 KB/sec)
2017-07-22T02:13:03.519 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/jdt/core/compiler/ecj/4.4.2/ecj-4.4.2.jar
2017-07-22T02:13:03.534 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/cassandra/cassandra-all/3.9/cassandra-all-3.9.jar
 (5923 KB at 647.0 KB/sec)
2017-07-22T02:13:03.534 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/caffinitas/ohc/ohc-core/0.4.3/ohc-core-0.4.3.jar
2017-07-22T02:13:03.596 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/caffinitas/ohc/ohc-core/0.4.3/ohc-core-0.4.3.jar
 (125 KB at 13.5 KB/sec)
2017-07-22T02:13:03.596 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/github/ben-manes/caffeine/caffeine/2.2.6/caffeine-2.2.6.jar
2017-07-22T02:13:03.792 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-all/4.0.39.Final/netty-all-4.0.39.Final.jar
 (2219 KB at 235.7 KB/sec)
2017-07-22T02:13:03.792 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/datastax/cassandra/cassandra-driver-core/3.1.1/cassandra-driver-core-3.1.1.jar
2017-07-22T02:13:03.899 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/ben-manes/caffeine/caffeine/2.2.6/caffeine-2.2.6.jar
 (926 KB at 97.2 KB/sec)
2017-07-22T02:13:03.996 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/datastax/cassandra/cassandra-driver-core/3.1.1/cassandra-driver-core-3.1.1.jar
 (1029 KB at 106.9 KB/sec)
2017-07-22T02:13:04.049 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/jdt/core/compiler/ecj/4.4.2/ecj-4.4.2.jar
 (2257 KB at 233.3 KB/sec)
2017-07-22T02:13:04.583 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/storm/storm-core/1.0.1/storm-core-1.0.1.jar
 (19650 KB at 1925.5 KB/sec)
2017-07-22T02:13:04.772 [INFO] Downloading: 
http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
2017-07-22T02:13:04.772 [INFO] Downloading: 
http://conjars.org/repo/cascading/cascading-hadoop/2.6.3/cascading-hadoop-2.6.3.jar
2017-07-22T02:13:04.773 [INFO] Downloading: 
http://conjars.org/repo/cascading/cascading-core/2.6.3/cascading-core-2.6.3.jar
2017-07-22T02:13:04.774 [INFO] Downloading: 
http://conjars.org/repo/riffle/riffle/0.1-dev/riffle-0.1-dev.jar
2017-07-22T02:13:04.779 [INFO] Downloading: 
http://conjars.org/repo/thirdparty/jgrapht-jdk1.6/0.8.1/jgrapht-jdk1.6-0.8.1.jar
2017-07-22T02:13:04.894 [INFO] Downloaded: 
http://conjars.org/repo/riffle/riffle/0.1-dev/riffle-0.1-dev.jar (12 KB at 91.6 
KB/sec)
2017-07-22T02:13:04.894 [INFO] Downloading: 
http://conjars.org/repo/cascading/cascading-local/2.6.3/cascading-local-2.6.3.jar
2017-07-22T02:13:04.949 [INFO] Downloaded: 
http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
 (48 KB at 267.9 KB/sec)
2017-07-22T02:13:05.066 [INFO] Downloaded: 
http://conjars.org/repo/cascading/cascading-local/2.6.3/cascading-local-2.6.3.jar
 (43 KB at 145.5 KB/sec)
2017-07-22T02:13:05.111 [INFO] Downloaded: 
http://conjars.org/repo/thirdparty/jgrapht-jdk1.6/0.8.1/jgrapht-jdk1.6-0.8.1.jar
 (230 KB at 680.3 KB/sec)

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4431

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3629

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #3630

[Apache Jenkins Server]

See 


--
GitHub pull request #3616 of commit 4f7d0b4ed70b40443aede1ce28d0939f08b1d04f, 
no merge conflicts.
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/3616/*:refs/remotes/origin/pr/3616/*
 > git rev-parse refs/remotes/origin/pr/3616/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/3616/merge^{commit} # timeout=10
 > git rev-parse origin/pr/3616/merge^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/3616/*:refs/remotes/origin/pr/3616/*
 > git rev-parse refs/remotes/origin/pr/3616/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/3616/merge^{commit} # timeout=10
 > git rev-parse origin/pr/3616/merge^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/3616/*:refs/remotes/origin/pr/3616/*
 > git rev-parse refs/remotes/origin/pr/3616/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/3616/merge^{commit} # timeout=10
 > git rev-parse origin/pr/3616/merge^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Not sending mail to unregistered user sourabhba...@google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user fi...@google.com
Not sending mail to unregistered user k...@google.com

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3631

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-2657) Create Solr IO

[Cao Manh Dat (JIRA)]

Cao Manh Dat created BEAM-2657:
--

 Summary: Create Solr IO
 Key: BEAM-2657
 URL: https://issues.apache.org/jira/browse/BEAM-2657
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-extensions
Reporter: Cao Manh Dat
Assignee: Davor Bonaci


I'm working on a new SolrIO ( this components borrow som design's idea from 
ElasticsearchIO ) providing both bounded source and sink.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2658) SerializableCoder has high precedence over ProtoCoder in CoderRegistry#getCoder

[Neville Li (JIRA)]

Neville Li created BEAM-2658:


 Summary: SerializableCoder has high precedence over ProtoCoder in 
CoderRegistry#getCoder
 Key: BEAM-2658
 URL: https://issues.apache.org/jira/browse/BEAM-2658
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Affects Versions: 2.0.0
Reporter: Neville Li
Assignee: Davor Bonaci
Priority: Minor


{{
public class CoderTest {
  public static void main(String[] args) throws CannotProvideCoderException {
PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
Pipeline pipeline = Pipeline.create(options);

Coder coder = 
pipeline.getCoderRegistry().getCoder(Timestamp.class);

// class org.apache.beam.sdk.coders.SerializableCoder
System.out.println(coder.getClass());
  }
}
}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2658) SerializableCoder has higher precedence over ProtoCoder in CoderRegistry#getCoder

[Neville Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neville Li updated BEAM-2658:
-
Summary: SerializableCoder has higher precedence over ProtoCoder in 
CoderRegistry#getCoder  (was: SerializableCoder has high precedence over 
ProtoCoder in CoderRegistry#getCoder)

> SerializableCoder has higher precedence over ProtoCoder in 
> CoderRegistry#getCoder
> -
>
> Key: BEAM-2658
> URL: https://issues.apache.org/jira/browse/BEAM-2658
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.0.0
>Reporter: Neville Li
>Assignee: Davor Bonaci
>Priority: Minor
>
> {{
> public class CoderTest {
>   public static void main(String[] args) throws CannotProvideCoderException {
> PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
> Pipeline pipeline = Pipeline.create(options);
> Coder coder = 
> pipeline.getCoderRegistry().getCoder(Timestamp.class);
> // class org.apache.beam.sdk.coders.SerializableCoder
> System.out.println(coder.getClass());
>   }
> }
> }}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)