Build failed in Jenkins: beam_PerformanceTests_Python #528

2017-11-05 Thread Apache Jenkins Server 
See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision e25d217f7f5272f7ec54a85dbe67f536a09eb460 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e25d217f7f5272f7ec54a85dbe67f536a09eb460
Commit message: "This closes #3991: [BEAM-2728] Extension for sketch-based 
statistics : HyperLogLog"
 > git rev-list e25d217f7f5272f7ec54a85dbe67f536a09eb460 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins266394331273480546.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2520429692518089347.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins43557084066078678.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25

Build failed in Jenkins: beam_PostCommit_Python_Verify #3485

2017-11-05 Thread Apache Jenkins Server 
See 


--
[...truncated 930.03 KB...]
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apach

Build failed in Jenkins: beam_PerformanceTests_Python #527

2017-11-05 Thread Apache Jenkins Server 
See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision e25d217f7f5272f7ec54a85dbe67f536a09eb460 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e25d217f7f5272f7ec54a85dbe67f536a09eb460
Commit message: "This closes #3991: [BEAM-2728] Extension for sketch-based 
statistics : HyperLogLog"
 > git rev-list e25d217f7f5272f7ec54a85dbe67f536a09eb460 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6410543803011308526.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2662725206511971022.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2591511972363358182.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (li

[jira] [Assigned] (BEAM-2271) Release guide or pom.xml needs update to avoid releasing Python binary artifacts

2017-11-05 Thread Sourabh Bajaj (JIRA) 

 [ 
https://issues.apache.org/jira/browse/BEAM-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj reassigned BEAM-2271:
---

Assignee: Ahmet Altay  (was: Sourabh Bajaj)

> Release guide or pom.xml needs update to avoid releasing Python binary 
> artifacts
> 
>
> Key: BEAM-2271
> URL: https://issues.apache.org/jira/browse/BEAM-2271
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Daniel Halperin
>Assignee: Ahmet Altay
> Fix For: 2.3.0
>
>
> The following directories (and children) were discovered in 2.0.0-RC2 and 
> were present in 0.6.0.
> {code}
> sdks/python: build   dist.eggs   nose-1.3.7-py2.7.egg  (and child 
> contents)
> {code}
> Ideally, these artifacts, which are created during setup and testing, would 
> get created in the {{sdks/python/target/}} subfolder where they will 
> automatically get ignored. More info below.
> For 2.0.0, we will manually remove these files from the source release RC3+. 
> This should be fixed before the next release.
> Here is a list of other paths that get excluded, should they be useful.
> {code}
> 
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/).*${project.build.directory}.*]
> 
> 
>  
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?maven-eclipse\.xml]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.project]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.classpath]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.iws]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.idea(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?out(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.ipr]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.iml]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.settings(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.externalToolBuilders(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.deployables(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.wtpmodules(/.*)?]
> 
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?cobertura\.ser]
> 
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?pom\.xml\.releaseBackup]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?release\.properties]
>   
> {code}
> This list is stored inside of this jar, which you can find by tracking 
> maven-assembly-plugin from the root apache pom: 
> https://mvnrepository.com/artifact/org.apache.apache.resources/apache-source-release-assembly-descriptor/1.0.6
> http://svn.apache.org/repos/asf/maven/pom/tags/apache-18/pom.xml



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-1585) Ability to add new file systems to beamFS in the python sdk

2017-11-05 Thread Sourabh Bajaj (JIRA) 

 [ 
https://issues.apache.org/jira/browse/BEAM-1585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj closed BEAM-1585.
---

> Ability to add new file systems to beamFS in the python sdk
> ---
>
> Key: BEAM-1585
> URL: https://issues.apache.org/jira/browse/BEAM-1585
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
> Fix For: Not applicable
>
>
> BEAM-1441 implements the new BeamFileSystem in the python SDK but currently 
> lacks the ability to add user implemented file systems.
> This needs to be executed in the worker so should be packaged correctly with 
> the pipeline code. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4080: [BEAM-2500] Add S3 FileSystem to Java SDK

2017-11-05 Thread jacobmarble 
GitHub user jacobmarble opened a pull request:

https://github.com/apache/beam/pull/4080

[BEAM-2500] Add S3 FileSystem to Java SDK

My first contribution, will submit ICLA later today.

Ran `mvn clean verify` against sdks/java/io only, because I'm currently in 
a place with limited power and bandwidth.

There is some technical discussion in the JIRA.
https://issues.apache.org/jira/browse/BEAM-2500

Please notice that there is one persistent thread pool, and that copy 
operations can create a new thread pool per copy. Happy to hear feedback and 
discuss.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Kochava/beam s3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4080.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4080


commit b7964d1420f9f80da254960f332c077b65642667
Author: Jacob Marble 
Date:   2017-10-01T23:31:25Z

Add S3FileSystem to SDKs/Java/IO

This module was originally modeled after GcsFileSystem and related, but
has diverged quite a bit.

commit fc4cd7564de68b40d92c04a02657428fc62aee2e
Author: Jacob Marble 
Date:   2017-11-05T22:12:01Z

bump Beam version to 2.3.0-SNAPSHOT




---

svn commit: r22866 - /dev/beam/2.2.0/

2017-11-05 Thread reuvenlax 
Author: reuvenlax
Date: Sun Nov  5 22:21:59 2017
New Revision: 22866

Log:
source release for Beam 2.2.0 RC3

Modified:
dev/beam/2.2.0/apache-beam-2.2.0-python.zip
dev/beam/2.2.0/apache-beam-2.2.0-python.zip.asc
dev/beam/2.2.0/apache-beam-2.2.0-python.zip.md5
dev/beam/2.2.0/apache-beam-2.2.0-python.zip.sha1
dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip
dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.asc
dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.md5
dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.sha1

Modified: dev/beam/2.2.0/apache-beam-2.2.0-python.zip
==
Binary files - no diff available.

Modified: dev/beam/2.2.0/apache-beam-2.2.0-python.zip.asc
==
--- dev/beam/2.2.0/apache-beam-2.2.0-python.zip.asc (original)
+++ dev/beam/2.2.0/apache-beam-2.2.0-python.zip.asc Sun Nov  5 22:21:59 2017
@@ -1,16 +1,16 @@
 -BEGIN PGP SIGNATURE-
 
-iQIcBAABAgAGBQJZ+TY3AAoJEDvHLhu5i3cIfZcP/2vRV3c/ZNhgWgYWBTl03U+t
-traLwfaNu2NKveGBPeR4WoVndg0FHfzwMKD5sEkU9K1MK6CHGJ9dUVCNiMMOaLc5
-9zuft9kTxxoy9hfPptachdQiUlKSiIh9sJafMiFF7JPEcHEHN2ohajTeP7OjxNm4
-I03KXoQRSU+WoepQuas+TVojeriK5lRQw+BkWXkQ9HsoR9vzxGOqNI3vDlLzRfr/
-9sCiSjm5L/JUUALTjBn4kWj5v0iIN0rMfu7v60dsR1a7PsMGOhG7SR7lOTZrezsq
-oJd4WGzvcbxENaggU88VnKyO2M6HuvoZ0/lsz/hNEzEd1Ebvo3geA+AUIt7RO3cV
-pOdEIs+8zeSY4G5ISPpbcoHtp+ODgi3sDEcrD8dwAytPx9kQXpgVGqtLVhnOX4ru
-ZfMvUcB1ArecJQJ8wEs0hPXYeg35OhOFR2Xt2byc3w8Vw8tQ2bujNEP2cg5m+BDS
-GfeKOsT0tElwIGg0sJBO8qagx7/CWa71CDtM9gu+yYj6btmFlQJ2v+nRJrXhCS00
-zQT2zGSs9F7VaPfEt8qDhVHgEhQdJOF32bh7YgdZBXbLa9XaSruAmK7a+/dk5EuU
-qpVT/dEXyChzMS+f4XPw03DUgpO5Ays4QN3IbbTmHvyKxCBBFp6fL/xWrM/SIeKz
-XEsQ95wCj+FZkDSBd3qt
-=U4wJ
+iQIcBAABAgAGBQJZ/46iAAoJEDvHLhu5i3cIXe0QAJ0c9t8o7FHZ22Ifm1ZSHH5/
+MxIkebiBFXMPnr6kFOz9ZQDzmRWJKe0z0sVhvqLr6y5KAOFYx49DkpdVwacdeMCq
+qSCGQ2GdxSGXowUJ5DEv3/WxUg0/Ofr6OjH2Wge2CUkeLONGwHPD8fX0owpuhLEV
+t4YxXJI4imu/Icx3YdrMKAMbSEjmiyAV22jBdv2ZQmxW5RHPpZPYsy11gUFhwN4Z
+cSgywAseYe2315B/O6LgYp+0MTiwkxx5hGdb+bsnWF5QBRDMwn+uSd6c2OC/3hK2
+8ZUEsQW/HiMv5IL1M4kYhtkROcsg63aCzRRo0y6O5PZ5Bhra53nTWnxIUAD50mE8
+vC9g8slNIgts9inbNZ8S9z7vxbyAWDdqQbvOa46LDJjByL5vCR3M5HV+jKNcnzEg
+fRhE1BwThju5tBzcslg8m1xojJz9MWWHKVX2ElrM2VdDGQXK+sUyvSh1KlLW5f9e
+tpcc+gOYbzBqomRqMu4KIyCBkHC+nYG7T1YuKCGOF9gJj34exdFgsRZiXFtR9so2
+zhzvzoEsy0ZiuhdE493rGq7KSugAGM7wP9q9zYWuPrYGJ45RHK/t38kiKN8h2ZT5
+Nn86hxADe+fg38BddvLbtI0ZUoA/cNLl6NdfJzNulpJuIuaaAR4dNuoNDa3mQDcj
+AjYKpTdk/N66kPyLR81t
+=FrB3
 -END PGP SIGNATURE-

Modified: dev/beam/2.2.0/apache-beam-2.2.0-python.zip.md5
==
--- dev/beam/2.2.0/apache-beam-2.2.0-python.zip.md5 (original)
+++ dev/beam/2.2.0/apache-beam-2.2.0-python.zip.md5 Sun Nov  5 22:21:59 2017
@@ -1 +1 @@
-0969c81dd953d2ac8a8a3326931e54b1  apache-beam-2.2.0-python.zip
+d58f4adf0052ed0d08836ce407df5318  apache-beam-2.2.0-python.zip

Modified: dev/beam/2.2.0/apache-beam-2.2.0-python.zip.sha1
==
--- dev/beam/2.2.0/apache-beam-2.2.0-python.zip.sha1 (original)
+++ dev/beam/2.2.0/apache-beam-2.2.0-python.zip.sha1 Sun Nov  5 22:21:59 2017
@@ -1 +1 @@
-8b1734238c35fb3b08d2355eae4943f2bc00a5d1  apache-beam-2.2.0-python.zip
+5e7e2b3c9aaacc00148f8e002d447a804620fd57  apache-beam-2.2.0-python.zip

Modified: dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip
==
Binary files - no diff available.

Modified: dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.asc
==
--- dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.asc (original)
+++ dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.asc Sun Nov  5 22:21:59 
2017
@@ -1,16 +1,16 @@
 -BEGIN PGP SIGNATURE-
 
-iQIzBAABCAAdFiEEuyC2JJPY3kqDVd0KO8cuG7mLdwgFAln5FC8ACgkQO8cuG7mL
-dwj/URAAiQlaVTCROZoZlxkZ2CWlNLp0ev43WEnHwRivVV0g8r5kZXGQ62u+83y9
-bB8AHMEOJqbKsAq9cmVqQiwaCov/bj5slozzQ9i/jxJcfDYHVlAYQtURkRXOrK73
-MwKBQPP7UJi1IluFRWxHakg26gUlDWW2E5jzNS9tlcBw1UaqP/3dbdV4T1f2IWCA
-zS8AA2ROIOdzyPpi7f4QgedCoHEX6UosbHDhQDAUtZldQYPNeU+XrGyzs4NlW4s3
-2zsgehMFIofGRA0cQgOAqMeufxIyI0Ux5mHVANa5ufqgCEklDRIfmLjT6GuO8Qu2
-s2vUR/9pEt88z/twJEMOYEv52xORPhgjVOis6JbeTujHULxX+1B7+ptB6WWaJsXa
-1qA1htYtMCev6rjw3SxglcZ3fxjzKrB/WOSD5kfab2zVQ3SY+CW01tTGtDnWJCGr
-jwXDDmPUQkFN0gnsTSIFw7YCvqRnIihU/aIa38d9XFxoT3MVkOMXLUUgYmNNDWLz
-+zvBhncRms0Ev7h8vPeeHQ1F0gc9ghNbnPdLnpd5aHXsbv9XFQpodsBaq3efqA77
-10gUlg+B4pcyiFj0aOM9w8A6mm0RdU8EdBmWGNAHIjeKQ7sKCtXj1T5BdSLReQHy
-UPj0BRtvcpXyVRoAismSQfUMnlSqyFbFgnjE1BarhNQLmWdOUag=
-=xX97
+iQIzBAABCAAdFiEEuyC2JJPY3kqDVd0KO8cuG7mLdwgFAln+zacACgkQO8cuG7mL
+dwhL7Q/9GAyWrW1ylfsFmujMXygiIUp6uolMfhtvl5tN4bG4/TREFQHVht9rk6sw
+AhVH6ivHT0M8g2jC9rug0mzGxPa6DYj5OozhEbnGJqgigdIoht+7OBY/OwdjXf2I
+lqke+s1gX4AlFFGJFF++VhYajPk4fn

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

2017-11-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16239745#comment-16239745
 ] 

ASF GitHub Bot commented on BEAM-2500:
--

GitHub user jacobmarble opened a pull request:

https://github.com/apache/beam/pull/4080

[BEAM-2500] Add S3 FileSystem to Java SDK

My first contribution, will submit ICLA later today.

Ran `mvn clean verify` against sdks/java/io only, because I'm currently in 
a place with limited power and bandwidth.

There is some technical discussion in the JIRA.
https://issues.apache.org/jira/browse/BEAM-2500

Please notice that there is one persistent thread pool, and that copy 
operations can create a new thread pool per copy. Happy to hear feedback and 
discuss.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Kochava/beam s3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4080.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4080


commit b7964d1420f9f80da254960f332c077b65642667
Author: Jacob Marble 
Date:   2017-10-01T23:31:25Z

Add S3FileSystem to SDKs/Java/IO

This module was originally modeled after GcsFileSystem and related, but
has diverged quite a bit.

commit fc4cd7564de68b40d92c04a02657428fc62aee2e
Author: Jacob Marble 
Date:   2017-11-05T22:12:01Z

bump Beam version to 2.3.0-SNAPSHOT




> Add support for S3 as a Apache Beam FileSystem
> --
>
> Key: BEAM-2500
> URL: https://issues.apache.org/jira/browse/BEAM-2500
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Luke Cwik
>Assignee: Jacob Marble
>Priority: Minor
> Attachments: hadoop_fs_patch.patch
>
>
> Note that this is for providing direct integration with S3 as an Apache Beam 
> FileSystem.
> There is already support for using the Hadoop S3 connector by depending on 
> the Hadoop File System module[1], configuring HadoopFileSystemOptions[2] with 
> a S3 configuration[3].
> 1: https://github.com/apache/beam/tree/master/sdks/java/io/hadoop-file-system
> 2: 
> https://github.com/apache/beam/blob/master/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.java#L53
> 3: https://wiki.apache.org/hadoop/AmazonS3



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3484

2017-11-05 Thread Apache Jenkins Server 
See 


--
[...truncated 933.59 KB...]
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apach

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

2017-11-05 Thread Jacob Marble (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16239725#comment-16239725
 ] 

Jacob Marble commented on BEAM-2500:


Thanks, Ismael. I have updated my fork and will submit a PR in a few minutes.

> Add support for S3 as a Apache Beam FileSystem
> --
>
> Key: BEAM-2500
> URL: https://issues.apache.org/jira/browse/BEAM-2500
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Luke Cwik
>Assignee: Jacob Marble
>Priority: Minor
> Attachments: hadoop_fs_patch.patch
>
>
> Note that this is for providing direct integration with S3 as an Apache Beam 
> FileSystem.
> There is already support for using the Hadoop S3 connector by depending on 
> the Hadoop File System module[1], configuring HadoopFileSystemOptions[2] with 
> a S3 configuration[3].
> 1: https://github.com/apache/beam/tree/master/sdks/java/io/hadoop-file-system
> 2: 
> https://github.com/apache/beam/blob/master/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.java#L53
> 3: https://wiki.apache.org/hadoop/AmazonS3



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PerformanceTests_Python #526

2017-11-05 Thread Apache Jenkins Server 
See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision e25d217f7f5272f7ec54a85dbe67f536a09eb460 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e25d217f7f5272f7ec54a85dbe67f536a09eb460
Commit message: "This closes #3991: [BEAM-2728] Extension for sketch-based 
statistics : HyperLogLog"
 > git rev-list e25d217f7f5272f7ec54a85dbe67f536a09eb460 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3973711214972929670.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7236186333708723818.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2413211655219762434.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (li

beam git commit: [maven-release-plugin] rollback changes from release preparation of v2.2.0-RC3

2017-11-05 Thread reuvenlax 
Repository: beam
Updated Branches:
  refs/heads/release-2.2.0 b75fe358e -> 36ae989af


[maven-release-plugin] rollback changes from release preparation of v2.2.0-RC3


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/36ae989a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/36ae989a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/36ae989a

Branch: refs/heads/release-2.2.0
Commit: 36ae989afe8d6c9a2d0e0e4e6b03ca05f72640b6
Parents: b75fe35
Author: Reuven Lax 
Authored: Sun Nov 5 08:51:50 2017 -0800
Committer: Reuven Lax 
Committed: Sun Nov 5 08:51:50 2017 -0800

--
 examples/java/pom.xml| 2 +-
 examples/java8/pom.xml   | 2 +-
 examples/pom.xml | 2 +-
 pom.xml  | 4 ++--
 runners/apex/pom.xml | 2 +-
 runners/core-construction-java/pom.xml   | 2 +-
 runners/core-java/pom.xml| 2 +-
 runners/direct-java/pom.xml  | 2 +-
 runners/flink/pom.xml| 2 +-
 runners/gcp/gcemd/pom.xml| 2 +-
 runners/gcp/gcsproxy/pom.xml | 2 +-
 runners/gcp/pom.xml  | 2 +-
 runners/gearpump/pom.xml | 2 +-
 runners/google-cloud-dataflow-java/pom.xml   | 2 +-
 runners/local-artifact-service-java/pom.xml  | 2 +-
 runners/pom.xml  | 2 +-
 runners/reference/pom.xml| 2 +-
 runners/spark/pom.xml| 2 +-
 sdks/common/fn-api/pom.xml   | 2 +-
 sdks/common/pom.xml  | 2 +-
 sdks/common/runner-api/pom.xml   | 2 +-
 sdks/go/pom.xml  | 2 +-
 sdks/java/build-tools/pom.xml| 2 +-
 sdks/java/container/pom.xml  | 2 +-
 sdks/java/core/pom.xml   | 2 +-
 sdks/java/extensions/google-cloud-platform-core/pom.xml  | 2 +-
 sdks/java/extensions/jackson/pom.xml | 2 +-
 sdks/java/extensions/join-library/pom.xml| 2 +-
 sdks/java/extensions/pom.xml | 2 +-
 sdks/java/extensions/protobuf/pom.xml| 2 +-
 sdks/java/extensions/sorter/pom.xml  | 2 +-
 sdks/java/extensions/sql/pom.xml | 2 +-
 sdks/java/harness/pom.xml| 2 +-
 sdks/java/io/amqp/pom.xml| 2 +-
 sdks/java/io/cassandra/pom.xml   | 2 +-
 sdks/java/io/common/pom.xml  | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/pom.xml   | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/pom.xml   | 2 +-
 .../io/elasticsearch-tests/elasticsearch-tests-common/pom.xml| 2 +-
 sdks/java/io/elasticsearch-tests/pom.xml | 2 +-
 sdks/java/io/elasticsearch/pom.xml   | 2 +-
 sdks/java/io/google-cloud-platform/pom.xml   | 2 +-
 sdks/java/io/hadoop-common/pom.xml   | 2 +-
 sdks/java/io/hadoop-file-system/pom.xml  | 2 +-
 sdks/java/io/hadoop/input-format/pom.xml | 2 +-
 sdks/java/io/hadoop/jdk1.8-tests/pom.xml | 2 +-
 sdks/java/io/hadoop/pom.xml  | 2 +-
 sdks/java/io/hbase/pom.xml   | 2 +-
 sdks/java/io/hcatalog/pom.xml| 2 +-
 sdks/java/io/jdbc/pom.xml| 2 +-
 sdks/java/io/jms/pom.xml | 2 +-
 sdks/java/io/kafka/pom.xml   | 2 +-
 sdks/java/io/kinesis/pom.xml | 2 +-
 sdks/java/io/mongodb/pom.xml | 2 +-
 sdks/java/io/mqtt/pom.xml| 2 +-
 sdks/java/io/pom.xml | 2 +-
 sdks/java/io/redis/pom.xml   | 2 +-
 sdks/java/io/solr/pom.xml| 2 +-
 sdks/java/io/tika/pom.xml

[beam] Git Push Summary

2017-11-05 Thread reuvenlax 
Repository: beam
Updated Tags:  refs/tags/v2.2.0-RC3 [created] 8567db27d

beam git commit: [maven-release-plugin] prepare release v2.2.0-RC3

2017-11-05 Thread reuvenlax 
Repository: beam
Updated Branches:
  refs/heads/release-2.2.0 2d254b183 -> b75fe358e


[maven-release-plugin] prepare release v2.2.0-RC3


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b75fe358
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b75fe358
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b75fe358

Branch: refs/heads/release-2.2.0
Commit: b75fe358e257b8f2bf096698a4336c1d32aa44eb
Parents: 2d254b1
Author: Reuven Lax 
Authored: Sun Nov 5 01:02:16 2017 -0800
Committer: Reuven Lax 
Committed: Sun Nov 5 01:02:16 2017 -0800

--
 examples/java/pom.xml| 2 +-
 examples/java8/pom.xml   | 2 +-
 examples/pom.xml | 2 +-
 pom.xml  | 4 ++--
 runners/apex/pom.xml | 2 +-
 runners/core-construction-java/pom.xml   | 2 +-
 runners/core-java/pom.xml| 2 +-
 runners/direct-java/pom.xml  | 2 +-
 runners/flink/pom.xml| 2 +-
 runners/gcp/gcemd/pom.xml| 2 +-
 runners/gcp/gcsproxy/pom.xml | 2 +-
 runners/gcp/pom.xml  | 2 +-
 runners/gearpump/pom.xml | 2 +-
 runners/google-cloud-dataflow-java/pom.xml   | 2 +-
 runners/local-artifact-service-java/pom.xml  | 2 +-
 runners/pom.xml  | 2 +-
 runners/reference/pom.xml| 2 +-
 runners/spark/pom.xml| 2 +-
 sdks/common/fn-api/pom.xml   | 2 +-
 sdks/common/pom.xml  | 2 +-
 sdks/common/runner-api/pom.xml   | 2 +-
 sdks/go/pom.xml  | 2 +-
 sdks/java/build-tools/pom.xml| 2 +-
 sdks/java/container/pom.xml  | 2 +-
 sdks/java/core/pom.xml   | 2 +-
 sdks/java/extensions/google-cloud-platform-core/pom.xml  | 2 +-
 sdks/java/extensions/jackson/pom.xml | 2 +-
 sdks/java/extensions/join-library/pom.xml| 2 +-
 sdks/java/extensions/pom.xml | 2 +-
 sdks/java/extensions/protobuf/pom.xml| 2 +-
 sdks/java/extensions/sorter/pom.xml  | 2 +-
 sdks/java/extensions/sql/pom.xml | 2 +-
 sdks/java/harness/pom.xml| 2 +-
 sdks/java/io/amqp/pom.xml| 2 +-
 sdks/java/io/cassandra/pom.xml   | 2 +-
 sdks/java/io/common/pom.xml  | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/pom.xml   | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/pom.xml   | 2 +-
 .../io/elasticsearch-tests/elasticsearch-tests-common/pom.xml| 2 +-
 sdks/java/io/elasticsearch-tests/pom.xml | 2 +-
 sdks/java/io/elasticsearch/pom.xml   | 2 +-
 sdks/java/io/google-cloud-platform/pom.xml   | 2 +-
 sdks/java/io/hadoop-common/pom.xml   | 2 +-
 sdks/java/io/hadoop-file-system/pom.xml  | 2 +-
 sdks/java/io/hadoop/input-format/pom.xml | 2 +-
 sdks/java/io/hadoop/jdk1.8-tests/pom.xml | 2 +-
 sdks/java/io/hadoop/pom.xml  | 2 +-
 sdks/java/io/hbase/pom.xml   | 2 +-
 sdks/java/io/hcatalog/pom.xml| 2 +-
 sdks/java/io/jdbc/pom.xml| 2 +-
 sdks/java/io/jms/pom.xml | 2 +-
 sdks/java/io/kafka/pom.xml   | 2 +-
 sdks/java/io/kinesis/pom.xml | 2 +-
 sdks/java/io/mongodb/pom.xml | 2 +-
 sdks/java/io/mqtt/pom.xml| 2 +-
 sdks/java/io/pom.xml | 2 +-
 sdks/java/io/redis/pom.xml   | 2 +-
 sdks/java/io/solr/pom.xml| 2 +-
 sdks/java/io/tika/pom.xml| 2 +-

Build failed in Jenkins: beam_PostCommit_Python_Verify #3483

2017-11-05 Thread Apache Jenkins Server 
See 


--
[...truncated 931.35 KB...]
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apach

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #4286

2017-11-05 Thread Apache Jenkins Server 
See

Build failed in Jenkins: beam_PerformanceTests_Python #525

2017-11-05 Thread Apache Jenkins Server 
See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision e25d217f7f5272f7ec54a85dbe67f536a09eb460 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e25d217f7f5272f7ec54a85dbe67f536a09eb460
Commit message: "This closes #3991: [BEAM-2728] Extension for sketch-based 
statistics : HyperLogLog"
 > git rev-list e25d217f7f5272f7ec54a85dbe67f536a09eb460 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2891641081300988035.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins601963662247766001.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins782708712307695106.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 2

Build failed in Jenkins: beam_PostCommit_Python_Verify #3482

2017-11-05 Thread Apache Jenkins Server 
See 


--
[...truncated 929.44 KB...]
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apach

[jira] [Assigned] (BEAM-3143) Fix type inference in Python 3 for generators

2017-11-05 Thread Ahmet Altay (JIRA) 

 [ 
https://issues.apache.org/jira/browse/BEAM-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reassigned BEAM-3143:
-

Assignee: (was: Ahmet Altay)

> Fix type inference in Python 3 for generators
> -
>
> Key: BEAM-3143
> URL: https://issues.apache.org/jira/browse/BEAM-3143
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: holdenk
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3143) Fix type inference in Python 3 for generators

2017-11-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16239430#comment-16239430
 ] 

ASF GitHub Bot commented on BEAM-3143:
--

GitHub user holdenk opened a pull request:

https://github.com/apache/beam/pull/4079

[BEAM-3143][WIP] Py3 type inferance

Initial work attempt to do type inference correctly in Python 3. Currently 
this doesn't work for for comprehension in Python 3.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/holdenk/beam BEAM-3143-py3-type-inferance

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4079.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4079


commit eac2b826a127e80617b09d761e31750b0f383297
Author: Holden Karau 
Date:   2017-11-05T04:28:46Z

Add a test for lists, switch div in trivial_inference

commit 3b63c6e87e26519b085161a2bf2fbcf31183e56c
Author: Holden Karau 
Date:   2017-08-21T08:18:57Z

Try and fix some of the type inferance, list generators are kind of not 
going to work super well as is.




> Fix type inference in Python 3 for generators
> -
>
> Key: BEAM-3143
> URL: https://issues.apache.org/jira/browse/BEAM-3143
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: holdenk
>Assignee: Ahmet Altay
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4079: [BEAM-3143][WIP] Py3 type inferance

2017-11-05 Thread holdenk 
GitHub user holdenk opened a pull request:

https://github.com/apache/beam/pull/4079

[BEAM-3143][WIP] Py3 type inferance

Initial work attempt to do type inference correctly in Python 3. Currently 
this doesn't work for for comprehension in Python 3.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/holdenk/beam BEAM-3143-py3-type-inferance

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4079.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4079


commit eac2b826a127e80617b09d761e31750b0f383297
Author: Holden Karau 
Date:   2017-11-05T04:28:46Z

Add a test for lists, switch div in trivial_inference

commit 3b63c6e87e26519b085161a2bf2fbcf31183e56c
Author: Holden Karau 
Date:   2017-08-21T08:18:57Z

Try and fix some of the type inferance, list generators are kind of not 
going to work super well as is.




---

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4285

2017-11-05 Thread Apache Jenkins Server 
See

[jira] [Commented] (BEAM-3141) Make coders & streams work in Python 3

2017-11-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-3141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16239426#comment-16239426
 ] 

ASF GitHub Bot commented on BEAM-3141:
--

GitHub user holdenk opened a pull request:

https://github.com/apache/beam/pull/4078

[BEAM-3141][WIP] Support the coders in Python 3

This adds support for the coders in Python 3 as well as fixes some other 
issues that were blocking `GameStatsTest`/

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/holdenk/beam BEAM-3141-coder-py3-support-r2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4078.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4078


commit f3dd6257e3765bf9117207fa8173027f92b64c60
Author: Holden Karau 
Date:   2017-11-03T18:18:26Z

Fix unicode imports and NoneType

commit 6f7f8aa6ea9b42f7ed39af68ccd806bc16173b1c
Author: Holden Karau 
Date:   2017-11-03T18:31:06Z

Swap the avroio over to BytesIO

commit 32f39f1e9023faea87a23c4bf9bd31897d9b2f8f
Author: Holden Karau 
Date:   2017-11-03T18:31:24Z

Swithc apiclinet to io StringIO from StringIO StringIO

commit 630d40d74eec5e3af693e5ab0fba3e4b3163ba62
Author: Holden Karau 
Date:   2017-11-03T18:39:31Z

Switch required packages based on Python version (currently only impacting 
avro) and add future geq 0.16.0

commit 9bd4e7f20770d6276ea16b58dfdb041210ae1a15
Author: Holden Karau 
Date:   2017-11-03T23:07:27Z

Rewrite lambda with expansion

commit f58668416f3f451166dbe644ab56581c4902d6b6
Author: Holden Karau 
Date:   2017-11-03T23:07:42Z

Fix items and builtin import and a long import

commit da9ebb7667df4992512177d1d0e8b9ce4cafc62e
Author: Holden Karau 
Date:   2017-08-20T21:11:07Z

Rich comparisions requirement

commit dfc4e9057acb417e261a5e59399e5e943861cc3d
Author: Holden Karau 
Date:   2017-08-20T21:42:33Z

Add __hash__ methods in places where needed (in progress)

commit ca1f224d080ea37fd0de3582e9a7a5847f476f34
Author: Holden Karau 
Date:   2017-08-21T09:03:52Z

Provide eq/hashing/sorting for TimestampedValue

commit ea502b5a95b42ded289a97bc59a3cfbd2b9e05d8
Author: Holden Karau 
Date:   2017-08-21T18:44:37Z

If we are passed in bytes for the lookup in pipeline_context do the lookup 
as a string

commit fee6a65eaef0b85386fa42cd02ef45da8e370d51
Author: Holden Karau 
Date:   2017-08-21T19:08:44Z

Provide a base ne using eq for windows

commit 099e115f190d86352cc9b01a0b4feb48b3442192
Author: Holden Karau 
Date:   2017-11-03T23:11:29Z

Switch filesystem to io.StringIO

commit b0c7cda8626b90a7bd260077f54da5c07ba2b3c4
Author: Holden Karau 
Date:   2017-11-03T23:47:13Z

Fix cirular dep introduced through Py3 import changes for 
CreatePTransformOverride.

commit b0af9cca420d97b9037be91ddba9719a01a272ad
Author: Holden Karau 
Date:   2017-11-04T00:00:34Z

Change slow_stream to bytes

commit 5890721770aca710d5f8f5194ade7a9f2a19fbc5
Author: Holden Karau 
Date:   2017-11-04T04:40:17Z

Handle slow_stream in py2 again

commit bc30732264463b12a49f445e0f369e361e5fb823
Author: Holden Karau 
Date:   2017-11-04T04:58:44Z

Selective fixes while running through 
apache_beam.examples.complete.game.game_stats_test.GameStatsTest, fix some 
rethrows. lambda expansion, imports, and metaclass

commit b2424e790e01d1a758458f052f37648850719705
Author: Holden Karau 
Date:   2017-11-04T05:00:14Z

metaclass rewrite rule

commit fb63800079b3b44e4b6ec59844c20813b1ba449a
Author: Holden Karau 
Date:   2017-11-04T06:36:00Z

Apply futurize on gen_protos

commit 767c1cc30d32edf21edf1916a47a8f5c779512fd
Author: Holden Karau 
Date:   2017-11-04T07:16:12Z

queue and raise rewrites

commit e8e40ac46097fd3b4ecc7f40383031755a0b265f
Author: Holden Karau 
Date:   2017-11-04T07:18:10Z

futurize -wn -f libfuturize.fixes.fix_future_standard_library_urllib 
./apache_beam

commit fe278de3039ceaae7af819cb119bb31f25d679fa
Author: Holden Karau 
Date:   2017-11-04T08:04:59Z

Finish up getting 
apache_beam.examples.complete.game.game_stats_test.GameStatsTest to run, fix 
serialized values map over, switch some exception re-throws, iteritems, and 
handle peaking at the eval context.

commit f2c78e6f0b66b11292035eac224914dc439d175c
Author: Holden Karau 
Date:   2017-11-04T08:21:04Z

Fix the write_byte on slow stream to just use bytes rather than encode

commit 8dbf4e72cdb5a4bf6a4280f472acca85ef3c991c
Author: Holden Karau 
Date:   2017-11-04T08:25:14Z

Fix some bytes coders tests to use bytes in py3

commit d43b70419c883ad30183893d967456a78595f11d
Author: Holden Karau 
Date:   2017-11-04T08:25:38Z

Change the fall through for encoding a bit and update past imports

commit 5c102b70cce3ad49c635d4ea52432356ddf00f50
Author: Holden Kara

[GitHub] beam pull request #4078: [BEAM-3141][WIP] Support the coders in Python 3

2017-11-05 Thread holdenk 
GitHub user holdenk opened a pull request:

https://github.com/apache/beam/pull/4078

[BEAM-3141][WIP] Support the coders in Python 3

This adds support for the coders in Python 3 as well as fixes some other 
issues that were blocking `GameStatsTest`/

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/holdenk/beam BEAM-3141-coder-py3-support-r2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4078.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4078


commit f3dd6257e3765bf9117207fa8173027f92b64c60
Author: Holden Karau 
Date:   2017-11-03T18:18:26Z

Fix unicode imports and NoneType

commit 6f7f8aa6ea9b42f7ed39af68ccd806bc16173b1c
Author: Holden Karau 
Date:   2017-11-03T18:31:06Z

Swap the avroio over to BytesIO

commit 32f39f1e9023faea87a23c4bf9bd31897d9b2f8f
Author: Holden Karau 
Date:   2017-11-03T18:31:24Z

Swithc apiclinet to io StringIO from StringIO StringIO

commit 630d40d74eec5e3af693e5ab0fba3e4b3163ba62
Author: Holden Karau 
Date:   2017-11-03T18:39:31Z

Switch required packages based on Python version (currently only impacting 
avro) and add future geq 0.16.0

commit 9bd4e7f20770d6276ea16b58dfdb041210ae1a15
Author: Holden Karau 
Date:   2017-11-03T23:07:27Z

Rewrite lambda with expansion

commit f58668416f3f451166dbe644ab56581c4902d6b6
Author: Holden Karau 
Date:   2017-11-03T23:07:42Z

Fix items and builtin import and a long import

commit da9ebb7667df4992512177d1d0e8b9ce4cafc62e
Author: Holden Karau 
Date:   2017-08-20T21:11:07Z

Rich comparisions requirement

commit dfc4e9057acb417e261a5e59399e5e943861cc3d
Author: Holden Karau 
Date:   2017-08-20T21:42:33Z

Add __hash__ methods in places where needed (in progress)

commit ca1f224d080ea37fd0de3582e9a7a5847f476f34
Author: Holden Karau 
Date:   2017-08-21T09:03:52Z

Provide eq/hashing/sorting for TimestampedValue

commit ea502b5a95b42ded289a97bc59a3cfbd2b9e05d8
Author: Holden Karau 
Date:   2017-08-21T18:44:37Z

If we are passed in bytes for the lookup in pipeline_context do the lookup 
as a string

commit fee6a65eaef0b85386fa42cd02ef45da8e370d51
Author: Holden Karau 
Date:   2017-08-21T19:08:44Z

Provide a base ne using eq for windows

commit 099e115f190d86352cc9b01a0b4feb48b3442192
Author: Holden Karau 
Date:   2017-11-03T23:11:29Z

Switch filesystem to io.StringIO

commit b0c7cda8626b90a7bd260077f54da5c07ba2b3c4
Author: Holden Karau 
Date:   2017-11-03T23:47:13Z

Fix cirular dep introduced through Py3 import changes for 
CreatePTransformOverride.

commit b0af9cca420d97b9037be91ddba9719a01a272ad
Author: Holden Karau 
Date:   2017-11-04T00:00:34Z

Change slow_stream to bytes

commit 5890721770aca710d5f8f5194ade7a9f2a19fbc5
Author: Holden Karau 
Date:   2017-11-04T04:40:17Z

Handle slow_stream in py2 again

commit bc30732264463b12a49f445e0f369e361e5fb823
Author: Holden Karau 
Date:   2017-11-04T04:58:44Z

Selective fixes while running through 
apache_beam.examples.complete.game.game_stats_test.GameStatsTest, fix some 
rethrows. lambda expansion, imports, and metaclass

commit b2424e790e01d1a758458f052f37648850719705
Author: Holden Karau 
Date:   2017-11-04T05:00:14Z

metaclass rewrite rule

commit fb63800079b3b44e4b6ec59844c20813b1ba449a
Author: Holden Karau 
Date:   2017-11-04T06:36:00Z

Apply futurize on gen_protos

commit 767c1cc30d32edf21edf1916a47a8f5c779512fd
Author: Holden Karau 
Date:   2017-11-04T07:16:12Z

queue and raise rewrites

commit e8e40ac46097fd3b4ecc7f40383031755a0b265f
Author: Holden Karau 
Date:   2017-11-04T07:18:10Z

futurize -wn -f libfuturize.fixes.fix_future_standard_library_urllib 
./apache_beam

commit fe278de3039ceaae7af819cb119bb31f25d679fa
Author: Holden Karau 
Date:   2017-11-04T08:04:59Z

Finish up getting 
apache_beam.examples.complete.game.game_stats_test.GameStatsTest to run, fix 
serialized values map over, switch some exception re-throws, iteritems, and 
handle peaking at the eval context.

commit f2c78e6f0b66b11292035eac224914dc439d175c
Author: Holden Karau 
Date:   2017-11-04T08:21:04Z

Fix the write_byte on slow stream to just use bytes rather than encode

commit 8dbf4e72cdb5a4bf6a4280f472acca85ef3c991c
Author: Holden Karau 
Date:   2017-11-04T08:25:14Z

Fix some bytes coders tests to use bytes in py3

commit d43b70419c883ad30183893d967456a78595f11d
Author: Holden Karau 
Date:   2017-11-04T08:25:38Z

Change the fall through for encoding a bit and update past imports

commit 5c102b70cce3ad49c635d4ea52432356ddf00f50
Author: Holden Karau 
Date:   2017-11-04T08:25:57Z

Only wrap in bytes for py3

commit 669c3f90c1b71cc34f89c9b3e52e95ac385b1333
Author: Holden Karau 
Date:   2017-11-05T04:09:21Z

Fix remaining coder issues accross py2/3




---