[jira] [Commented] (BEAM-3088) BigQuery source should consider streaming buffer when determining estimated sizes of tables

2017-10-21 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BEAM-3088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214157#comment-16214157
 ] 

ASF GitHub Bot commented on BEAM-3088:
--

GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/4025

[BEAM-3088] Improves size estimation of BigQueryTableSource.

Updates BigQueryTableSource to consider data in streaming buffer when 
determining estimated size.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/beam bq_size_estimation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4025.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4025


commit 501b43800e95a8722315c43c7379725407d04f7c
Author: chamik...@google.com 
Date:   2017-10-22T02:20:07Z

Updates BigQueryTableSource to consider data in streaming buffer when 
determining estimated size.




> BigQuery source should consider streaming buffer when determining estimated 
> sizes of tables
> ---
>
> Key: BEAM-3088
> URL: https://issues.apache.org/jira/browse/BEAM-3088
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>
> Currently BigQuery table source determines estimated size using 
> table.numBytes property.
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableSource.java#L100
> If BigQuery table has data in the streaming buffer, size of that data will 
> not be reflected by table.numBytes. To better estimate size of table, data in 
> the streaming buffer has to be considered as well. Size of data in streaming 
> buffer can be determined based on property streamingBuffer.estimatedBytes 
> according to following.
> https://cloud.google.com/bigquery/docs/reference/rest/v2/tables



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[GitHub] beam pull request #4025: [BEAM-3088] Improves size estimation of BigQueryTab...

2017-10-21 Thread chamikaramj
GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/4025

[BEAM-3088] Improves size estimation of BigQueryTableSource.

Updates BigQueryTableSource to consider data in streaming buffer when 
determining estimated size.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/beam bq_size_estimation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4025.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4025


commit 501b43800e95a8722315c43c7379725407d04f7c
Author: chamik...@google.com 
Date:   2017-10-22T02:20:07Z

Updates BigQueryTableSource to consider data in streaming buffer when 
determining estimated size.




---


[jira] [Created] (BEAM-3088) BigQuery source should consider streaming buffer when determining estimated sizes of tables

2017-10-21 Thread Chamikara Jayalath (JIRA)
Chamikara Jayalath created BEAM-3088:


 Summary: BigQuery source should consider streaming buffer when 
determining estimated sizes of tables
 Key: BEAM-3088
 URL: https://issues.apache.org/jira/browse/BEAM-3088
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Reporter: Chamikara Jayalath
Assignee: Chamikara Jayalath


Currently BigQuery table source determines estimated size using table.numBytes 
property.
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableSource.java#L100

If BigQuery table has data in the streaming buffer, size of that data will not 
be reflected by table.numBytes. To better estimate size of table, data in the 
streaming buffer has to be considered as well. Size of data in streaming buffer 
can be determined based on property streamingBuffer.estimatedBytes according to 
following.
https://cloud.google.com/bigquery/docs/reference/rest/v2/tables





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4207

2017-10-21 Thread Apache Jenkins Server
See 




Build failed in Jenkins: beam_PerformanceTests_Python #474

2017-10-21 Thread Apache Jenkins Server
See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision a71f00078836e33f6aabc7b424f932f85d0ae80a (origin/master)
Commit message: "This closes #3994"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a71f00078836e33f6aabc7b424f932f85d0ae80a
 > git rev-list a71f00078836e33f6aabc7b424f932f85d0ae80a # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7587238649091923165.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4261869658496392451.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2179401270404289834.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7307968592422057750.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: grpcio<2.0,>=1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: httplib2<0.10,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5084

2017-10-21 Thread Apache Jenkins Server
See 


--
[...truncated 299.97 KB...]
2017-10-22T00:03:45.912 [INFO] 

2017-10-22T00:03:45.912 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
MQTT
2017-10-22T00:03:45.912 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-22T00:03:45.912 [INFO] 

2017-10-22T00:03:45.912 [INFO]  
   
2017-10-22T00:03:45.913 [INFO] 

2017-10-22T00:03:45.913 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Redis
2017-10-22T00:03:45.913 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-22T00:03:45.913 [INFO] 

2017-10-22T00:03:45.913 [INFO]  
   
2017-10-22T00:03:45.914 [INFO] 

2017-10-22T00:03:45.914 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Solr
2017-10-22T00:03:45.914 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-22T00:03:45.914 [INFO] 

2017-10-22T00:03:45.914 [INFO]  
   
2017-10-22T00:03:45.914 [INFO] 

2017-10-22T00:03:45.915 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Tika
2017-10-22T00:03:45.915 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-22T00:03:45.915 [INFO] 

2017-10-22T00:03:45.915 [INFO]  
   
2017-10-22T00:03:45.915 [INFO] 

2017-10-22T00:03:45.915 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: XML
2017-10-22T00:03:45.915 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-22T00:03:45.915 [INFO] 

2017-10-22T00:03:45.916 [INFO]  
   
2017-10-22T00:03:45.916 [INFO] 

2017-10-22T00:03:45.916 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Starter
2017-10-22T00:03:45.916 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-22T00:03:45.916 [INFO] 

2017-10-22T00:03:45.916 [INFO]  
   
2017-10-22T00:03:45.916 [INFO] 

2017-10-22T00:03:45.916 [INFO] Skipping Apache Beam :: Examples :: Java
2017-10-22T00:03:45.917 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-22T00:03:45.917 [INFO] 

2017-10-22T00:03:45.917 [INFO]  
   
2017-10-22T00:03:45.917 [INFO] 

2017-10-22T00:03:45.917 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Examples
2017-10-22T00:03:45.917 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-22T00:03:45.917 [INFO] 

2017-10-22T00:03:45.918 [INFO]  
   
2017-10-22T00:03:45.918 [INFO] 

2017-10-22T00:03:45.918 [INFO] Skipping Apache Beam :: Examples :: Java 8
2017-10-22T00:03:45.918 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-22T00:03:45.918 [INFO] 

2017-10-22T00:03:45.919 [INFO]  
   
2017-10-22T00:03:45.919 [INFO] 

2017-10-22T00:03:45.919 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Examples - Java 8
2017-10-22T00:03:45.919 [INFO] This project has been banned from the build due 
to previous failures.

Build failed in Jenkins: beam_PostCommit_Python_Verify #3402

2017-10-21 Thread Apache Jenkins Server
See 


--
[...truncated 888.07 KB...]
copying apache_beam/runners/common.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow #4206

2017-10-21 Thread Apache Jenkins Server
See 




Build failed in Jenkins: beam_PerformanceTests_Python #473

2017-10-21 Thread Apache Jenkins Server
See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision a71f00078836e33f6aabc7b424f932f85d0ae80a (origin/master)
Commit message: "This closes #3994"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a71f00078836e33f6aabc7b424f932f85d0ae80a
 > git rev-list a71f00078836e33f6aabc7b424f932f85d0ae80a # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2033207693905946855.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2471928375143909365.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins754690828021045092.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8326269229205033049.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: grpcio<2.0,>=1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: httplib2<0.10,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5083

2017-10-21 Thread Apache Jenkins Server
See 


--
[...truncated 299.99 KB...]
2017-10-21T18:11:13.068 [INFO] 

2017-10-21T18:11:13.068 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
MQTT
2017-10-21T18:11:13.068 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-21T18:11:13.068 [INFO] 

2017-10-21T18:11:13.068 [INFO]  
   
2017-10-21T18:11:13.068 [INFO] 

2017-10-21T18:11:13.068 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Redis
2017-10-21T18:11:13.068 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-21T18:11:13.068 [INFO] 

2017-10-21T18:11:13.070 [INFO]  
   
2017-10-21T18:11:13.070 [INFO] 

2017-10-21T18:11:13.070 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Solr
2017-10-21T18:11:13.070 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-21T18:11:13.070 [INFO] 

2017-10-21T18:11:13.070 [INFO]  
   
2017-10-21T18:11:13.070 [INFO] 

2017-10-21T18:11:13.070 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Tika
2017-10-21T18:11:13.070 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-21T18:11:13.070 [INFO] 

2017-10-21T18:11:13.071 [INFO]  
   
2017-10-21T18:11:13.071 [INFO] 

2017-10-21T18:11:13.071 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: XML
2017-10-21T18:11:13.071 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-21T18:11:13.071 [INFO] 

2017-10-21T18:11:13.071 [INFO]  
   
2017-10-21T18:11:13.071 [INFO] 

2017-10-21T18:11:13.071 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Starter
2017-10-21T18:11:13.071 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-21T18:11:13.071 [INFO] 

2017-10-21T18:11:13.072 [INFO]  
   
2017-10-21T18:11:13.072 [INFO] 

2017-10-21T18:11:13.072 [INFO] Skipping Apache Beam :: Examples :: Java
2017-10-21T18:11:13.072 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-21T18:11:13.072 [INFO] 

2017-10-21T18:11:13.072 [INFO]  
   
2017-10-21T18:11:13.072 [INFO] 

2017-10-21T18:11:13.073 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Examples
2017-10-21T18:11:13.077 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-21T18:11:13.077 [INFO] 

2017-10-21T18:11:13.077 [INFO]  
   
2017-10-21T18:11:13.077 [INFO] 

2017-10-21T18:11:13.077 [INFO] Skipping Apache Beam :: Examples :: Java 8
2017-10-21T18:11:13.077 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-21T18:11:13.077 [INFO] 

2017-10-21T18:11:13.078 [INFO]  
   
2017-10-21T18:11:13.078 [INFO] 

2017-10-21T18:11:13.078 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Examples - Java 8
2017-10-21T18:11:13.078 [INFO] This project has been banned from the build due 
to previous failures.

Build failed in Jenkins: beam_PostCommit_Python_Verify #3401

2017-10-21 Thread Apache Jenkins Server
See 


--
[...truncated 885.59 KB...]
copying apache_beam/runners/common.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying 

Build failed in Jenkins: beam_PerformanceTests_Python #472

2017-10-21 Thread Apache Jenkins Server
See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision a71f00078836e33f6aabc7b424f932f85d0ae80a (origin/master)
Commit message: "This closes #3994"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a71f00078836e33f6aabc7b424f932f85d0ae80a
 > git rev-list a71f00078836e33f6aabc7b424f932f85d0ae80a # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2433225915614449351.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7310847315584923148.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3664133487873889786.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2193165265691904566.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: grpcio<2.0,>=1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: httplib2<0.10,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink #4122

2017-10-21 Thread Apache Jenkins Server
See 




Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5082

2017-10-21 Thread Apache Jenkins Server
See 


--
[...truncated 300.01 KB...]
2017-10-21T12:02:51.082 [INFO] 

2017-10-21T12:02:51.082 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
MQTT
2017-10-21T12:02:51.082 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-21T12:02:51.082 [INFO] 

2017-10-21T12:02:51.082 [INFO]  
   
2017-10-21T12:02:51.082 [INFO] 

2017-10-21T12:02:51.082 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Redis
2017-10-21T12:02:51.082 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-21T12:02:51.082 [INFO] 

2017-10-21T12:02:51.083 [INFO]  
   
2017-10-21T12:02:51.083 [INFO] 

2017-10-21T12:02:51.083 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Solr
2017-10-21T12:02:51.083 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-21T12:02:51.083 [INFO] 

2017-10-21T12:02:51.083 [INFO]  
   
2017-10-21T12:02:51.083 [INFO] 

2017-10-21T12:02:51.083 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Tika
2017-10-21T12:02:51.083 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-21T12:02:51.083 [INFO] 

2017-10-21T12:02:51.083 [INFO]  
   
2017-10-21T12:02:51.083 [INFO] 

2017-10-21T12:02:51.083 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: XML
2017-10-21T12:02:51.083 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-21T12:02:51.083 [INFO] 

2017-10-21T12:02:51.083 [INFO]  
   
2017-10-21T12:02:51.083 [INFO] 

2017-10-21T12:02:51.083 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Starter
2017-10-21T12:02:51.083 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-21T12:02:51.083 [INFO] 

2017-10-21T12:02:51.083 [INFO]  
   
2017-10-21T12:02:51.083 [INFO] 

2017-10-21T12:02:51.083 [INFO] Skipping Apache Beam :: Examples :: Java
2017-10-21T12:02:51.083 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-21T12:02:51.083 [INFO] 

2017-10-21T12:02:51.083 [INFO]  
   
2017-10-21T12:02:51.083 [INFO] 

2017-10-21T12:02:51.083 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Examples
2017-10-21T12:02:51.083 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-21T12:02:51.083 [INFO] 

2017-10-21T12:02:51.083 [INFO]  
   
2017-10-21T12:02:51.083 [INFO] 

2017-10-21T12:02:51.083 [INFO] Skipping Apache Beam :: Examples :: Java 8
2017-10-21T12:02:51.083 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-21T12:02:51.083 [INFO] 

2017-10-21T12:02:51.083 [INFO]  
   
2017-10-21T12:02:51.083 [INFO] 

2017-10-21T12:02:51.083 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Examples - Java 8
2017-10-21T12:02:51.083 [INFO] This project has been banned from the build due 
to previous failures.

Build failed in Jenkins: beam_PostCommit_Python_Verify #3400

2017-10-21 Thread Apache Jenkins Server
See 


--
[...truncated 883.82 KB...]
copying apache_beam/runners/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #4204

2017-10-21 Thread Apache Jenkins Server
See 


--
[...truncated 21.22 MB...]
[INFO] 2017-10-21T07:45:38.249Z: (8fff0a555e71cde1): Executing operation 
PAssert$328/GroupGlobally/Create.Values/Read(CreateSource)+PAssert$328/GroupGlobally/WindowIntoDummy/Window.Assign+PAssert$328/GroupGlobally/GroupDummyAndContents/Reify+PAssert$328/GroupGlobally/GroupDummyAndContents/Write
[INFO] 2017-10-21T07:45:38.263Z: (e477c106f1e6f615): Executing operation 
PAssert$328/GroupGlobally/GatherAllOutputs/GroupByKey/Read+PAssert$328/GroupGlobally/GatherAllOutputs/GroupByKey/GroupByWindow+PAssert$328/GroupGlobally/GatherAllOutputs/Values/Values/Map+PAssert$328/GroupGlobally/RewindowActuals/Window.Assign+PAssert$328/GroupGlobally/KeyForDummy/AddKeys/Map+PAssert$328/GroupGlobally/GroupDummyAndContents/Reify+PAssert$328/GroupGlobally/GroupDummyAndContents/Write
[INFO] 2017-10-21T07:45:38.340Z: (aafbc1dea50d965a): Workers have started 
successfully.
[INFO] 2017-10-21T07:45:40.855Z: (8db9cc5364413bc7): Executing operation 
PAssert$329/GroupGlobally/GroupDummyAndContents/Close
[INFO] 2017-10-21T07:45:40.871Z: (8db9cc53644133a4): Executing operation 
PAssert$329/GroupGlobally/GroupDummyAndContents/Read+PAssert$329/GroupGlobally/GroupDummyAndContents/GroupByWindow+PAssert$329/GroupGlobally/Values/Values/Map+PAssert$329/GroupGlobally/ParDo(Concat)+PAssert$329/GetPane/Map+PAssert$329/RunChecks+PAssert$329/VerifyAssertions/ParDo(DefaultConclude)
[INFO] 2017-10-21T07:45:41.284Z: (d63f0d0b7570481a): Workers have started 
successfully.
[INFO] 2017-10-21T07:45:40.770Z: (6ba7e59a135d7486): Autoscaling: Raised the 
number of workers to 2 based on the rate of progress in the currently running 
step(s).
[INFO] 2017-10-21T07:45:42.273Z: (e518ca445766bb50): Executing operation 
PAssert$330/GroupGlobally/GatherAllOutputs/GroupByKey/Close
[INFO] 2017-10-21T07:45:42.290Z: (e518ca445766bf78): Executing operation 
PAssert$330/GroupGlobally/GroupDummyAndContents/Create
[INFO] 2017-10-21T07:45:42.336Z: (e518ca445766b1dc): Executing operation 
PAssert$330/GroupGlobally/Create.Values/Read(CreateSource)+PAssert$330/GroupGlobally/WindowIntoDummy/Window.Assign+PAssert$330/GroupGlobally/GroupDummyAndContents/Reify+PAssert$330/GroupGlobally/GroupDummyAndContents/Write
[INFO] 2017-10-21T07:45:42.347Z: (45e51e70914bb5ee): Executing operation 
PAssert$330/GroupGlobally/GatherAllOutputs/GroupByKey/Read+PAssert$330/GroupGlobally/GatherAllOutputs/GroupByKey/GroupByWindow+PAssert$330/GroupGlobally/GatherAllOutputs/Values/Values/Map+PAssert$330/GroupGlobally/RewindowActuals/Window.Assign+PAssert$330/GroupGlobally/KeyForDummy/AddKeys/Map+PAssert$330/GroupGlobally/GroupDummyAndContents/Reify+PAssert$330/GroupGlobally/GroupDummyAndContents/Write
[INFO] 2017-10-21T07:45:47.104Z: (817a370044b5ff75): Autoscaling: Resized 
worker pool from 1 to 0.
[INFO] 2017-10-21T07:45:47.106Z: (817a370044b5f693): Autoscaling: Would further 
reduce the number of workers but reached the minimum number allowed for the job.
[INFO] 2017-10-21T07:45:47.929Z: (e477c106f1e6f183): Executing operation 
PAssert$328/GroupGlobally/GroupDummyAndContents/Close
[INFO] 2017-10-21T07:45:47.947Z: (e477c106f1e6f7bc): Executing operation 
PAssert$328/GroupGlobally/GroupDummyAndContents/Read+PAssert$328/GroupGlobally/GroupDummyAndContents/GroupByWindow+PAssert$328/GroupGlobally/Values/Values/Map+PAssert$328/GroupGlobally/ParDo(Concat)+PAssert$328/GetPane/Map+PAssert$328/RunChecks+PAssert$328/VerifyAssertions/ParDo(DefaultConclude)
[INFO] 2017-10-21T07:45:50.055Z: (45e51e70914bb578): Executing operation 
PAssert$330/GroupGlobally/GroupDummyAndContents/Close
[INFO] 2017-10-21T07:45:50.072Z: (45e51e70914bb383): Executing operation 
PAssert$330/GroupGlobally/GroupDummyAndContents/Read+PAssert$330/GroupGlobally/GroupDummyAndContents/GroupByWindow+PAssert$330/GroupGlobally/Values/Values/Map+PAssert$330/GroupGlobally/ParDo(Concat)+PAssert$330/GetPane/Map+PAssert$330/RunChecks+PAssert$330/VerifyAssertions/ParDo(DefaultConclude)
[INFO] 2017-10-21T07:45:48.702Z: (a6d0046086ef7e16): Cleaning up.
[INFO] 2017-10-21T07:45:48.708Z: (a6d0046086ef7092): Stopping worker pool...
[INFO] 2017-10-21T07:45:52.872Z: (b2339a651d0c41df): Cleaning up.
[INFO] 2017-10-21T07:45:52.876Z: (b2339a651d0c49c3): Stopping worker pool...
[INFO] 2017-10-21T07:45:52.818Z: (76d76ae1b5a37773): Executing operation 
View.AsMap/View.CreatePCollectionView/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
[INFO] 2017-10-21T07:45:52.837Z: (76d76ae1b5a375a6): Executing operation 
View.AsMap/View.CreatePCollectionView/GBKaSVForSize/Create
[INFO] 2017-10-21T07:45:52.850Z: (f7b3f57bb2ceb752): Executing operation 
View.AsMap/View.CreatePCollectionView/GBKaSVForKeys/Create
[INFO] 2017-10-21T07:45:52.901Z: (f7b3f57bb2ceb231): Executing operation 

Build failed in Jenkins: beam_PerformanceTests_Python #471

2017-10-21 Thread Apache Jenkins Server
See 


Changes:

[altay] Unit test for label pipeline option

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision a71f00078836e33f6aabc7b424f932f85d0ae80a (origin/master)
Commit message: "This closes #3994"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a71f00078836e33f6aabc7b424f932f85d0ae80a
 > git rev-list 66fcda99c1e5902ff4d0981b887c70f7afe9380a # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6670417914018210687.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1572457989060175940.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4445079751784129548.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7246760989251792717.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: grpcio<2.0,>=1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: httplib2<0.10,>=0.8 in 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink #4121

2017-10-21 Thread Apache Jenkins Server
See 


--
[...truncated 543.90 KB...]
2017-10-21T06:07:57.700 [INFO] Excluding 
com.google.protobuf:protobuf-java-util:jar:3.2.0 from the shaded jar.
2017-10-21T06:07:57.700 [INFO] Excluding com.google.code.gson:gson:jar:2.7 from 
the shaded jar.
2017-10-21T06:07:57.700 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.9 from the shaded jar.
2017-10-21T06:07:57.700 [INFO] Including com.google.guava:guava:jar:20.0 in the 
shaded jar.
2017-10-21T06:07:57.700 [INFO] Excluding 
com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
2017-10-21T06:07:57.700 [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.25 from 
the shaded jar.
2017-10-21T06:07:57.700 [INFO] Excluding joda-time:joda-time:jar:2.4 from the 
shaded jar.
2017-10-21T06:07:57.701 [INFO] Excluding 
org.apache.commons:commons-compress:jar:1.14 from the shaded jar.
2017-10-21T06:07:57.701 [INFO] Excluding 
com.google.auto.service:auto-service:jar:1.0-rc2 from the shaded jar.
2017-10-21T06:07:57.701 [INFO] Excluding com.google.auto:auto-common:jar:0.3 
from the shaded jar.
2017-10-21T06:07:57.701 [INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from 
the shaded jar.
2017-10-21T06:07:57.701 [INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 
from the shaded jar.
2017-10-21T06:08:00.951 [INFO] Replacing original artifact with shaded artifact.
2017-10-21T06:08:00.951 [INFO] Replacing 

 with 

2017-10-21T06:08:00.951 [INFO] Replacing original test artifact with shaded 
test artifact.
2017-10-21T06:08:00.951 [INFO] Replacing 

 with 

2017-10-21T06:08:00.952 [INFO] Dependency-reduced POM written at: 

2017-10-21T06:08:01.214 [INFO] 
2017-10-21T06:08:01.214 [INFO] --- maven-surefire-plugin:2.20:test 
(validates-runner-tests) @ beam-runners-flink_2.10 ---
2017-10-21T06:08:01.322 [INFO] Surefire report directory: 

2017-10-21T06:08:01.323 [INFO] Using configured provider 
org.apache.maven.surefire.junitcore.JUnitCoreProvider
2017-10-21T06:08:01.323 [INFO] parallel='none', perCoreThreadCount=true, 
threadCount=4, useUnlimitedThreads=false, threadCountSuites=0, 
threadCountClasses=0, threadCountMethods=0, parallelOptimized=true
2017-10-21T06:08:01.324 [INFO] 
2017-10-21T06:08:01.324 [INFO] 
---
2017-10-21T06:08:01.324 [INFO]  T E S T S
2017-10-21T06:08:01.325 [INFO] 
---
2017-10-21T06:08:05.579 [INFO] Running 
org.apache.beam.sdk.values.PCollectionTupleTest
2017-10-21T06:08:17.652 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 12.067 s - in org.apache.beam.sdk.values.PCollectionTupleTest
2017-10-21T06:08:17.652 [INFO] Running org.apache.beam.sdk.io.ReadTest
2017-10-21T06:08:17.785 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.131 s - in org.apache.beam.sdk.io.ReadTest
2017-10-21T06:08:17.785 [INFO] Running 
org.apache.beam.sdk.io.TextIOReadTest$BasicIOTest
2017-10-21T06:08:17.813 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.026 s - in org.apache.beam.sdk.io.TextIOReadTest$BasicIOTest
2017-10-21T06:08:17.813 [INFO] Running org.apache.beam.sdk.transforms.KeysTest
2017-10-21T06:08:22.741 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 4.926 s - in org.apache.beam.sdk.transforms.KeysTest
2017-10-21T06:08:22.741 [INFO] Running 
org.apache.beam.sdk.transforms.ToStringTest
2017-10-21T06:08:32.009 [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 9.266 s - in org.apache.beam.sdk.transforms.ToStringTest
2017-10-21T06:08:32.009 [INFO] Running 
org.apache.beam.sdk.transforms.ReshuffleTest
2017-10-21T06:08:43.883 [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 11.872 s - in org.apache.beam.sdk.transforms.ReshuffleTest
2017-10-21T06:08:43.883 [INFO] Running org.apache.beam.sdk.transforms.ViewTest
2017-10-21T06:09:37.930 [INFO] Tests run: 34, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 54.044 s - in org.apache.beam.sdk.transforms.ViewTest

[jira] [Created] (BEAM-3087) Extend lock scope in Flink UnboundedSourceWrapper

2017-10-21 Thread Aljoscha Krettek (JIRA)
Aljoscha Krettek created BEAM-3087:
--

 Summary: Extend lock scope in Flink UnboundedSourceWrapper
 Key: BEAM-3087
 URL: https://issues.apache.org/jira/browse/BEAM-3087
 Project: Beam
  Issue Type: Bug
  Components: runner-flink
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek
 Fix For: 2.3.0


In {{UnboundedSourceWrapper}} the lock scope is not big enough: we synchronise 
in {{emitElement()}} but should instead synchronise inside the reader loop in 
{{run()}} because the {{Source}} interface does not allow concurrent calls.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Assigned] (BEAM-3087) Extend lock scope in Flink UnboundedSourceWrapper

2017-10-21 Thread Aljoscha Krettek (JIRA)

 [ 
https://issues.apache.org/jira/browse/BEAM-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reassigned BEAM-3087:
--

Assignee: (was: Aljoscha Krettek)

> Extend lock scope in Flink UnboundedSourceWrapper
> -
>
> Key: BEAM-3087
> URL: https://issues.apache.org/jira/browse/BEAM-3087
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
> Fix For: 2.3.0
>
>
> In {{UnboundedSourceWrapper}} the lock scope is not big enough: we 
> synchronise in {{emitElement()}} but should instead synchronise inside the 
> reader loop in {{run()}} because the {{Source}} interface does not allow 
> concurrent calls.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)