date:20180512

[beam] branch master updated (3a52bb1 -> bb319db)

2018-05-12 Thread jkff

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 3a52bb1  Merge pull request #5312: Add a Pipeline Constructor to 
QueryablePipeline
 add 453336e  [BEAM-3515] Portable translation of SplittableProcessKeyed
 add 911e634  Fixes compile error in FlinkExecutableStageFunctionTest
 add 11590a1  Sickbays flaky test
 new bb319db  Merge pull request #5277: [BEAM-3515] Portable translation of 
SplittableProcessKeyed

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../pipeline/src/main/proto/beam_runner_api.proto  |  14 ++-
 .../core/construction/PTransformTranslation.java   |   3 +
 .../core/construction/ParDoTranslation.java|  64 +--
 .../runners/core/construction/SplittableParDo.java | 107 --
 .../core/construction/EnvironmentsTest.java|   2 +
 .../core/construction/ParDoTranslationTest.java|   2 +-
 .../apache/beam/runners/direct/DirectRunner.java   |   2 +-
 .../runners/flink/FlinkTransformOverrides.java |   3 +-
 .../FlinkExecutableStageFunctionTest.java  |  24 ++--
 .../dataflow/DataflowPipelineTranslator.java   |  36 +++---
 .../dataflow/PrimitiveParDoSingleFactory.java  |  37 +++---
 .../beam/runners/dataflow/TransformTranslator.java |   5 +
 .../dataflow/DataflowPipelineTranslatorTest.java   | 124 +
 .../fnexecution/data/GrpcDataServiceTest.java  |   2 +
 14 files changed, 293 insertions(+), 132 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[jira] [Commented] (BEAM-4260) Document usage for hcatalog 1.1

2018-05-12 Thread Tim Robertson (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-4260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472957#comment-16472957
 ] 

Tim Robertson commented on BEAM-4260:
-

Given {{HCatalogIO}} only provides JDoc (no readme etc), how about we add to 
the JDdoc something along the lines of:

"HCatalogIO is designed to run on HCatlog versions 2.x. To package for use on 
1.x see BEAM-4260"

> Document usage for hcatalog 1.1
> ---
>
> Key: BEAM-4260
> URL: https://issues.apache.org/jira/browse/BEAM-4260
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-hcatalog
>Affects Versions: 2.4.0
>Reporter: Tim Robertson
>Assignee: Ismaël Mejía
>Priority: Minor
>
> The {{HCatalogIO}} does not work with environments providing Hive Server 1.x 
> which is in widespread use - as an example the latest Cloudera (5.14.2) 
> provides 1.1.x
>  
> The {{HCatalogIO}} marks it's Hive dependencies as provided, so I believe the 
> intention was to be open to multiple versions.
>  
> The issues come from the following:  
>  - use of {{HCatUtil.getHiveMetastoreClient(hiveConf)}} while previous 
> versions used the [now 
> deprecated|https://github.com/apache/hive/blob/master/hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java#L586]
>  {{getHiveClient(HiveConf hiveConf)}}  
>  - Changes to the signature of {{RetryingMetaStoreClient.getProxy(...)}}
>  
> Given this doesn't work in a major Hadoop distro, and will not until the next 
> CDH release later in 2018 (i.e. widespread adoption only expected in 2019) I 
> think it would be worthwhile providing a fix/workaround.
> I _think_ building for 2.3 and relocating in your own app might be a 
> workaround although I'm still testing it.  If that is successful I'd propose 
> adding it to the project README or in a separate markdown file linked from 
> the README.
> Does that sound like a reasonable approach please?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1615

2018-05-12 Thread Apache Jenkins Server

See 


Changes:

[ekirpichov] [BEAM-3515] Portable translation of SplittableProcessKeyed

[ekirpichov] Fixes compile error in FlinkExecutableStageFunctionTest

[ekirpichov] Sickbays flaky test

--
Started by GitHub push by jkff
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision bb319dbfaa81c27d7ba6b0a465994aa656143100 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f bb319dbfaa81c27d7ba6b0a465994aa656143100
Commit message: "Merge pull request #5277: [BEAM-3515] Portable translation of 
SplittableProcessKeyed"
 > git rev-list --no-walk 3a52bb108f3f2937c42d81d1e9b8b055c8047e1a # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins2178032194908146260.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ekirpic...@gmail.com

[beam] 01/01: Merge pull request #5277: [BEAM-3515] Portable translation of SplittableProcessKeyed

2018-05-12 Thread jkff

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit bb319dbfaa81c27d7ba6b0a465994aa656143100
Merge: 3a52bb1 11590a1
Author: Eugene Kirpichov 
AuthorDate: Sat May 12 00:21:12 2018 -0700

Merge pull request #5277: [BEAM-3515] Portable translation of 
SplittableProcessKeyed

[BEAM-3515] Portable translation of SplittableProcessKeyed

 .../pipeline/src/main/proto/beam_runner_api.proto  |  14 ++-
 .../core/construction/PTransformTranslation.java   |   3 +
 .../core/construction/ParDoTranslation.java|  64 +--
 .../runners/core/construction/SplittableParDo.java | 107 --
 .../core/construction/EnvironmentsTest.java|   2 +
 .../core/construction/ParDoTranslationTest.java|   2 +-
 .../apache/beam/runners/direct/DirectRunner.java   |   2 +-
 .../runners/flink/FlinkTransformOverrides.java |   3 +-
 .../FlinkExecutableStageFunctionTest.java  |  24 ++--
 .../dataflow/DataflowPipelineTranslator.java   |  36 +++---
 .../dataflow/PrimitiveParDoSingleFactory.java  |  37 +++---
 .../beam/runners/dataflow/TransformTranslator.java |   5 +
 .../dataflow/DataflowPipelineTranslatorTest.java   | 124 +
 .../fnexecution/data/GrpcDataServiceTest.java  |   2 +
 14 files changed, 293 insertions(+), 132 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[jira] [Work logged] (BEAM-3515) Use portable ParDoPayload for SDF in DataflowRunner

2018-05-12 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3515?focusedWorklogId=101408&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101408
 ]

ASF GitHub Bot logged work on BEAM-3515:


Author: ASF GitHub Bot
Created on: 12/May/18 07:21
Start Date: 12/May/18 07:21
Worklog Time Spent: 10m 
  Work Description: jkff commented on issue #5277: [BEAM-3515] Portable 
translation of SplittableProcessKeyed
URL: https://github.com/apache/beam/pull/5277#issuecomment-388536356
 
 
   I am not sure what's wrong with the archetypes but this is definitely not 
related to the current PR - the same Gradle tasks fail at master as well.
   
   Running ./gradlew :javaPreCommit locally with these two tasks excluded 
passes. I'm going to merge this.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101408)
Time Spent: 5h  (was: 4h 50m)

> Use portable ParDoPayload for SDF in DataflowRunner
> ---
>
> Key: BEAM-3515
> URL: https://issues.apache.org/jira/browse/BEAM-3515
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Eugene Kirpichov
>Priority: Major
>  Labels: portability
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> The Java-specific blobs transmitted to Dataflow need more context, in the 
> form of portability framework protos.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3515) Use portable ParDoPayload for SDF in DataflowRunner

2018-05-12 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3515?focusedWorklogId=101409&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101409
 ]

ASF GitHub Bot logged work on BEAM-3515:


Author: ASF GitHub Bot
Created on: 12/May/18 07:21
Start Date: 12/May/18 07:21
Worklog Time Spent: 10m 
  Work Description: jkff closed pull request #5277: [BEAM-3515] Portable 
translation of SplittableProcessKeyed
URL: https://github.com/apache/beam/pull/5277
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/model/pipeline/src/main/proto/beam_runner_api.proto 
b/model/pipeline/src/main/proto/beam_runner_api.proto
index 33448316921..11646239b93 100644
--- a/model/pipeline/src/main/proto/beam_runner_api.proto
+++ b/model/pipeline/src/main/proto/beam_runner_api.proto
@@ -240,15 +240,18 @@ message StandardPTransforms {
 // Less well-known. Payload: WriteFilesPayload.
 WRITE_FILES = 4 [(beam_urn) = "beam:transform:write_files:v1"];
   }
+  // Payload for all of these: CombinePayload
   enum CombineComponents {
 COMBINE_PGBKCV = 0 [(beam_urn) = "beam:transform:combine_pgbkcv:v1"];
 COMBINE_MERGE_ACCUMULATORS = 1 [(beam_urn) = 
"beam:transform:combine_merge_accumulators:v1"];
 COMBINE_EXTRACT_OUTPUTS = 2 [(beam_urn) = 
"beam:transform:combine_extract_outputs:v1"];
   }
-
-  // This field is needed only as a work-around for a proto compiler bug.
-  // See https://github.com/google/protobuf/issues/4514
-  int32 ignored = 1;
+  // Payload for all of these: ParDoPayload containing the user's SDF
+  enum SplittableParDoComponents {
+PAIR_WITH_RESTRICTION = 0 [(beam_urn) = 
"beam:transform:sdf_pair_with_restriction:v1"];
+SPLIT_RESTRICTION = 1 [(beam_urn) = 
"beam:transform:sdf_split_restriction:v1"];
+PROCESS_KEYED_ELEMENTS = 2 [(beam_urn) = 
"beam:transform:sdf_process_keyed_elements:v1"];
+  }
 }
 
 message StandardSideInputTypes {
@@ -350,6 +353,9 @@ message ParDoPayload {
 
   // Whether the DoFn is splittable
   bool splittable = 6;
+
+  // (Required if splittable == true) Id of the restriction coder.
+  string restriction_coder_id = 7;
 }
 
 // Parameters that a UDF might require.
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
index 2eb27873105..37886605296 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
@@ -37,6 +37,7 @@
 import org.apache.beam.model.pipeline.v1.RunnerApi;
 import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
 import org.apache.beam.model.pipeline.v1.RunnerApi.StandardPTransforms;
+import 
org.apache.beam.model.pipeline.v1.RunnerApi.StandardPTransforms.SplittableParDoComponents;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.runners.AppliedPTransform;
 import org.apache.beam.sdk.transforms.PTransform;
@@ -82,6 +83,8 @@
   StandardPTransforms.Composites.RESHUFFLE);
   public static final String WRITE_FILES_TRANSFORM_URN =
   getUrn(StandardPTransforms.Composites.WRITE_FILES);
+  public static final String SPLITTABLE_PROCESS_KEYED_URN =
+  getUrn(SplittableParDoComponents.PROCESS_KEYED_ELEMENTS);
 
   private static final Map, 
TransformPayloadTranslator>
   KNOWN_PAYLOAD_TRANSLATORS = loadTransformPayloadTranslators();
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
index 6365d77dccc..d5478498798 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
@@ -48,6 +48,7 @@
 import org.apache.beam.model.pipeline.v1.RunnerApi.SideInput;
 import org.apache.beam.model.pipeline.v1.RunnerApi.SideInput.Builder;
 import 
org.apache.beam.runners.core.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.runners.AppliedPTransform;
 import org.apache.beam.sdk.state.StateSpec;
@@ -60,6 +61,7 @@
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.ParDo.MultiOutput;
 import org.apache.beam.sdk.transf

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #372

2018-05-12 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1616

2018-05-12 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision bb319dbfaa81c27d7ba6b0a465994aa656143100 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f bb319dbfaa81c27d7ba6b0a465994aa656143100
Commit message: "Merge pull request #5277: [BEAM-3515] Portable translation of 
SplittableProcessKeyed"
 > git rev-list --no-walk bb319dbfaa81c27d7ba6b0a465994aa656143100 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins8262055065470400243.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ekirpic...@gmail.com

[jira] [Work logged] (BEAM-3999) Futurize and fix python 2 compatibility for internal subpackage

2018-05-12 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3999?focusedWorklogId=101448&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101448
 ]

ASF GitHub Bot logged work on BEAM-3999:


Author: ASF GitHub Bot
Created on: 12/May/18 10:04
Start Date: 12/May/18 10:04
Worklog Time Spent: 10m 
  Work Description: RobbeSneyders commented on issue #5334: [BEAM-3999] 
Futurize internal subpackage
URL: https://github.com/apache/beam/pull/5334#issuecomment-388544578
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101448)
Time Spent: 20m  (was: 10m)

> Futurize and fix python 2 compatibility for internal subpackage
> ---
>
> Key: BEAM-3999
> URL: https://issues.apache.org/jira/browse/BEAM-3999
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4005) Futurize and fix python 2 compatibility for tools subpackage

2018-05-12 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-4005?focusedWorklogId=101449&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101449
 ]

ASF GitHub Bot logged work on BEAM-4005:


Author: ASF GitHub Bot
Created on: 12/May/18 10:09
Start Date: 12/May/18 10:09
Worklog Time Spent: 10m 
  Work Description: RobbeSneyders commented on a change in pull request 
#5343: [BEAM-4005] Futurize tools subpackage
URL: https://github.com/apache/beam/pull/5343#discussion_r187769527
 
 

 ##
 File path: sdks/python/apache_beam/tools/distribution_counter_microbenchmark.py
 ##
 @@ -23,11 +23,16 @@
   python -m apache_beam.tools.distribution_counter_microbenchmark
 """
 
+from __future__ import absolute_import
+from __future__ import division
 from __future__ import print_function
 
 import random
 import sys
 import time
+from builtins import range
+
+from past.utils import old_div
 
 Review comment:
   Can you replace old_div with the new Python 3 division? Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101449)
Time Spent: 20m  (was: 10m)

> Futurize and fix python 2 compatibility for tools subpackage
> 
>
> Key: BEAM-4005
> URL: https://issues.apache.org/jira/browse/BEAM-4005
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #226

2018-05-12 Thread Apache Jenkins Server

See 


Changes:

[35550199+arkash] BEAM-3246 Merge splits if they exceed 15K -- added checks 
to

[tgroh] Add a Pipeline Constructor to QueryablePipeline

[tgroh] Add Generic Types to StageBundleFactory

[tgroh] Remove `Components` from ProcessBundleDescriptors

[aromanenko.dev] [BEAM-4048] Refactor COMBINE mode for reading/writing from/to 
Pub/Sub

[Pablo] Adding context switch to operations missing it.

[sidhom] [BEAM-2597] Flink batch ExecutableStage operator

[thw] Fix compile error in RemoteExecutionTest.

[tgroh] Suppress more flaky tests.

[ekirpichov] [BEAM-3515] Portable translation of SplittableProcessKeyed

[ekirpichov] Fixes compile error in FlinkExecutableStageFunctionTest

[ekirpichov] Sickbays flaky test

--
[...truncated 1.49 MB...]
May 12, 2018 11:06:47 AM 
org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory
 create
INFO: No stagingLocation provided, falling back to gcpTempLocation
May 12, 2018 11:06:47 AM org.apache.beam.runners.dataflow.DataflowRunner 
fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from 
the classpath: will stage 109 files. Enable logging at DEBUG level to see which 
files will be staged.
May 12, 2018 11:06:48 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing 
implications related to Google Compute Engine usage and other Google Cloud 
Services.
May 12, 2018 11:06:48 AM org.apache.beam.runners.dataflow.util.PackageUtil 
stageClasspathElements
INFO: Uploading 109 files from PipelineOptions.filesToStage to staging location 
to prepare for execution.
May 12, 2018 11:06:48 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/tmp/groovy-generated-2903576834384006648-tmpdir/word-count-beam/target/classes 
to 
gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/classes-U_EOK_Acdr0wM6gh48Ya4g.jar
May 12, 2018 11:06:48 AM org.apache.beam.runners.dataflow.util.PackageUtil 
stageClasspathElements
INFO: Staging files complete: 108 files cached, 1 files newly uploaded
May 12, 2018 11:06:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ReadLines/Read as step s1
May 12, 2018 11:06:49 AM org.apache.beam.sdk.io.FileBasedSource 
getEstimatedSizeBytes
INFO: Filepattern gs://apache-beam-samples/shakespeare/* matched 43 files with 
total size 5284696
May 12, 2018 11:06:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WordCount.CountWords/ParDo(ExtractWords) as step s2
May 12, 2018 11:06:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WordCount.CountWords/Count.PerElement/Init/Map as step s3
May 12, 2018 11:06:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey as step 
s4
May 12, 2018 11:06:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues
 as step s5
May 12, 2018 11:06:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding MapElements/Map as step s6
May 12, 2018 11:06:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign as step s7
May 12, 2018 11:06:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles 
as step s8
May 12, 2018 11:06:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten as step 
s9
May 12, 2018 11:06:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten as step 
s10
May 12, 2018 11:06:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum as step s11
May 12, 2018 11:06:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections as 
step s12
May 12, 2018 11:06:49 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)
 as step s13
May 12, 2018 11:06:49 AM 
org.apache.beam.r

Build failed in Jenkins: beam_PerformanceTests_Spark #1702

2018-05-12 Thread Apache Jenkins Server

See 


Changes:

[ekirpichov] [BEAM-3515] Portable translation of SplittableProcessKeyed

[ekirpichov] Fixes compile error in FlinkExecutableStageFunctionTest

[ekirpichov] Sickbays flaky test

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision bb319dbfaa81c27d7ba6b0a465994aa656143100 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f bb319dbfaa81c27d7ba6b0a465994aa656143100
Commit message: "Merge pull request #5277: [BEAM-3515] Portable translation of 
SplittableProcessKeyed"
 > git rev-list --no-walk 3a52bb108f3f2937c42d81d1e9b8b055c8047e1a # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins489093102568158953.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins2614161852404223276.sh
+ rm -rf .env
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins8210018324112720811.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins8173877629829253545.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (39.1.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins1256972828646278089.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5170494019148339988.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (39.1.0)
Collecting colorlog[windows]==2.6.0 (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt (line 18))
Collecting futures>=3.0.3 (from -r PerfKitBenchmarker/requirements.txt (line 
19))
  Using cached 
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20)) (3.12)
Collecting pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached 
https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23)) (3.2.3.post2)
Collecting contextlib2>=0.5.1 (from -r PerfKitBenchmarker/requirements.txt 
(line 24))
  Using cached 
https://files.pythonhosted.org/packages/a2/71/8273a7eeed0aff6a854237ab5453bc9aa67deb49df4832801c21f0ff3782/contextlib2-0.5.5-py2.py3-none-any.whl
Collecting pywinrm (from -r PerfKitBenchmarker/requirements.txt (line 25))
  Using cached 
https://files.pythonhos

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #159

2018-05-12 Thread Apache Jenkins Server

See 


Changes:

[ekirpichov] [BEAM-3515] Portable translation of SplittableProcessKeyed

[ekirpichov] Fixes compile error in FlinkExecutableStageFunctionTest

[ekirpichov] Sickbays flaky test

--
[...truncated 642.47 KB...]
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.net.ConnectException: Call From 
xmlioit0writethenreadall--05120506-6bv8-harness-3gkx.c.apache-beam-testing.internal/10.128.0.5
 to 164.128.211.130.bc.googleusercontent.com:9000 failed on connection 
exception: java.net.ConnectException: Connection refused; For more details see: 
 http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:923)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at 
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
at 
org.

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT_HDFS #160

2018-05-12 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #161

2018-05-12 Thread Apache Jenkins Server

See 


Changes:

[ekirpichov] [BEAM-3515] Portable translation of SplittableProcessKeyed

[ekirpichov] Fixes compile error in FlinkExecutableStageFunctionTest

[ekirpichov] Sickbays flaky test

--
[...truncated 425.76 KB...]
INFO: No server chosen by WritableServerSelector from cluster description 
ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, 
all=[ServerDescription{address=104.197.110.43:27017, type=UNKNOWN, 
state=CONNECTING}]}. Waiting for 3 ms before timing out
May 12, 2018 12:12:30 PM com.mongodb.diagnostics.logging.SLF4JLogger info
INFO: Exception in monitor thread while connecting to server 
104.197.110.43:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
at com.mongodb.connection.SocketStream.open(SocketStream.java:63)
at 
com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:114)
at 
com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:128)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at 
com.mongodb.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:50)
at com.mongodb.connection.SocketStream.open(SocketStream.java:58)
... 3 more


Gradle Test Executor 1 finished executing tests.

> Task :beam-sdks-java-io-mongodb:integrationTest

org.apache.beam.sdk.io.mongodb.MongoDBIOIT > testWriteAndRead FAILED
java.lang.RuntimeException: com.mongodb.MongoSocketReadException: 
Prematurely reached end of stream
at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
at 
com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491)
at 
com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
at 
com.mongodb.connection.UsageTrackingInternalConnection.receiveMessage(UsageTrackingInternalConnection.java:102)
at 
com.mongodb.connection.DefaultConnectionPool$PooledConnection.receiveMessage(DefaultConnectionPool.java:435)
at 
com.mongodb.connection.WriteCommandProtocol.receiveMessage(WriteCommandProtocol.java:234)
at 
com.mongodb.connection.WriteCommandProtocol.execute(WriteCommandProtocol.java:104)
at 
com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:67)
at 
com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:37)
at 
com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:159)
at 
com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:286)
at 
com.mongodb.connection.DefaultServerConnection.insertCommand(DefaultServerConnection.java:115)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run$2.executeWriteCommandProtocol(MixedBulkWriteOperation.java:455)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run$RunExecutor.execute(MixedBulkWriteOperation.java:646)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run.execute(MixedBulkWriteOperation.java:401)
at 
com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:179)
at 
com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:230)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:221)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of clu

Build failed in Jenkins: beam_PerformanceTests_TextIOIT_HDFS #166

2018-05-12 Thread Apache Jenkins Server

See 


Changes:

[ekirpichov] [BEAM-3515] Portable translation of SplittableProcessKeyed

[ekirpichov] Fixes compile error in FlinkExecutableStageFunctionTest

[ekirpichov] Sickbays flaky test

--
[...truncated 480.10 KB...]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.net.ConnectException: Call From 
textioit0writethenreadall-05120506-jab7-harness-9xl2.c.apache-beam-testing.internal/10.128.0.7
 to 101.117.224.35.bc.googleusercontent.com:9000 failed on connection 
exception: java.net.ConnectException: Connection refused; For more details see: 
 http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:923)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at 
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
at org.apache.hadoop.ipc.Client.call(Client.java:1451)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.Pro

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1617

2018-05-12 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision bb319dbfaa81c27d7ba6b0a465994aa656143100 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f bb319dbfaa81c27d7ba6b0a465994aa656143100
Commit message: "Merge pull request #5277: [BEAM-3515] Portable translation of 
SplittableProcessKeyed"
 > git rev-list --no-walk bb319dbfaa81c27d7ba6b0a465994aa656143100 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins919856769656193695.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ekirpic...@gmail.com

[jira] [Work logged] (BEAM-4201) Integration Tests for PubsubIO

2018-05-12 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-4201?focusedWorklogId=101475&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101475
 ]

ASF GitHub Bot logged work on BEAM-4201:


Author: ASF GitHub Bot
Created on: 12/May/18 17:59
Start Date: 12/May/18 17:59
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5347: [BEAM-4201][SQL] Add 
integration test for PubsubIO JSON -> SQL
URL: https://github.com/apache/beam/pull/5347#issuecomment-388572306
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101475)
Time Spent: 50m  (was: 40m)

> Integration Tests for PubsubIO
> --
>
> Key: BEAM-4201
> URL: https://issues.apache.org/jira/browse/BEAM-4201
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Add integration tests for PubsubIO



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_Spark #1703

2018-05-12 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PerformanceTests_JDBC #564

2018-05-12 Thread Apache Jenkins Server

See 


--
[...truncated 91.05 KB...]
:beam-runners-google-cloud-dataflow-java:classes (Thread[Task worker for ':' 
Thread 7,5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for ':' 
Thread 7,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:compileTestJava' is 
02226f286cf0a774c1da080deed227de
Skipping task ':beam-sdks-java-io-google-cloud-platform:compileTestJava' as it 
is up-to-date.
:beam-sdks-java-io-google-cloud-platform:compileTestJava (Thread[Task worker 
for ':' Thread 5,5,main]) completed. Took 0.065 secs.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':',5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:testClasses UP-TO-DATE
Skipping task ':beam-sdks-java-io-google-cloud-platform:testClasses' as it has 
no actions.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':',5,main]) completed. Took 0.0 secs.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':',5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar' is 
928dcae609b653d5f7e124341b6e0b6f
Caching disabled for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-sdks-java-io-google-cloud-platform:shadowTestJar' as it is 
up-to-date.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':',5,main]) completed. Took 0.073 secs.

> Task :beam-runners-google-cloud-dataflow-java:shadowJar UP-TO-DATE
Build cache key for task ':beam-runners-google-cloud-dataflow-java:shadowJar' 
is 5150b836fda2131c59ad67534b32f918
Caching disabled for task ':beam-runners-google-cloud-dataflow-java:shadowJar': 
Caching has not been enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for ':' 
Thread 7,5,main]) completed. Took 0.084 secs.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 7,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:compileTestJava' is 
473644363180d8e4c4d48a688ea2ac2e
Skipping task ':beam-runners-google-cloud-dataflow-java:compileTestJava' as it 
is up-to-date.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 7,5,main]) completed. Took 0.061 secs.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 7,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:testClasses UP-TO-DATE
Skipping task ':beam-runners-google-cloud-dataflow-java:testClasses' as it has 
no actions.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 7,5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 7,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar' is 
fcc2f532355f1126b4a5f1667b99924f
Caching disabled for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowTestJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 7,5,main]) completed. Took 0.029 secs.
:beam-sdks-java-io-jdbc:compileTestJava (Thread[Task worker for ':' Thread 
7,5,main]) started.

> Task :beam-sdks-java-io-jdbc:compileTestJava UP-TO-DATE
Build cache key for task ':beam-sdks-java-io-jdbc:compileTestJava' is 
31b04fb18740c933b4f25f3f943c2f8f
Skipping task ':beam-sdks-java-io-jdbc:compileTestJava' as it is up-to-date.
:beam-sdks-java-io-jdbc:compileTestJava (Thread[Task worker for ':' Thread 
7,5,main]) completed. Took 0.046 secs.
:beam-sdks-java-io-jdbc:testClasses (Thread[Task worker for ':' Thread 
7,5,main]) started.

> Task :beam-sdks-java-io-jdbc:testClasses UP-TO-DATE
Skipping task ':beam-sdks-java-io-jdbc:testClasses' as it has no actions.
:beam-sdks-java-io-jdbc:testClasses (Thread[Task worker for ':' Thread 
7,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-io-jdbc:integrationTest (Thread[Task worker for ':' Thread 
15,5,main]) started.
Gradle Test Executor 1 started executing tests.

> Task :beam-sdks-java-io-jdbc:integrationTest
Build cache key for task ':beam-sdks-java-io-jdbc:integrationTest' is 
459ad0805

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT_HDFS #167

2018-05-12 Thread Apache Jenkins Server

See

Jenkins build is back to normal : beam_PerformanceTests_XmlIOIT_HDFS #160

2018-05-12 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #162

2018-05-12 Thread Apache Jenkins Server

See 


--
[...truncated 450.94 KB...]
at 
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:230)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:221)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
at 
org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:129)
at 
org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:200)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:383)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:355)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:286)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, servers=[{address=35.184.17.29:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(Clu

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT_HDFS #161

2018-05-12 Thread Apache Jenkins Server

See 


--
[...truncated 796.64 KB...]
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:923)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:138)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.AssignWindowsParDoFnFactory$AssignWindowsParDoFn.processElement(AssignWindowsParDoFnFactory.java:118)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
at 
org.apache.beam.sdk.io.avro.AvroIOIT$DeterministicallyConstructAvroRecordsFn.processElement(AvroIOIT.java:132)
at 
org.apache.beam.sdk.io.avro.AvroIOIT$DeterministicallyConstructAvroRecordsFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
at 
org.apache.beam.sdk.io.common.FileBasedIOITHelper$DeterministicallyConstructTestTextLineFn.processElement(FileBasedIOITHelper.java:77)
at 
org.apache.beam.sdk.io.common.FileBasedIOITHelper$DeterministicallyConstructTestTextLineFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:200)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:383)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:355)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:286)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(Datafl

[jira] [Work logged] (BEAM-4005) Futurize and fix python 2 compatibility for tools subpackage

2018-05-12 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-4005?focusedWorklogId=101486&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101486
 ]

ASF GitHub Bot logged work on BEAM-4005:


Author: ASF GitHub Bot
Created on: 12/May/18 18:34
Start Date: 12/May/18 18:34
Worklog Time Spent: 10m 
  Work Description: RobbeSneyders commented on a change in pull request 
#5343: [BEAM-4005] Futurize tools subpackage
URL: https://github.com/apache/beam/pull/5343#discussion_r187769527
 
 

 ##
 File path: sdks/python/apache_beam/tools/distribution_counter_microbenchmark.py
 ##
 @@ -23,11 +23,16 @@
   python -m apache_beam.tools.distribution_counter_microbenchmark
 """
 
+from __future__ import absolute_import
+from __future__ import division
 from __future__ import print_function
 
 import random
 import sys
 import time
+from builtins import range
+
+from past.utils import old_div
 
 Review comment:
   Can you replace old_div with the new Python 3 division?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101486)
Time Spent: 0.5h  (was: 20m)

> Futurize and fix python 2 compatibility for tools subpackage
> 
>
> Key: BEAM-4005
> URL: https://issues.apache.org/jira/browse/BEAM-4005
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4201) Integration Tests for PubsubIO

2018-05-12 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-4201?focusedWorklogId=101488&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101488
 ]

ASF GitHub Bot logged work on BEAM-4201:


Author: ASF GitHub Bot
Created on: 12/May/18 18:56
Start Date: 12/May/18 18:56
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5347: [BEAM-4201][SQL] Add 
integration test for PubsubIO JSON -> SQL
URL: https://github.com/apache/beam/pull/5347#issuecomment-388575678
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101488)
Time Spent: 1h  (was: 50m)

> Integration Tests for PubsubIO
> --
>
> Key: BEAM-4201
> URL: https://issues.apache.org/jira/browse/BEAM-4201
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Add integration tests for PubsubIO



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1618

2018-05-12 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision bb319dbfaa81c27d7ba6b0a465994aa656143100 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f bb319dbfaa81c27d7ba6b0a465994aa656143100
Commit message: "Merge pull request #5277: [BEAM-3515] Portable translation of 
SplittableProcessKeyed"
 > git rev-list --no-walk bb319dbfaa81c27d7ba6b0a465994aa656143100 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins8671460174012946620.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ekirpic...@gmail.com

Build failed in Jenkins: beam_PerformanceTests_TextIOIT_HDFS #168

2018-05-12 Thread Apache Jenkins Server

See 


--
[...truncated 50.92 KB...]
> Task :beam-sdks-java-maven-archetypes-starter:processResources
> Task :beam-sdks-java-build-tools:jar
> Task :beam-sdks-java-maven-archetypes-starter:processTestResources
> Task :beam-sdks-java-io-elasticsearch-tests-common:shadowJar
> Task :beam-sdks-java-io-elasticsearch-tests-common:processTestResources 
> NO-SOURCE
> Task :beam-sdks-java-io-elasticsearch-tests-common:jar
> Task :beam-sdks-java-maven-archetypes-examples:generateSources
> Task :beam-model-pipeline:extractIncludeProto
> Task :beam-model-pipeline:extractProto
> Task :beam-sdks-java-extensions-sql:copyFmppTemplatesFromCalciteCore
> Task :beam-sdks-java-extensions-sql:copyFmppTemplatesFromSrc
> Task :beam-sdks-java-maven-archetypes-examples:processResources
> Task :beam-sdks-java-maven-archetypes-examples:processTestResources
> Task :beam-model-pipeline:generateProto

> Task :beam-runners-apex:buildDependencyTree
See the report at: 
file://

> Task :beam-runners-apex:processResources NO-SOURCE
> Task :beam-runners-apex:processTestResources
> Task :beam-sdks-java-build-tools:compileTestJava FROM-CACHE
> Task :beam-sdks-java-build-tools:processTestResources NO-SOURCE
> Task :beam-sdks-java-build-tools:testClasses UP-TO-DATE
> Task :beam-sdks-java-build-tools:packageTests
> Task :beam-model-pipeline:compileJava FROM-CACHE
> Task :beam-model-pipeline:processResources
> Task :beam-model-pipeline:classes
> Task :beam-sdks-java-extensions-sql:generateFmppSources

> Task :beam-sdks-python:setupVirtualenv
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...
> Task :beam-sdks-java-build-tools:install

> Task :beam-sdks-java-extensions-sql:compileJavacc
Warning: Lookahead adequacy checking not being performed since option LOOKAHEAD 
is more than 1.  Set option FORCE_LA_CHECK to true to force checking.
File "TokenMgrError.java" does not exist.  Will create one.
File "ParseException.java" does not exist.  Will create one.
File "Token.java" does not exist.  Will create one.
File "SimpleCharStream.java" does not exist.  Will create one.
Parser generated with 0 errors and 1 warnings.

> Task :beam-sdks-java-core:compileJava FROM-CACHE

> Task :beam-sdks-python:setupVirtualenv
done.
Running virtualenv with interpreter /usr/bin/python2

> Task :beam-sdks-java-core:processResources
> Task :beam-sdks-java-core:classes
> Task :beam-model-pipeline:shadowJar
> Task :beam-model-job-management:extractIncludeProto
> Task :beam-model-fn-execution:extractIncludeProto
> Task :beam-sdks-java-extensions-sql:processResources
> Task :beam-sdks-java-extensions-sql:processTestResources NO-SOURCE
> Task :beam-model-pipeline:jar
> Task :beam-model-job-management:generateProto
> Task :beam-model-pipeline:extractIncludeTestProto
> Task :beam-model-pipeline:extractTestProto
> Task :beam-model-pipeline:generateTestProto NO-SOURCE
> Task :beam-model-pipeline:compileTestJava NO-SOURCE
> Task :beam-model-pipeline:processTestResources NO-SOURCE
> Task :beam-model-pipeline:testClasses UP-TO-DATE
> Task :beam-model-pipeline:packageTests
> Task :beam-model-fn-execution:generateProto
> Task :beam-model-pipeline:install
> Task :beam-model-job-management:compileJava FROM-CACHE
> Task :beam-model-job-management:classes
> Task :beam-model-fn-execution:compileJava FROM-CACHE
> Task :beam-model-fn-execution:classes

> Task :beam-sdks-python:setupVirtualenv
Requirement already up-to-date: tox==3.0.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (3.0.0)
Collecting grpcio-tools==1.3.5
  Using cached 
https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl
Requirement not upgraded as not directly required: pluggy<1.0,>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox==3.0.0) (0.6.0)
Requirement not upgraded as not directly required: six in 
/usr/local/lib/python2.7/dist-packages (from tox==3.0.0) (1.11.0)
Requirement not upgraded as not directly required: virtualenv>=1.11.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox==3.0.0) (15.2.0)
Requirement not upgraded as not directly required: py>=1.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox==3.0.0) (1.5.3)
Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)
  Using cached 
https://files.pythonhosted.org/packages/0d/54/b647a6323be6526be27b2c90bb0427

Build failed in Jenkins: beam_PerformanceTests_JDBC #565

2018-05-12 Thread Apache Jenkins Server

See 


--
[...truncated 91.51 KB...]
:beam-runners-google-cloud-dataflow-java:classes (Thread[Task worker for ':' 
Thread 6,5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for ':' 
Thread 6,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:compileTestJava' is 
1c5695627e26e73e313a48416020af01
Skipping task ':beam-sdks-java-io-google-cloud-platform:compileTestJava' as it 
is up-to-date.
:beam-sdks-java-io-google-cloud-platform:compileTestJava (Thread[Task worker 
for ':' Thread 12,5,main]) completed. Took 0.073 secs.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 12,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:testClasses UP-TO-DATE
Skipping task ':beam-sdks-java-io-google-cloud-platform:testClasses' as it has 
no actions.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 12,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 12,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:shadowJar UP-TO-DATE
Build cache key for task ':beam-runners-google-cloud-dataflow-java:shadowJar' 
is d45c99689620487fd0bfc5d92030ed62
Caching disabled for task ':beam-runners-google-cloud-dataflow-java:shadowJar': 
Caching has not been enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for ':' 
Thread 6,5,main]) completed. Took 0.042 secs.

> Task :beam-sdks-java-io-google-cloud-platform:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar' is 
16dfee8d92d7b4b512423ca303b9cc17
Caching disabled for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-sdks-java-io-google-cloud-platform:shadowTestJar' as it is 
up-to-date.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 12,5,main]) completed. Took 0.038 secs.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 12,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:compileTestJava' is 
4ad58b0c18bd403b9438589b4d0dd0de
Skipping task ':beam-runners-google-cloud-dataflow-java:compileTestJava' as it 
is up-to-date.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 12,5,main]) completed. Took 0.069 secs.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 12,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:testClasses UP-TO-DATE
Skipping task ':beam-runners-google-cloud-dataflow-java:testClasses' as it has 
no actions.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 12,5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 12,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar' is 
be1f057251aee0e50adb64237c1e636d
Caching disabled for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowTestJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 12,5,main]) completed. Took 0.034 secs.
:beam-sdks-java-io-jdbc:compileTestJava (Thread[Task worker for ':' Thread 
12,5,main]) started.

> Task :beam-sdks-java-io-jdbc:compileTestJava UP-TO-DATE
Build cache key for task ':beam-sdks-java-io-jdbc:compileTestJava' is 
f2a62f351931e49c42873a3a376dfc99
Skipping task ':beam-sdks-java-io-jdbc:compileTestJava' as it is up-to-date.
:beam-sdks-java-io-jdbc:compileTestJava (Thread[Task worker for ':' Thread 
12,5,main]) completed. Took 0.088 secs.
:beam-sdks-java-io-jdbc:testClasses (Thread[Task worker for ':' Thread 
12,5,main]) started.

> Task :beam-sdks-java-io-jdbc:testClasses UP-TO-DATE
Skipping task ':beam-sdks-java-io-jdbc:testClasses' as it has no actions.
:beam-sdks-java-io-jdbc:testClasses (Thread[Task worker for ':' Thread 
12,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-io-jdbc:integrationTest (Thread[Task worker for ':' Thread 
12,5,main]) started.
Gradle Test Executor 1 started executing tests.

> Task :beam-sdks-java-io-jdbc:integrationTest
Build cache key for task ':be

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #163

2018-05-12 Thread Apache Jenkins Server

See 


--
[...truncated 437.18 KB...]
at 
com.mongodb.connection.DefaultConnectionPool$PooledConnection.receiveMessage(DefaultConnectionPool.java:435)
at 
com.mongodb.connection.WriteCommandProtocol.receiveMessage(WriteCommandProtocol.java:234)
at 
com.mongodb.connection.WriteCommandProtocol.execute(WriteCommandProtocol.java:104)
at 
com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:67)
at 
com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:37)
at 
com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:159)
at 
com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:286)
at 
com.mongodb.connection.DefaultServerConnection.insertCommand(DefaultServerConnection.java:115)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run$2.executeWriteCommandProtocol(MixedBulkWriteOperation.java:455)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run$RunExecutor.execute(MixedBulkWriteOperation.java:646)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run.execute(MixedBulkWriteOperation.java:401)
at 
com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:179)
at 
com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:230)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:221)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, servers=[{address=104.198.23.164:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:219)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, servers=[{address=104.198.23.164:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:219)

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT_HDFS #162

2018-05-12 Thread Apache Jenkins Server

See 


--
[...truncated 630.99 KB...]
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:923)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:138)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.AssignWindowsParDoFnFactory$AssignWindowsParDoFn.processElement(AssignWindowsParDoFnFactory.java:118)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
at 
org.apache.beam.sdk.io.avro.AvroIOIT$DeterministicallyConstructAvroRecordsFn.processElement(AvroIOIT.java:132)
at 
org.apache.beam.sdk.io.avro.AvroIOIT$DeterministicallyConstructAvroRecordsFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
at 
org.apache.beam.sdk.io.common.FileBasedIOITHelper$DeterministicallyConstructTestTextLineFn.processElement(FileBasedIOITHelper.java:77)
at 
org.apache.beam.s

[jira] [Work logged] (BEAM-4201) Integration Tests for PubsubIO

2018-05-12 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-4201?focusedWorklogId=101503&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101503
 ]

ASF GitHub Bot logged work on BEAM-4201:


Author: ASF GitHub Bot
Created on: 13/May/18 02:33
Start Date: 13/May/18 02:33
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on a change in pull request 
#5347: [BEAM-4201][SQL] Add integration test for PubsubIO JSON -> SQL
URL: https://github.com/apache/beam/pull/5347#discussion_r187787527
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsubSignal.java
 ##
 @@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsub;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static java.util.stream.Collectors.toList;
+import static org.apache.beam.sdk.io.gcp.pubsub.TestPubsub.createTopicName;
+
+import com.google.common.collect.ImmutableSet;
+import io.grpc.StatusRuntimeException;
+import java.io.IOException;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.SubscriptionPath;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.TopicPath;
+import org.apache.beam.sdk.state.SetState;
+import org.apache.beam.sdk.state.StateSpec;
+import org.apache.beam.sdk.state.StateSpecs;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.transforms.WithKeys;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.POutput;
+import org.joda.time.DateTime;
+import org.joda.time.Duration;
+import org.junit.rules.TestRule;
+import org.junit.runner.Description;
+import org.junit.runners.model.Statement;
+
+/**
+ * Test rule which observes elements and checks whether they match the success 
creteria.
+ */
+public class TestPubsubSignal implements TestRule {
+  private static final String TOPIC_FORMAT = "projects/%s/topics/%s-result1";
+  private static final String SUBSCRIPTION_FORMAT = 
"projects/%s/subscriptions/%s";
+  private static final String NO_ID_ATTRIBUTE = null;
+  private static final String NO_TIMESTAMP_ATTRIBUTE = null;
+
+  PubsubClient pubsub;
+  private TestPubsubOptions pipelineOptions;
+  private String resultTopicPath;
+
+  /**
+   * Creates an instance of this rule.
+   *
+   * Loads GCP configuration from {@link TestPipelineOptions}.
+   */
+  public static TestPubsubSignal create() {
+TestPubsubOptions options =
+TestPipeline.testingPipelineOptions().as(TestPubsubOptions.class);
+return new TestPubsubSignal(options);
+  }
+
+  private TestPubsubSignal(TestPubsubOptions pipelineOptions) {
+this.pipelineOptions = pipelineOptions;
+  }
+
+  @Override
+  public Statement apply(Statement base, Description description) {
+return new Statement() {
+  @Override
+  public void evaluate() throws Throwable {
+if (TestPubsubSignal.this.pubsub != null) {
+  throw new AssertionError(
+  "Pubsub client was not shutdown in previous test. "
+  + "Topic path is'" + resultTopicPath + "'. "
+  + "Current test: " + description.getDisplayName());
+}
+
+try {
+  initializePubsub(description);
+  base.evaluate();
+} finally {
+  tearDown();
+}
+  }
+};
+  }
+
+  private void initializePubsub(Description description) throws IOException {
+pubsub = PubsubGrpcClient.FACTORY.newClient(
+NO_TIMESTAMP_ATTRIBUTE, NO_ID_ATTRIBUTE, pipelineOptions);
+
+// Example topic name:
+//
integ-test-TestClassName-testMethodName-2018-12-11-23-32-333--result
+String resultTopicPathT

[jira] [Work logged] (BEAM-4201) Integration Tests for PubsubIO

2018-05-12 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-4201?focusedWorklogId=101502&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101502
 ]

ASF GitHub Bot logged work on BEAM-4201:


Author: ASF GitHub Bot
Created on: 13/May/18 02:33
Start Date: 13/May/18 02:33
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on a change in pull request 
#5347: [BEAM-4201][SQL] Add integration test for PubsubIO JSON -> SQL
URL: https://github.com/apache/beam/pull/5347#discussion_r187787448
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java
 ##
 @@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsub;
+
+import static java.util.stream.Collectors.toList;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.TopicPath;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.joda.time.DateTime;
+import org.joda.time.Instant;
+import org.joda.time.format.DateTimeFormat;
+import org.joda.time.format.DateTimeFormatter;
+import org.junit.rules.TestRule;
+import org.junit.runner.Description;
+import org.junit.runners.model.Statement;
+
+/**
+ * Test rule which creates a new topic with randomized name and exposed a 
method to publishto it.
+ *
+ * Deletes topic on shutdown.
+ */
+public class TestPubsub implements TestRule {
+  private static final DateTimeFormatter DATETIME_FORMAT =
+  DateTimeFormat.forPattern("-MM-dd-HH-mm-ss-SSS");
+  private static final String TOPIC_FORMAT = "projects/%s/topics/%s";
+  private static final String TOPIC_PREFIX = "integ-test-";
+  private static final String NO_ID_ATTRIBUTE = null;
+  private static final String NO_TIMESTAMP_ATTRIBUTE = null;
+
+  PubsubClient pubsub;
+  private TestPubsubOptions pipelineOptions;
+  private String eventsTopicPath;
+
+  /**
+   * Creates an instance of this rule.
+   *
+   * Loads GCP configuration from {@link TestPipelineOptions}.
+   */
+  public static TestPubsub create() {
+TestPubsubOptions options =
+TestPipeline.testingPipelineOptions().as(TestPubsubOptions.class);
+return new TestPubsub(options);
+  }
+
+  private TestPubsub(TestPubsubOptions pipelineOptions) {
+this.pipelineOptions = pipelineOptions;
+  }
+
+  @Override
+  public Statement apply(Statement base, Description description) {
+return new Statement() {
+  @Override
+  public void evaluate() throws Throwable {
+if (TestPubsub.this.pubsub != null) {
+  throw new AssertionError(
+  "Pubsub client was not shutdown in previous test. "
+  + "Topic path is'" + eventsTopicPath + "'. "
+  + "Current test: " + description.getDisplayName());
+}
+
+try {
+  initializePubsub(description);
+  base.evaluate();
+} finally {
+  tearDown();
+}
+  }
+};
+  }
+
+  private void initializePubsub(Description description) throws IOException {
+pubsub = PubsubGrpcClient.FACTORY.newClient(
+NO_TIMESTAMP_ATTRIBUTE, NO_ID_ATTRIBUTE, pipelineOptions);
+String eventsTopicPathTmp =
+String.format(TOPIC_FORMAT, pipelineOptions.getProject(), 
createTopicName(description));
+
+pubsub.createTopic(new TopicPath(eventsTopicPathTmp));
+
+eventsTopicPath = eventsTopicPathTmp;
+  }
+
+  private void tearDown() throws IOException {
+if (pubsub == null) {
+  return;
+}
+
+try {
+  if (eventsTopicPath != null) {
+pubsub.deleteTopic(new TopicPath(eventsTopicPath));
+  }
+} finally {
+  pubsub.close();
+  pubsub = null;
+  eventsTopicPath = null;
+}
+  }
+
+  /**
+   * Generates randomized topic name.
+   *
+   * Example:
+   * 'TestClassName-testMethodName-2018-12-11-23-32-333-'
+   */
+  static String createTopicName(Description description) throws IOException {
+StringBuilder topicName = n

[jira] [Work logged] (BEAM-4201) Integration Tests for PubsubIO

2018-05-12 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-4201?focusedWorklogId=101501&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101501
 ]

ASF GitHub Bot logged work on BEAM-4201:


Author: ASF GitHub Bot
Created on: 13/May/18 02:33
Start Date: 13/May/18 02:33
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on a change in pull request 
#5347: [BEAM-4201][SQL] Add integration test for PubsubIO JSON -> SQL
URL: https://github.com/apache/beam/pull/5347#discussion_r187787452
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java
 ##
 @@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsub;
+
+import static java.util.stream.Collectors.toList;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.TopicPath;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.joda.time.DateTime;
+import org.joda.time.Instant;
+import org.joda.time.format.DateTimeFormat;
+import org.joda.time.format.DateTimeFormatter;
+import org.junit.rules.TestRule;
+import org.junit.runner.Description;
+import org.junit.runners.model.Statement;
+
+/**
+ * Test rule which creates a new topic with randomized name and exposed a 
method to publishto it.
 
 Review comment:
   typo: publishto it


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101501)
Time Spent: 1h 10m  (was: 1h)

> Integration Tests for PubsubIO
> --
>
> Key: BEAM-4201
> URL: https://issues.apache.org/jira/browse/BEAM-4201
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Add integration tests for PubsubIO



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4201) Integration Tests for PubsubIO

2018-05-12 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-4201?focusedWorklogId=101500&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101500
 ]

ASF GitHub Bot logged work on BEAM-4201:


Author: ASF GitHub Bot
Created on: 13/May/18 02:33
Start Date: 13/May/18 02:33
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on a change in pull request 
#5347: [BEAM-4201][SQL] Add integration test for PubsubIO JSON -> SQL
URL: https://github.com/apache/beam/pull/5347#discussion_r187787458
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java
 ##
 @@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsub;
+
+import static java.util.stream.Collectors.toList;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.TopicPath;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.joda.time.DateTime;
+import org.joda.time.Instant;
+import org.joda.time.format.DateTimeFormat;
+import org.joda.time.format.DateTimeFormatter;
+import org.junit.rules.TestRule;
+import org.junit.runner.Description;
+import org.junit.runners.model.Statement;
+
+/**
+ * Test rule which creates a new topic with randomized name and exposed a 
method to publishto it.
+ *
+ * Deletes topic on shutdown.
+ */
+public class TestPubsub implements TestRule {
 
 Review comment:
   This is super cool


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101500)
Time Spent: 1h 10m  (was: 1h)

> Integration Tests for PubsubIO
> --
>
> Key: BEAM-4201
> URL: https://issues.apache.org/jira/browse/BEAM-4201
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Add integration tests for PubsubIO



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4201) Integration Tests for PubsubIO

2018-05-12 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-4201?focusedWorklogId=101504&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101504
 ]

ASF GitHub Bot logged work on BEAM-4201:


Author: ASF GitHub Bot
Created on: 13/May/18 02:34
Start Date: 13/May/18 02:34
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5347: [BEAM-4201][SQL] 
Add integration test for PubsubIO JSON -> SQL
URL: https://github.com/apache/beam/pull/5347#issuecomment-388596659
 
 
   (btw this is awesome framework buliding)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101504)
Time Spent: 1h 20m  (was: 1h 10m)

> Integration Tests for PubsubIO
> --
>
> Key: BEAM-4201
> URL: https://issues.apache.org/jira/browse/BEAM-4201
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Add integration tests for PubsubIO



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1619

2018-05-12 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision bb319dbfaa81c27d7ba6b0a465994aa656143100 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f bb319dbfaa81c27d7ba6b0a465994aa656143100
Commit message: "Merge pull request #5277: [BEAM-3515] Portable translation of 
SplittableProcessKeyed"
 > git rev-list --no-walk bb319dbfaa81c27d7ba6b0a465994aa656143100 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins1788429669519605313.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ekirpic...@gmail.com

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #148

2018-05-12 Thread Apache Jenkins Server

See 


Changes:

[ekirpichov] [BEAM-3515] Portable translation of SplittableProcessKeyed

[ekirpichov] Fixes compile error in FlinkExecutableStageFunctionTest

[ekirpichov] Sickbays flaky test

--
[...truncated 1.39 KB...]
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesContainer_Dataflow] $ /bin/bash -xe 
/tmp/jenkins6042822098393815212.sh
+ cd src
+ bash sdks/python/container/run_validatescontainer.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# Where to store integration test outputs.
GCS_LOCATION=gs://temp-storage-for-end-to-end-tests

# Project for the container and integration test
PROJECT=apache-beam-testing

# Verify in the root of the repository
test -d sdks/python/container

# Verify docker and gcloud commands exist
command -v docker
/usr/bin/docker
command -v gcloud
/usr/bin/gcloud
docker -v
Docker version 17.05.0-ce, build 89658be
gcloud -v
Google Cloud SDK 191.0.0
alpha 2018.02.23
beta 2018.02.23
bq 2.0.29
core 2018.02.23
gsutil 4.28

# ensure gcloud is version 186 or above
TMPDIR=$(mktemp -d)
mktemp -d
gcloud_ver=$(gcloud -v | head -1 | awk '{print $4}')
gcloud -v | head -1 | awk '{print $4}'
if [[ "$gcloud_ver" < "186" ]]
then
  pushd $TMPDIR
  curl 
https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-186.0.0-linux-x86_64.tar.gz
 --output gcloud.tar.gz
  tar xf gcloud.tar.gz
  ./google-cloud-sdk/install.sh --quiet
  . ./google-cloud-sdk/path.bash.inc
  popd
  gcloud components update --quiet || echo 'gcloud components update failed'
  gcloud -v
fi

# Build the container
TAG=$(date +%Y%m%d-%H%M%S)
date +%Y%m%d-%H%M%S
CONTAINER=us.gcr.io/$PROJECT/$USER/python
echo "Using container $CONTAINER"
Using container us.gcr.io/apache-beam-testing/jenkins/python
./gradlew :beam-sdks-python-container:docker 
-Pdocker-repository-root=us.gcr.io/$PROJECT/$USER -Pdocker-tag=$TAG
Parallel execution with configuration on demand is an incubating feature.

> Configure project :
Applying build_rules.gradle to beam
createPerformanceTestHarness with default configuration for project beam
Adding 47 .gitignore exclusions to Apache Rat

> Configure project :beam-sdks-python-container
Applying build_rules.gradle to beam-sdks-python-container
applyGoNature with default configuration for project beam-sdks-python-container
applyDockerNature with default configuration for project 
beam-sdks-python-container
containerImageName with [name:python] for project beam-sdks-python-container

> Configure project :beam-sdks-go
Applying build_rules.gradle to beam-sdks-go
applyGoNature with default configuration for project beam-sdks-go

> Task :beam-sdks-go:prepare
Use project GOPATH: 


> Task :beam-sdks-python-container:prepare
Use project GOPATH: 


> Task :beam-sdks-python-container:dockerClean UP-TO-DATE

> Task :beam-sdks-python:setupVirtualenv
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
Collecting tox==3.0.0
  Using cached 
https://files.pythonhosted.org/packages/e6/41/4dcfd713282bf3213b0384320fa8841e4db032ddcb80bc08a540159d42a8/tox-3.0.0-py2.py3-none-any.whl
Collecting grpcio-tools==1.3.5
  Using cached 
https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)
  Using cached 
https://files.pythonhosted.org/packages/82/05/43e3947125a2137cba4746135c75934ceed1863f27e050fc560052104a71/pluggy-0.6.0-py2-none-any.whl
Requirement not upgraded as not directly required: six in 
/usr/local/lib/python2.7/dist-packages (from tox==3.0.0) (1.11.0)
Requirement not upgraded as not directly required: virtualenv>=1.11.2 in 
/usr/lib/python2.7/dist-packages (from tox==3.0.0) (15.0.1)
Collecting py>=1.4.17 (from tox==3.0.0)
  Using cached 
https://files.pythonhosted.org/packages/67/a5/f77982214dd4c8fd104b066f249adea2c49e25e8703d284382eb5e9ab35a/py-1.5.3-py2.py3-none-any.whl
Collecting grpcio>=1.3.5 (from grpcio-tools==

Build failed in Jenkins: beam_PerformanceTests_TextIOIT_HDFS #169

2018-05-12 Thread Apache Jenkins Server

See 


--
[...truncated 51.43 KB...]
> Task :beam-sdks-java-extensions-sql:copyFmppTemplatesFromSrc
> Task :beam-model-pipeline:extractIncludeProto
> Task :beam-model-pipeline:extractProto
> Task :beam-sdks-java-maven-archetypes-examples:processResources
> Task :beam-sdks-java-maven-archetypes-examples:processTestResources
> Task :beam-model-pipeline:generateProto

> Task :beam-runners-apex:buildDependencyTree
See the report at: 
file://

> Task :beam-runners-apex:processResources NO-SOURCE
> Task :beam-runners-apex:processTestResources

> Task :beam-sdks-python:setupVirtualenv
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...
> Task :beam-sdks-java-build-tools:compileTestJava FROM-CACHE
> Task :beam-sdks-java-build-tools:processTestResources NO-SOURCE
> Task :beam-sdks-java-build-tools:testClasses UP-TO-DATE
> Task :beam-sdks-java-build-tools:packageTests
> Task :beam-model-pipeline:compileJava FROM-CACHE
> Task :beam-model-pipeline:processResources
> Task :beam-model-pipeline:classes
> Task :beam-sdks-java-extensions-sql:generateFmppSources
> Task :beam-sdks-java-build-tools:install

> Task :beam-sdks-python:setupVirtualenv
done.
Running virtualenv with interpreter /usr/bin/python2

> Task :beam-sdks-java-core:compileJava FROM-CACHE
> Task :beam-sdks-java-core:processResources
> Task :beam-sdks-java-core:classes
> Task :beam-model-pipeline:shadowJar
> Task :beam-model-fn-execution:extractIncludeProto

> Task :beam-sdks-java-extensions-sql:compileJavacc
Java Compiler Compiler Version 4.0 (Parser Generator)
(type "javacc" with no arguments for help)
Warning: Bad option "-grammar_encoding=UTF-8" will be ignored.
Reading from file 

 . . .
Note: UNICODE_INPUT option is specified. Please make sure you create the 
parser/lexer using a Reader with the correct character encoding.
Warning: Lookahead adequacy checking not being performed since option LOOKAHEAD 
is more than 1.  Set option FORCE_LA_CHECK to true to force checking.
File "TokenMgrError.java" does not exist.  Will create one.
File "ParseException.java" does not exist.  Will create one.
File "Token.java" does not exist.  Will create one.
File "SimpleCharStream.java" does not exist.  Will create one.
Parser generated with 0 errors and 1 warnings.

> Task :beam-model-job-management:extractIncludeProto
> Task :beam-model-pipeline:jar
> Task :beam-model-job-management:generateProto
> Task :beam-model-fn-execution:generateProto
> Task :beam-model-pipeline:extractIncludeTestProto
> Task :beam-model-pipeline:extractTestProto
> Task :beam-model-pipeline:generateTestProto NO-SOURCE
> Task :beam-model-pipeline:compileTestJava NO-SOURCE
> Task :beam-model-pipeline:processTestResources NO-SOURCE
> Task :beam-model-pipeline:testClasses UP-TO-DATE
> Task :beam-model-pipeline:packageTests
> Task :beam-model-pipeline:install
> Task :beam-model-fn-execution:compileJava FROM-CACHE
> Task :beam-model-fn-execution:classes
> Task :beam-model-job-management:compileJava FROM-CACHE
> Task :beam-model-job-management:classes

> Task :beam-sdks-python:setupVirtualenv
Requirement already up-to-date: tox==3.0.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (3.0.0)
Collecting grpcio-tools==1.3.5
  Using cached 
https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl

> Task :beam-sdks-java-extensions-sql:processResources
> Task :beam-sdks-java-extensions-sql:processTestResources NO-SOURCE

> Task :beam-sdks-python:setupVirtualenv
Requirement not upgraded as not directly required: pluggy<1.0,>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox==3.0.0) (0.6.0)
Requirement not upgraded as not directly required: six in 
/usr/local/lib/python2.7/dist-packages (from tox==3.0.0) (1.11.0)
Requirement not upgraded as not directly required: virtualenv>=1.11.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox==3.0.0) (15.2.0)
Requirement not upgraded as not directly required: py>=1.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox==3.0.0) (1.5.3)
Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)
  Using cached 
https://files.pythonhosted.org/packages/0d/54/b647a6323be6526be27b2c90b

Build failed in Jenkins: beam_PerformanceTests_JDBC #566

2018-05-12 Thread Apache Jenkins Server

See 


--
[...truncated 91.86 KB...]
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 12,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 12,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:compileJava UP-TO-DATE
Build cache key for task ':beam-runners-google-cloud-dataflow-java:compileJava' 
is f1530a6e2694664305ca42cb327c8d52
Skipping task ':beam-runners-google-cloud-dataflow-java:compileJava' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:compileJava (Thread[Task worker for 
':' Thread 10,5,main]) completed. Took 0.075 secs.
:beam-runners-google-cloud-dataflow-java:classes (Thread[Task worker for ':' 
Thread 10,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:classes UP-TO-DATE
Skipping task ':beam-runners-google-cloud-dataflow-java:classes' as it has no 
actions.
:beam-runners-google-cloud-dataflow-java:classes (Thread[Task worker for ':' 
Thread 10,5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for ':' 
Thread 10,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar' is 
119f060228d65affa697ad3137d553a0
Caching disabled for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-sdks-java-io-google-cloud-platform:shadowTestJar' as it is 
up-to-date.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 12,5,main]) completed. Took 0.041 secs.

> Task :beam-runners-google-cloud-dataflow-java:shadowJar UP-TO-DATE
Build cache key for task ':beam-runners-google-cloud-dataflow-java:shadowJar' 
is 1b38f6d053082fefc4b77af54285fc00
Caching disabled for task ':beam-runners-google-cloud-dataflow-java:shadowJar': 
Caching has not been enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for ':' 
Thread 10,5,main]) completed. Took 0.039 secs.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 10,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:compileTestJava' is 
559b8b1c3d9d75e964583184d42f719d
Skipping task ':beam-runners-google-cloud-dataflow-java:compileTestJava' as it 
is up-to-date.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 10,5,main]) completed. Took 0.069 secs.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 10,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:testClasses UP-TO-DATE
Skipping task ':beam-runners-google-cloud-dataflow-java:testClasses' as it has 
no actions.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 10,5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 10,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar' is 
07f2020535102bed4787f6bcd23c21bb
Caching disabled for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowTestJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 10,5,main]) completed. Took 0.039 secs.
:beam-sdks-java-io-jdbc:compileTestJava (Thread[Task worker for ':' Thread 
10,5,main]) started.

> Task :beam-sdks-java-io-jdbc:compileTestJava UP-TO-DATE
Build cache key for task ':beam-sdks-java-io-jdbc:compileTestJava' is 
dcad884ce041c74402b809bfeaa2d7b5
Skipping task ':beam-sdks-java-io-jdbc:compileTestJava' as it is up-to-date.
:beam-sdks-java-io-jdbc:compileTestJava (Thread[Task worker for ':' Thread 
10,5,main]) completed. Took 0.098 secs.
:beam-sdks-java-io-jdbc:testClasses (Thread[Task worker for ':' Thread 
10,5,main]) started.

> Task :beam-sdks-java-io-jdbc:testClasses UP-TO-DATE
Skipping task ':beam-sdks-java-io-jdbc:testClasses' as it has no actions.
:beam-sdks-java-io-jdbc:testClasses (Thread[Task worker for ':' Thread 
10,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-io-jdbc:integrationTest (Thread[Task worker for ':' Thread 
10,5,main]) started.
Gradle Test Executor 1 started executing tests.

> Task :beam-sdks-java-io-jdbc:integrationTest
Build cache key for task ':beam-sdks-java-io-jdbc:integ

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #164

2018-05-12 Thread Apache Jenkins Server

See 


--
[...truncated 451.97 KB...]
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
at 
org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:129)
at 
org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:200)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:383)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:355)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:286)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, servers=[{address=35.232.89.144:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:219)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, servers=[{address=35.232.89.144:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.g

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT_HDFS #163

2018-05-12 Thread Apache Jenkins Server

See 


--
[...truncated 785.63 KB...]
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:923)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at 
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
at org.apache.hadoop.ipc.Client.call(Client.java:1451)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.crea

42 matches

Mail list logo