Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1512

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam23 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision e0567246354defdef3abc0b41feb8d846fa647b0 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e0567246354defdef3abc0b41feb8d846fa647b0
Commit message: "Merge pull request #5234: Add KV and Window Coder Utilities to 
ModelCoders"
 > git rev-list --no-walk e0567246354defdef3abc0b41feb8d846fa647b0 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins4626257764703174276.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user ankurgoe...@gmail.com
Not sending mail to unregistered user mair...@google.com

[jira] [Work logged] (BEAM-4071) Java: PortableRunner - p.run() via given JobService

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=97752=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97752
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 03/May/18 03:06
Start Date: 03/May/18 03:06
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on a change in pull request 
#5150:  [BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r185689507
 
 

 ##
 File path: runners/reference/java/build.gradle
 ##
 @@ -27,8 +27,11 @@ framework to execute user-definied functions."""
 dependencies {
   shadow project(path: ":beam-model-pipeline", configuration: "shadow")
   shadow project(path: ":beam-runners-core-construction-java", configuration: 
"shadow")
+  shadow project(path: ":beam-sdks-java-fn-execution", configuration: "shadow")
   shadow library.java.slf4j_api
-  testCompile library.java.junit
+  shadowTest project(path: ":beam-runners-core-construction-java", 
configuration: "shadowTest")
 
 Review comment:
   Yes - https://issues.apache.org/jira/browse/BEAM-3138


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97752)
Time Spent: 19h 10m  (was: 19h)

> Java: PortableRunner - p.run() via given JobService
> ---
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
> Fix For: 2.5.0
>
>  Time Spent: 19h 10m
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a portable job 
> server. The job server itself is expected to be started up out-of-band. The 
> "PortableRunner" should take an option indicating the Job API endpoint and 
> defer other runner configurations to the backend itself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4071) Java: PortableRunner - p.run() via given JobService

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=97754=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97754
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 03/May/18 03:06
Start Date: 03/May/18 03:06
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on a change in pull request 
#5150:  [BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r185689566
 
 

 ##
 File path: runners/reference/java/build.gradle
 ##
 @@ -27,8 +27,11 @@ framework to execute user-definied functions."""
 dependencies {
   shadow project(path: ":beam-model-pipeline", configuration: "shadow")
   shadow project(path: ":beam-runners-core-construction-java", configuration: 
"shadow")
+  shadow project(path: ":beam-sdks-java-fn-execution", configuration: "shadow")
   shadow library.java.slf4j_api
-  testCompile library.java.junit
+  shadowTest project(path: ":beam-runners-core-construction-java", 
configuration: "shadowTest")
 
 Review comment:
   For now, IMO it is fine to just move it to `src/main/java` and mark 
`@Internal` if you don't want to do any other edits.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97754)
Time Spent: 19h 20m  (was: 19h 10m)

> Java: PortableRunner - p.run() via given JobService
> ---
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
> Fix For: 2.5.0
>
>  Time Spent: 19h 20m
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a portable job 
> server. The job server itself is expected to be started up out-of-band. The 
> "PortableRunner" should take an option indicating the Job API endpoint and 
> defer other runner configurations to the backend itself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #265

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add KV and Window Coder Utilities to ModelCoders

[tgroh] Use ModelCoders in ProcessBundleDescriptors

--
[...truncated 18.84 MB...]
INFO: 2018-05-03T02:50:14.344Z: Unzipping flatten s18-u63 for input 
s19.output-c61
May 03, 2018 2:50:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:50:14.375Z: Fusing unzipped copy of 
PAssert$3/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous), through 
flatten s18-u63, into producer 
PAssert$3/CreateActual/FilterActuals/Window.Assign
May 03, 2018 2:50:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:50:14.413Z: Fusing consumer 
PAssert$3/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous) into 
PAssert$3/CreateActual/FilterActuals/Window.Assign
May 03, 2018 2:50:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:50:14.444Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 03, 2018 2:50:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:50:14.482Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 03, 2018 2:50:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:50:14.521Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
May 03, 2018 2:50:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:50:14.550Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
May 03, 2018 2:50:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:50:14.587Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
May 03, 2018 2:50:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:50:14.624Z: Fusing consumer 
PAssert$3/CreateActual/RewindowActuals/Window.Assign into 
PAssert$3/CreateActual/Flatten.Iterables/FlattenIterables/FlatMap
May 03, 2018 2:50:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:50:14.651Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
May 03, 2018 2:50:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:50:14.684Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
May 03, 2018 2:50:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:50:14.712Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
 into 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #264

[Apache Jenkins Server]

See 


Changes:

[sidhom] Remove StateRequestHandler from JobBundleFactory

--
[...truncated 19.17 MB...]
May 03, 2018 2:38:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:38:55.634Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
May 03, 2018 2:38:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:38:55.663Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
May 03, 2018 2:38:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:38:55.696Z: Fusing consumer 
PAssert$3/CreateActual/ParDo(Anonymous) into 
PAssert$3/CreateActual/RewindowActuals/Window.Assign
May 03, 2018 2:38:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:38:55.728Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into PAssert$3/CreateActual/ParDo(Anonymous)
May 03, 2018 2:38:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:38:55.765Z: Fusing consumer 
Combine.globally(Count)/ProduceDefault into 
Combine.globally(Count)/CreateVoid/Read(CreateSource)
May 03, 2018 2:38:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:38:55.797Z: Fusing consumer 
Combine.globally(Count)/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow) into 
Combine.globally(Count)/Values/Values/Map
May 03, 2018 2:38:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:38:55.829Z: Fusing consumer 
Combine.globally(Count)/Combine.perKey(Count)/Combine.GroupedValues into 
Combine.globally(Count)/Combine.perKey(Count)/GroupByKey/Read
May 03, 2018 2:38:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:38:55.855Z: Fusing consumer 
Combine.globally(Count)/Combine.perKey(Count)/GroupByKey+Combine.globally(Count)/Combine.perKey(Count)/Combine.GroupedValues/Partial
 into Combine.globally(Count)/WithKeys/AddKeys/Map
May 03, 2018 2:38:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:38:55.876Z: Fusing consumer 
DatastoreV1.Read/Reshuffle/Reshuffle/GroupByKey/GroupByWindow into 
DatastoreV1.Read/Reshuffle/Reshuffle/GroupByKey/Read
May 03, 2018 2:38:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:38:55.911Z: Fusing consumer 
Combine.globally(Count)/Combine.perKey(Count)/GroupByKey/Reify into 
Combine.globally(Count)/Combine.perKey(Count)/GroupByKey+Combine.globally(Count)/Combine.perKey(Count)/Combine.GroupedValues/Partial
May 03, 2018 2:38:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:38:55.945Z: Fusing consumer 
Combine.globally(Count)/WithKeys/AddKeys/Map into DatastoreV1.Read/Read
May 03, 2018 2:38:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:38:55.982Z: Fusing consumer 
DatastoreV1.Read/Reshuffle/Pair with random key into DatastoreV1.Read/Split
May 03, 2018 2:38:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:38:56.017Z: Fusing consumer 
DatastoreV1.Read/Reshuffle/Reshuffle/GroupByKey/Write into 
DatastoreV1.Read/Reshuffle/Reshuffle/GroupByKey/Reify
May 03, 2018 2:38:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:38:56.051Z: Fusing consumer 
DatastoreV1.Read/Reshuffle/Reshuffle/GroupByKey/Reify into 
DatastoreV1.Read/Reshuffle/Reshuffle/Window.Into()/Window.Assign
May 03, 2018 2:38:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T02:38:56.085Z: Fusing consumer 
Combine.globally(Count)/Values/Values/Map into 
Combine.globally(Count)/Combine.perKey(Count)/Combine.GroupedValues/Extract
May 03, 2018 2:38:57 AM 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1511

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add KV and Window Coder Utilities to ModelCoders

[tgroh] Use ModelCoders in ProcessBundleDescriptors

--
Started by GitHub push by tgroh
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision e0567246354defdef3abc0b41feb8d846fa647b0 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e0567246354defdef3abc0b41feb8d846fa647b0
Commit message: "Merge pull request #5234: Add KV and Window Coder Utilities to 
ModelCoders"
 > git rev-list --no-walk 04b8b44d96b457501df7b6f30280f267a4825a3b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins1257674018655684362.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user ankurgoe...@gmail.com
Not sending mail to unregistered user mair...@google.com

[beam] branch master updated (04b8b44 -> e056724)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 04b8b44  Merge pull request #5263: Remove StateRequestHandler from 
JobBundleFactory
 add fffbdaa  Add KV and Window Coder Utilities to ModelCoders
 add 3a33483  Use ModelCoders in ProcessBundleDescriptors
 new e056724  Merge pull request #5234: Add KV and Window Coder Utilities 
to ModelCoders

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../pipeline/src/main/proto/beam_runner_api.proto  |  10 ++
 .../core/construction/ModelCoderRegistrar.java |   5 +
 .../runners/core/construction/ModelCoders.java |  52 +
 .../runners/core/construction/ModelCodersTest.java | 120 +
 .../control/ProcessBundleDescriptors.java  |  22 ++--
 5 files changed, 193 insertions(+), 16 deletions(-)
 create mode 100644 
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ModelCodersTest.java

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1510

[Apache Jenkins Server]

See 


Changes:

[sidhom] Remove StateRequestHandler from JobBundleFactory

--
Started by GitHub push by tgroh
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 04b8b44d96b457501df7b6f30280f267a4825a3b (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 04b8b44d96b457501df7b6f30280f267a4825a3b
Commit message: "Merge pull request #5263: Remove StateRequestHandler from 
JobBundleFactory"
 > git rev-list --no-walk 6ac67f75eceb54ebe9990bc77f93c09902c28e00 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins1973148569305293894.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user ankurgoe...@gmail.com
Not sending mail to unregistered user mair...@google.com

[beam] branch master updated (6ac67f7 -> 04b8b44)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 6ac67f7  Merge pull request #5259: SpannerIO fixes
 add 5464996  Remove StateRequestHandler from JobBundleFactory
 new 04b8b44  Merge pull request #5263: Remove StateRequestHandler from 
JobBundleFactory

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../org/apache/beam/runners/fnexecution/control/JobBundleFactory.java | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] 01/01: Merge pull request #5263: Remove StateRequestHandler from JobBundleFactory

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 04b8b44d96b457501df7b6f30280f267a4825a3b
Merge: 6ac67f7 5464996
Author: Thomas Groh 
AuthorDate: Wed May 2 18:51:30 2018 -0700

Merge pull request #5263: Remove StateRequestHandler from JobBundleFactory

 .../org/apache/beam/runners/fnexecution/control/JobBundleFactory.java | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #263

[Apache Jenkins Server]

See 


--
[...truncated 18.83 MB...]
May 03, 2018 12:52:16 AM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.FilterExamplesTest > testProjectionFn 
STANDARD_ERROR
May 03, 2018 12:52:16 AM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.JoinExamplesTest > testExtractCountryInfoFn 
STANDARD_ERROR
May 03, 2018 12:52:16 AM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.JoinExamplesTest > testExtractEventDataFn 
STANDARD_ERROR
May 03, 2018 12:52:17 AM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.DebuggingWordCountTest > testDebuggingWordCount 
STANDARD_ERROR
May 03, 2018 12:52:17 AM org.apache.beam.sdk.io.FileBasedSource 
getEstimatedSizeBytes
INFO: Filepattern 
/tmp/junit8703048259214916703/junit6687294031692171187.tmp matched 1 files with 
total size 54
May 03, 2018 12:52:17 AM org.apache.beam.sdk.io.FileBasedSource split
INFO: Splitting filepattern 
/tmp/junit8703048259214916703/junit6687294031692171187.tmp into bundles of size 
3 took 1 ms and produced 1 files and 18 bundles

org.apache.beam.examples.WordCountTest > testExtractWordsFn STANDARD_ERROR
May 03, 2018 12:52:17 AM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.subprocess.ExampleEchoPipelineTest > 
testExampleEchoPipeline STANDARD_ERROR
May 03, 2018 12:52:19 AM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-Echo1409110507523819686.sh 
May 03, 2018 12:52:19 AM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
May 03, 2018 12:52:19 AM 
org.apache.beam.examples.subprocess.utils.FileUtils copyFileFromGCSToWorker
INFO: Moving File /tmp/test-Echo1409110507523819686.sh to 
/tmp/test-Echoo4606077102089523250/test-Echo1409110507523819686.sh 
May 03, 2018 12:52:19 AM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-EchoAgain2873987014956674910.sh 
May 03, 2018 12:52:19 AM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
May 03, 2018 12:52:19 AM 
org.apache.beam.examples.subprocess.utils.FileUtils copyFileFromGCSToWorker
INFO: Moving File /tmp/test-EchoAgain2873987014956674910.sh to 
/tmp/test-Echoo4606077102089523250/test-EchoAgain2873987014956674910.sh 

org.apache.beam.examples.complete.game.HourlyTeamScoreTest > 
testUserScoresFilter STANDARD_OUT
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,12,144795563,2015-11-19 
09:53:53.444
GOT user19_BisqueBilby,BisqueBilby,8,144795563,2015-11-19 09:53:53.444
GOT user2_AmberCockatoo,AmberCockatoo,13,144796569,2015-11-19 
12:41:31.053
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,11,144795563,2015-11-19 
09:53:53.444
GOT 
user0_AndroidGreenEchidna,AndroidGreenEchidna,0,144796569,2015-11-19 
12:41:31.053
GOT user3_BananaEmu,BananaEmu,17,144796569,2015-11-19 12:41:31.053
GOT user0_MagentaKangaroo,MagentaKangaroo,4,144796569,2015-11-19 
12:41:31.053
GOT user6_AmberNumbat,AmberNumbat,11,144795563,2015-11-19 09:53:53.444
GOT user0_MagentaKangaroo,MagentaKangaroo,3,144795563,2015-11-19 
09:53:53.444
GOT user7_AlmondWallaby,AlmondWallaby,15,144795563,2015-11-19 
09:53:53.444
GOT user18_BananaEmu,BananaEmu,7,144796569,2015-11-19 12:41:31.053
GOT user18_ApricotCaneToad,ApricotCaneToad,14,144796569,2015-11-19 
12:41:31.053
GOT user18_BananaEmu,BananaEmu,1,144796569,2015-11-19 12:41:31.053
GOT user19_BisqueBilby,BisqueBilby,6,144795563,2015-11-19 09:53:53.444
GOT user13_ApricotQuokka,ApricotQuokka,15,144795563,2015-11-19 
09:53:53.444

org.apache.beam.examples.complete.game.UserScoreTest > testTeamScoreSums 
STANDARD_OUT
GOT user7_AlmondWallaby,AlmondWallaby,15,144795563,2015-11-19 
09:53:53.444
GOT user13_ApricotQuokka,ApricotQuokka,15,144795563,2015-11-19 
09:53:53.444
GOT user0_MagentaKangaroo,MagentaKangaroo,3,144795563,2015-11-19 
09:53:53.444

[jira] [Work logged] (BEAM-3972) Flink runner translates batch pipelines directly by proto

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3972?focusedWorklogId=97736=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97736
 ]

ASF GitHub Bot logged work on BEAM-3972:


Author: ASF GitHub Bot
Created on: 03/May/18 00:42
Start Date: 03/May/18 00:42
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5226: 
[BEAM-3972] Translate portable batch pipelines by proto
URL: https://github.com/apache/beam/pull/5226#discussion_r185677199
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/wire/LengthPrefixUnknownCoders.java
 ##
 @@ -14,7 +14,7 @@
  * the License.
  */
 
-package org.apache.beam.runners.fnexecution.graph;
+package org.apache.beam.runners.fnexecution.wire;
 
 Review comment:
   @jkff I just looked at the past runs under 
https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/.
 It looks like _every_ recent run has either been aborted or failed due to 
worker timeout issues.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97736)
Time Spent: 8.5h  (was: 8h 20m)

> Flink runner translates batch pipelines directly by proto
> -
>
> Key: BEAM-3972
> URL: https://issues.apache.org/jira/browse/BEAM-3972
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Major
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> The non-portable runner uses reydrated pipelines which lack necessary 
> information. The portable Flink runner needs to translate pipelines directly 
> by proto in order to wire components into individual executable stages 
> correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT_HDFS #122

[Apache Jenkins Server]

See 


Changes:

[axelmagn] Create ArtifactSource Interface

[tgroh] Remove unused Test Utilities, Test

[axelmagn] Create RemoteBundle interface

[axelmagn] Create JobInfo for job declaration by operators.

[mairbek] Wait for input collection

[mairbek] Expose DatabaseAdminClient

[mairbek] Consistently use lower case table name and column names

[axelmagn] Create BundleFactory chain.

[mairbek] Addressed comments

--
[...truncated 980.82 KB...]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.net.ConnectException: Call From 
textioit0writethenreadall-05021720-mrn3-harness-0cgr.c.apache-beam-testing.internal/10.128.0.52
 to 39.6.225.35.bc.googleusercontent.com:9000 failed on connection exception: 
java.net.ConnectException: Connection refused; For more details see:  
http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:923)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at 
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT_HDFS #122

[Apache Jenkins Server]

See 


Changes:

[axelmagn] Create ArtifactSource Interface

[tgroh] Remove unused Test Utilities, Test

[axelmagn] Create RemoteBundle interface

[axelmagn] Create JobInfo for job declaration by operators.

[mairbek] Wait for input collection

[mairbek] Expose DatabaseAdminClient

[mairbek] Consistently use lower case table name and column names

[axelmagn] Create BundleFactory chain.

[mairbek] Addressed comments

--
[...truncated 877.57 KB...]
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:923)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:138)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.AssignWindowsParDoFnFactory$AssignWindowsParDoFn.processElement(AssignWindowsParDoFnFactory.java:118)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
at 
org.apache.beam.sdk.io.avro.AvroIOIT$DeterministicallyConstructAvroRecordsFn.processElement(AvroIOIT.java:132)
at 
org.apache.beam.sdk.io.avro.AvroIOIT$DeterministicallyConstructAvroRecordsFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
at 
org.apache.beam.sdk.io.common.FileBasedIOITHelper$DeterministicallyConstructTestTextLineFn.processElement(FileBasedIOITHelper.java:77)
at 
org.apache.beam.sdk.io.common.FileBasedIOITHelper$DeterministicallyConstructTestTextLineFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:200)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:383)
at 

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #123

[Apache Jenkins Server]

See 


Changes:

[axelmagn] Create ArtifactSource Interface

[tgroh] Remove unused Test Utilities, Test

[axelmagn] Create RemoteBundle interface

[axelmagn] Create JobInfo for job declaration by operators.

[mairbek] Wait for input collection

[mairbek] Expose DatabaseAdminClient

[mairbek] Consistently use lower case table name and column names

[axelmagn] Create BundleFactory chain.

[mairbek] Addressed comments

--
[...truncated 438.74 KB...]
at 
com.mongodb.connection.DefaultConnectionPool$PooledConnection.receiveMessage(DefaultConnectionPool.java:435)
at 
com.mongodb.connection.WriteCommandProtocol.receiveMessage(WriteCommandProtocol.java:234)
at 
com.mongodb.connection.WriteCommandProtocol.execute(WriteCommandProtocol.java:104)
at 
com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:67)
at 
com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:37)
at 
com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:159)
at 
com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:286)
at 
com.mongodb.connection.DefaultServerConnection.insertCommand(DefaultServerConnection.java:115)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run$2.executeWriteCommandProtocol(MixedBulkWriteOperation.java:455)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run$RunExecutor.execute(MixedBulkWriteOperation.java:646)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run.execute(MixedBulkWriteOperation.java:401)
at 
com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:179)
at 
com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:230)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:221)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, servers=[{address=35.202.118.20:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:219)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, servers=[{address=35.202.118.20:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at 

Jenkins build is back to normal : beam_PerformanceTests_XmlIOIT_HDFS #121

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_JDBC #525

[Apache Jenkins Server]

See 


Changes:

[axelmagn] Create ArtifactSource Interface

[tgroh] Remove unused Test Utilities, Test

[axelmagn] Create RemoteBundle interface

[axelmagn] Create JobInfo for job declaration by operators.

[mairbek] Wait for input collection

[mairbek] Expose DatabaseAdminClient

[mairbek] Consistently use lower case table name and column names

[axelmagn] Create BundleFactory chain.

[mairbek] Addressed comments

--
[...truncated 90.85 KB...]
:beam-runners-google-cloud-dataflow-java:classes (Thread[Task worker for 
':',5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for 
':',5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:compileTestJava' is 
629effcbd69261b43d4aba4280d2c301
Skipping task ':beam-sdks-java-io-google-cloud-platform:compileTestJava' as it 
is up-to-date.
:beam-sdks-java-io-google-cloud-platform:compileTestJava (Thread[Task worker 
for ':' Thread 8,5,main]) completed. Took 0.117 secs.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 8,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:testClasses UP-TO-DATE
Skipping task ':beam-sdks-java-io-google-cloud-platform:testClasses' as it has 
no actions.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 8,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 9,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:shadowJar UP-TO-DATE
Build cache key for task ':beam-runners-google-cloud-dataflow-java:shadowJar' 
is 87ee918606014c13f3f0eae3d81f0235
Caching disabled for task ':beam-runners-google-cloud-dataflow-java:shadowJar': 
Caching has not been enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for 
':',5,main]) completed. Took 0.078 secs.

> Task :beam-sdks-java-io-google-cloud-platform:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar' is 
65f206f16bb9c424e52cc6e1caa85cd9
Caching disabled for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-sdks-java-io-google-cloud-platform:shadowTestJar' as it is 
up-to-date.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 9,5,main]) completed. Took 0.029 secs.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 9,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:compileTestJava' is 
015e9f24a71a65762df2966a8fc346a9
Skipping task ':beam-runners-google-cloud-dataflow-java:compileTestJava' as it 
is up-to-date.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 9,5,main]) completed. Took 0.064 secs.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 9,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:testClasses UP-TO-DATE
Skipping task ':beam-runners-google-cloud-dataflow-java:testClasses' as it has 
no actions.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 9,5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 9,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar' is 
559b494056880265a77918fc818fb91d
Caching disabled for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowTestJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 9,5,main]) completed. Took 0.049 secs.
:beam-sdks-java-io-jdbc:compileTestJava (Thread[Task worker for ':' Thread 
9,5,main]) started.

> Task :beam-sdks-java-io-jdbc:compileTestJava UP-TO-DATE
Build cache key for task ':beam-sdks-java-io-jdbc:compileTestJava' is 
8560102795c58fbb2c9ca087736606c8
Skipping task ':beam-sdks-java-io-jdbc:compileTestJava' as it is up-to-date.
:beam-sdks-java-io-jdbc:compileTestJava (Thread[Task worker for ':' Thread 
9,5,main]) completed. Took 0.064 secs.
:beam-sdks-java-io-jdbc:testClasses (Thread[Task worker for ':' Thread 
9,5,main]) started.

> Task :beam-sdks-java-io-jdbc:testClasses UP-TO-DATE
Skipping task 

[jira] [Commented] (BEAM-4228) The FlinkRunner shouldn't require all of the values for a key to fit in memory

[Ben Sidhom (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461769#comment-16461769
 ] 

Ben Sidhom commented on BEAM-4228:
--

For context, see [https://github.com/apache/beam/pull/5226/files#r185652571.]

> The FlinkRunner shouldn't require all of the values for a key to fit in memory
> --
>
> Key: BEAM-4228
> URL: https://issues.apache.org/jira/browse/BEAM-4228
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Thomas Groh
>Priority: Major
>
> The use of a reducer that adds all of the elements that it consumes to a list 
> is the primary way in which this occurs - if instead, we produce a filtered 
> iterable, or a collection of filtered iterables, we can lazily iterate over 
> all of the contained elements without having to buffer all of the elements.
>  
> For an example of where this occurs, see {{Concatenate}} in  
> {{FlinkBatchPortablePipelineTranslator}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3972) Flink runner translates batch pipelines directly by proto

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3972?focusedWorklogId=97734=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97734
 ]

ASF GitHub Bot logged work on BEAM-3972:


Author: ASF GitHub Bot
Created on: 03/May/18 00:19
Start Date: 03/May/18 00:19
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5226: 
[BEAM-3972] Translate portable batch pipelines by proto
URL: https://github.com/apache/beam/pull/5226#discussion_r185671288
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
 ##
 @@ -0,0 +1,601 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.collect.BiMap;
+import com.google.common.collect.ImmutableBiMap;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.protobuf.InvalidProtocolBufferException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import 
org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.SideInputId;
+import org.apache.beam.runners.core.construction.CoderTranslation;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.RehydratedComponents;
+import org.apache.beam.runners.core.construction.WindowingStrategyTranslation;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.core.construction.graph.PipelineNode;
+import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
+import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
+import org.apache.beam.runners.flink.translation.functions.FlinkAssignWindows;
+import 
org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageFunction;
+import 
org.apache.beam.runners.flink.translation.functions.FlinkExecutableStagePruningFunction;
+import 
org.apache.beam.runners.flink.translation.functions.FlinkPartialReduceFunction;
+import org.apache.beam.runners.flink.translation.functions.FlinkReduceFunction;
+import org.apache.beam.runners.flink.translation.types.CoderTypeInformation;
+import org.apache.beam.runners.flink.translation.types.KvKeySelector;
+import org.apache.beam.runners.flink.translation.wrappers.ImpulseInputFormat;
+import org.apache.beam.runners.fnexecution.wire.WireCoders;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderRegistry;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.ListCoder;
+import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.join.RawUnionValue;
+import org.apache.beam.sdk.transforms.join.UnionCoder;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import 

Build failed in Jenkins: beam_PerformanceTests_Python #1221

[Apache Jenkins Server]

See 


Changes:

[axelmagn] Create ArtifactSource Interface

[tgroh] Remove unused Test Utilities, Test

[axelmagn] Create RemoteBundle interface

[axelmagn] Create JobInfo for job declaration by operators.

[mairbek] Wait for input collection

[mairbek] Expose DatabaseAdminClient

[mairbek] Consistently use lower case table name and column names

[axelmagn] Create BundleFactory chain.

[mairbek] Addressed comments

--
[...truncated 4.42 KB...]
  Using cached 
https://files.pythonhosted.org/packages/a2/71/8273a7eeed0aff6a854237ab5453bc9aa67deb49df4832801c21f0ff3782/contextlib2-0.5.5-py2.py3-none-any.whl
Collecting pywinrm (from -r PerfKitBenchmarker/requirements.txt (line 25))
  Using cached 
https://files.pythonhosted.org/packages/0d/12/13a3117bbd2230043aa32dcfa2198c33269665eaa1a8fa26174ce49b338f/pywinrm-0.3.0-py2.py3-none-any.whl
Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages 
(from absl-py->-r PerfKitBenchmarker/requirements.txt (line 14)) (1.11.0)
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15)) (1.0)
Collecting colorama; extra == "windows" (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/db/c8/7dcf9dbcb22429512708fe3a547f8b6101c0d02137acbd892505aee57adf/colorama-0.3.9-py2.py3-none-any.whl
Collecting requests-ntlm>=0.3.0 (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
  Using cached 
https://files.pythonhosted.org/packages/03/4b/8b9a1afde8072c4d5710d9fa91433d504325821b038e00237dc8d6d833dc/requests_ntlm-1.1.0-py2.py3-none-any.whl
Requirement already satisfied: requests>=2.9.1 in 
/usr/local/lib/python2.7/dist-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.18.4)
Collecting xmltodict (from pywinrm->-r PerfKitBenchmarker/requirements.txt 
(line 25))
  Using cached 
https://files.pythonhosted.org/packages/42/a9/7e99652c6bc619d19d58cdd8c47560730eb5825d43a7e25db2e1d776ceb7/xmltodict-0.11.0-py2.py3-none-any.whl
Requirement already satisfied: cryptography>=1.3 in 
/usr/local/lib/python2.7/dist-packages (from requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.2.2)
Collecting ntlm-auth>=1.0.2 (from requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
  Using cached 
https://files.pythonhosted.org/packages/69/bc/230987c0dc22c763529330b2e669dbdba374d6a10c1f61232274184731be/ntlm_auth-1.1.0-py2.py3-none-any.whl
Requirement already satisfied: certifi>=2017.4.17 in 
/usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2018.4.16)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in 
/usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (3.0.4)
Requirement already satisfied: idna<2.7,>=2.5 in 
/usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.6)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in 
/usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (1.22)
Requirement already satisfied: cffi>=1.7; platform_python_implementation != 
"PyPy" in /usr/local/lib/python2.7/dist-packages (from 
cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (1.11.5)
Requirement already satisfied: enum34; python_version < "3" in 
/usr/local/lib/python2.7/dist-packages (from 
cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (1.1.6)
Requirement already satisfied: asn1crypto>=0.21.0 in 
/usr/local/lib/python2.7/dist-packages (from 
cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (0.24.0)
Requirement already satisfied: ipaddress; python_version < "3" in 
/usr/local/lib/python2.7/dist-packages (from 
cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (1.0.22)
Requirement already satisfied: pycparser in 
/usr/local/lib/python2.7/dist-packages (from cffi>=1.7; 
platform_python_implementation != 
"PyPy"->cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.18)
Installing collected packages: absl-py, colorama, colorlog, blinker, futures, 
pint, numpy, contextlib2, ntlm-auth, requests-ntlm, xmltodict, pywinrm
Successfully installed absl-py-0.2.0 blinker-1.4 colorama-0.3.9 colorlog-2.6.0 
contextlib2-0.5.5 futures-3.2.0 ntlm-auth-1.1.0 numpy-1.13.3 pint-0.8.1 
pywinrm-0.3.0 requests-ntlm-1.1.0 xmltodict-0.11.0
[beam_PerformanceTests_Python] $ 

[jira] [Work logged] (BEAM-3972) Flink runner translates batch pipelines directly by proto

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3972?focusedWorklogId=97733=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97733
 ]

ASF GitHub Bot logged work on BEAM-3972:


Author: ASF GitHub Bot
Created on: 03/May/18 00:19
Start Date: 03/May/18 00:19
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5226: 
[BEAM-3972] Translate portable batch pipelines by proto
URL: https://github.com/apache/beam/pull/5226#discussion_r185654927
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
 ##
 @@ -0,0 +1,607 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.collect.BiMap;
+import com.google.common.collect.ImmutableBiMap;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.protobuf.InvalidProtocolBufferException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import 
org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.SideInputId;
+import org.apache.beam.runners.core.construction.CoderTranslation;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.RehydratedComponents;
+import org.apache.beam.runners.core.construction.WindowingStrategyTranslation;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.core.construction.graph.PipelineNode;
+import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
+import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
+import org.apache.beam.runners.flink.translation.functions.FlinkAssignWindows;
+import 
org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageFunction;
+import 
org.apache.beam.runners.flink.translation.functions.FlinkExecutableStagePruningFunction;
+import 
org.apache.beam.runners.flink.translation.functions.FlinkPartialReduceFunction;
+import org.apache.beam.runners.flink.translation.functions.FlinkReduceFunction;
+import org.apache.beam.runners.flink.translation.types.CoderTypeInformation;
+import org.apache.beam.runners.flink.translation.types.KvKeySelector;
+import org.apache.beam.runners.flink.translation.wrappers.ImpulseInputFormat;
+import org.apache.beam.runners.fnexecution.wire.WireCoders;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderRegistry;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.ListCoder;
+import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.join.RawUnionValue;
+import org.apache.beam.sdk.transforms.join.UnionCoder;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import 

[jira] [Assigned] (BEAM-4227) Hidden annotation does not hide methods from help text

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-4227:
---

Assignee: (was: Kenneth Knowles)

> Hidden annotation does not hide methods from help text
> --
>
> Key: BEAM-4227
> URL: https://issues.apache.org/jira/browse/BEAM-4227
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.4.0
>Reporter: Paul Gerver
>Priority: Minor
>
> The javadoc for *@Hidden* mentions that it hides methods and/or interfaces, 
> but the filtering of *@Hidden* for *--help* only works for interfaces while 
> methods with the annotation are still visible.
>  
> For example, in *PipelineOptions* the optionsId getter is set with the 
> *Hidden* annotation, but when *–help* is executed, it appears in the list:
> {noformat}
> org.apache.beam.sdk.options.PipelineOptions:
>   --jobName=
> Default: JobNameFactory
> Name of the pipeline execution.It must match the regular expression
> '[a-z]([-a-z0-9]{0,38}[a-z0-9])?'.It defaults to
> ApplicationName-UserName-Date-RandomInteger
>   --optionsId=
> Default: AtomicLongFactory
>   ...
>   --userAgent=
> Default: UserAgentFactory
>   ...{noformat}
> A proper fix should be skip methods marked with the Hidden class when 
> performing print help [1]. Additionally, a test should be added to 
> *PipelineOptionsReflectorTest* [2] or factory test.
> [1] 
> [https://github.com/apache/beam/blob/19bdad654550958f5749ebfcedc3981240127b05/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java#L581]
>  [2] 
> [https://github.com/apache/beam/blob/48f9cce76cf87ffe7073c3ad9c34a9f2f6de4b76/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsReflectorTest.java#L120]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4186) Need to be able to set QuerySplitter in DatastoreIO.v1()

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4186?focusedWorklogId=97730=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97730
 ]

ASF GitHub Bot logged work on BEAM-4186:


Author: ASF GitHub Bot
Created on: 02/May/18 23:47
Start Date: 02/May/18 23:47
Worklog Time Spent: 10m 
  Work Description: fyellin commented on issue #5246: [BEAM-4186] Enable 
withQuerySplitter
URL: https://github.com/apache/beam/pull/5246#issuecomment-386155060
 
 
   To fix failures, I have added `com.google.rpc`.  This package consists of 
nothing but protocol buffers giving error codes and status. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97730)
Time Spent: 0.5h  (was: 20m)

> Need to be able to set QuerySplitter in DatastoreIO.v1()
> 
>
> Key: BEAM-4186
> URL: https://issues.apache.org/jira/browse/BEAM-4186
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-gcp
>Affects Versions: 2.4.0
>Reporter: Frank Yellin
>Assignee: Frank Yellin
>Priority: Minor
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> I want to add a method
>       withQuerySplitter(QuerySplitter querySplitter)
> to DatastoreV1.Reader.  The implementation is fairly straightforward, except 
> for enforcing the requirement that the query splitter must be Serializable 
> for this to work.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4186) Need to be able to set QuerySplitter in DatastoreIO.v1()

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4186?focusedWorklogId=97728=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97728
 ]

ASF GitHub Bot logged work on BEAM-4186:


Author: ASF GitHub Bot
Created on: 02/May/18 23:40
Start Date: 02/May/18 23:40
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #5246: [BEAM-4186] 
Enable withQuerySplitter
URL: https://github.com/apache/beam/pull/5246#issuecomment-386153856
 
 
   R: @chamikaramj 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97728)
Time Spent: 20m  (was: 10m)

> Need to be able to set QuerySplitter in DatastoreIO.v1()
> 
>
> Key: BEAM-4186
> URL: https://issues.apache.org/jira/browse/BEAM-4186
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-gcp
>Affects Versions: 2.4.0
>Reporter: Frank Yellin
>Assignee: Frank Yellin
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> I want to add a method
>       withQuerySplitter(QuerySplitter querySplitter)
> to DatastoreV1.Reader.  The implementation is fairly straightforward, except 
> for enforcing the requirement that the query splitter must be Serializable 
> for this to work.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3710) Implement portable Combiner lifting decision logic

[Daniel Oliveira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Oliveira resolved BEAM-3710.
---
   Resolution: Invalid
Fix Version/s: Not applicable

This sub-task isn't necessary, all existing decision logic of whether to lift 
or not continues to work for portable pipelines.

> Implement portable Combiner lifting decision logic
> --
>
> Key: BEAM-3710
> URL: https://issues.apache.org/jira/browse/BEAM-3710
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core, sdk-java-harness
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>  Labels: portability
> Fix For: Not applicable
>
>
> The logic for deciding if a Combiner should be lifted or not needs to be 
> implemented somewhere. If it's possible to provide all the necessary 
> information for runners to perform this check then it should be implemented 
> runner-side. Otherwise it should be implemented SDK-side.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3944) Convert beam_PerformanceTests_Python to use Gradle

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner reassigned BEAM-3944:
--

Assignee: Łukasz Gajowy

> Convert beam_PerformanceTests_Python to use Gradle
> --
>
> Key: BEAM-3944
> URL: https://issues.apache.org/jira/browse/BEAM-3944
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3944) Convert beam_PerformanceTests_Python to use Gradle

[Scott Wegner (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461718#comment-16461718
 ] 

Scott Wegner commented on BEAM-3944:


[~ŁukaszG] are you working on this?

> Convert beam_PerformanceTests_Python to use Gradle
> --
>
> Key: BEAM-3944
> URL: https://issues.apache.org/jira/browse/BEAM-3944
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Chamikara Jayalath
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4058) Removal of replaced MavenInstall jobs

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4058:
---
Issue Type: Task  (was: Sub-task)
Parent: (was: BEAM-3249)

> Removal of replaced MavenInstall jobs
> -
>
> Key: BEAM-4058
> URL: https://issues.apache.org/jira/browse/BEAM-4058
> Project: Beam
>  Issue Type: Task
>  Components: build-system
>Reporter: Pablo Estrada
>Priority: Major
>
> With migration from Gradle to Maven, existing MavenInstall jobs should be 
> removed from the jenkins configuration directory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4058) Removal of replaced MavenInstall jobs

[Scott Wegner (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461711#comment-16461711
 ] 

Scott Wegner commented on BEAM-4058:


I'm going to remove this from the release blocker list; I don't think we should 
start deleting maven stuff until we've declared we're officially dropping 
support for the maven build.

> Removal of replaced MavenInstall jobs
> -
>
> Key: BEAM-4058
> URL: https://issues.apache.org/jira/browse/BEAM-4058
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Pablo Estrada
>Priority: Major
>
> With migration from Gradle to Maven, existing MavenInstall jobs should be 
> removed from the jenkins configuration directory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3255) Update release process to use Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3255?focusedWorklogId=97714=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97714
 ]

ASF GitHub Bot logged work on BEAM-3255:


Author: ASF GitHub Bot
Created on: 02/May/18 22:59
Start Date: 02/May/18 22:59
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #424: [BEAM-3255] Updating 
release guide to use Gradle commands
URL: https://github.com/apache/beam-site/pull/424#issuecomment-386147051
 
 
   I think I agree with that. I've covered all the other TODOs that we ran into 
during our demo with Ahmet. So we can do the final review for this PR.
   
   r: @aaltay PTAL


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97714)
Time Spent: 7h 20m  (was: 7h 10m)

> Update release process to use Gradle
> 
>
> Key: BEAM-3255
> URL: https://issues.apache.org/jira/browse/BEAM-3255
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Alan Myrvold
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> This task is about configuring Gradle to generate pom's and artifacts 
> required to perform a release and update the nightly release snapshot Jenkins 
> jobs found here 
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy
> We will also require some integration tests to run against the released 
> nightly snapshot artifacts to ensure that what was built is valid.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3973) Allow to disable batch API in SpannerIO

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath updated BEAM-3973:
-
Priority: Blocker  (was: Major)

> Allow to disable batch API in SpannerIO
> ---
>
> Key: BEAM-3973
> URL: https://issues.apache.org/jira/browse/BEAM-3973
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.4.0
>Reporter: Mairbek Khadikov
>Assignee: Mairbek Khadikov
>Priority: Blocker
> Fix For: 2.5.0
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> In 2.4.0, SpannerIO#read has been migrated to use batch API. The batch API 
> provides abstractions to scale out reads from Spanner, but it requires the 
> query to be root-partitionable. The root-partitionable queries cover majority 
> of the use cases, however there are examples when running arbitrary query is 
> useful. For example, reading all the table names from the 
> information_schema.* and reading the content of those tables in the next 
> step. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3973) Allow to disable batch API in SpannerIO

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461708#comment-16461708
 ] 

Chamikara Jayalath commented on BEAM-3973:
--

I think this is a 2.5.0 blocker. Mairbek, can you confirm ?

> Allow to disable batch API in SpannerIO
> ---
>
> Key: BEAM-3973
> URL: https://issues.apache.org/jira/browse/BEAM-3973
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.4.0
>Reporter: Mairbek Khadikov
>Assignee: Mairbek Khadikov
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> In 2.4.0, SpannerIO#read has been migrated to use batch API. The batch API 
> provides abstractions to scale out reads from Spanner, but it requires the 
> query to be root-partitionable. The root-partitionable queries cover majority 
> of the use cases, however there are examples when running arbitrary query is 
> useful. For example, reading all the table names from the 
> information_schema.* and reading the content of those tables in the next 
> step. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3255) Update release process to use Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3255?focusedWorklogId=97713=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97713
 ]

ASF GitHub Bot logged work on BEAM-3255:


Author: ASF GitHub Bot
Created on: 02/May/18 22:55
Start Date: 02/May/18 22:55
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #424: [BEAM-3255] Updating 
release guide to use Gradle commands
URL: https://github.com/apache/beam-site/pull/424#issuecomment-386146224
 
 
   @pabloem Can you elaborate on what you're referring to with 
`dataflow.container_version`? I don't see this in the existing release 
documentation or your current changes.
   
   I want to get this unblocked so that we can follow this release guide for 
the 2.5.0 release.
   
   I seem to recall that we need to update `dataflow.container_version` as some 
part of the release, but I don't think the requirements would change with the 
Gradle migration. Put otherwise: if documentation about 
`dataflow.container_version` is missing, then we should track that as a bug, 
but it shouldn't hold up this doc update.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97713)
Time Spent: 7h 10m  (was: 7h)

> Update release process to use Gradle
> 
>
> Key: BEAM-3255
> URL: https://issues.apache.org/jira/browse/BEAM-3255
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Alan Myrvold
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 7h 10m
>  Remaining Estimate: 0h
>
> This task is about configuring Gradle to generate pom's and artifacts 
> required to perform a release and update the nightly release snapshot Jenkins 
> jobs found here 
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy
> We will also require some integration tests to run against the released 
> nightly snapshot artifacts to ensure that what was built is valid.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4147) Abstractions for artifact delivery via arbitrary storage backends

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461703#comment-16461703
 ] 

Eugene Kirpichov commented on BEAM-4147:


This covers an ArtifactStagingService and ArtifactRetrievalService that work by 
placing the artifacts onto a distributed filesystem, e.g. HDFS or GCS.

This is needed for pretty much all distributed runners, even Flink (since its 
DistributedCache doesn't do this yet).

> Abstractions for artifact delivery via arbitrary storage backends
> -
>
> Key: BEAM-4147
> URL: https://issues.apache.org/jira/browse/BEAM-4147
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>
> We need a way to wire in arbitrary runner artifact storage backends into the 
> job server and through to artifact staging on workers. This requires some new 
> abstractions in front of the job service.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4147) Abstractions for artifact delivery via arbitrary storage backends

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov updated BEAM-4147:
---
Summary: Abstractions for artifact delivery via arbitrary storage backends  
(was: Artifact source abstractions)

> Abstractions for artifact delivery via arbitrary storage backends
> -
>
> Key: BEAM-4147
> URL: https://issues.apache.org/jira/browse/BEAM-4147
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>
> We need a way to wire in arbitrary runner artifact storage backends into the 
> job server and through to artifact staging on workers. This requires some new 
> abstractions in front of the job service.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4229) PostCommits broken due to SpannerReadIT tests

[Daniel Oliveira (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461699#comment-16461699
 ] 

Daniel Oliveira commented on BEAM-4229:
---

Funny timing, I was literally just about to create this bug.

Are you sure it's caused by PR#4946? The earliest example of the failure I see 
is much earlier: 
[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/174/]

I believe this failure was actually caused by 
[PR#4947|https://github.com/apache/beam/pull/4947]. I tested by attempting to 
run the target the postcommit fails for before and after reverting that commit. 
Before reverting it, it would fail with the exact same error that pops up on 
Jenkins. After reverting that commit it fails because of authentication issues 
during the test, which I suppose is to be expected.

I believe the issue is due to the BigQuery dependency added in that PR, but I 
have no clue where to even begin solving dependency issues so I've gotten a bit 
stuck.

> PostCommits broken due to SpannerReadIT tests
> -
>
> Key: BEAM-4229
> URL: https://issues.apache.org/jira/browse/BEAM-4229
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Scott Wegner
>Assignee: Mairbek Khadikov
>Priority: Major
>
> It appears that [PR#4946|https://github.com/apache/beam/pull/4946] broken 
> Java post-commit tests: 
> [https://scans.gradle.com/s/ajlgofucflem2/tests/failed] 
> {quote}:beam-runners-google-cloud-dataflow-java:googleCloudPlatformIntegrationTestorg.apache.beam.sdk.io.gcp.spanner.SpannerReadIT
>  » testQuery (0.134s)
> java.lang.NoClassDefFoundError: 
> com/google/api/gax/retrying/ExceptionRetryAlgorithm
> Caused by: java.lang.ClassNotFoundException: 
> com.google.api.gax.retrying.ExceptionRetryAlgorithm{quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4170) Gradle publish -Prelease should create source .zip

[Scott Wegner (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461697#comment-16461697
 ] 

Scott Wegner commented on BEAM-4170:


We've updated release documentation with instructions to  pull source zip from 
GitHub: https://github.com/pabloem/beam-site/pull/2

With these changes, it does not need to be integrated into the gradle build.

> Gradle publish -Prelease should create source .zip
> --
>
> Key: BEAM-4170
> URL: https://issues.apache.org/jira/browse/BEAM-4170
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Affects Versions: 2.5.0
>Reporter: Alan Myrvold
>Priority: Major
> Fix For: Not applicable
>
>
> When I published locally with gradle, I didn't see the source .zip but I see 
> it in 2.4.0.
> ./gradlew publish -Prelease -PdistMgmtSnapshotsUrl=http://someserver 
> --no-parallel
> In 2.4.0 it was 
> https://repo1.maven.org/maven2/org/apache/beam/beam-parent/2.4.0/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-4170) Gradle publish -Prelease should create source .zip

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner closed BEAM-4170.
--
   Resolution: Won't Fix
Fix Version/s: Not applicable

> Gradle publish -Prelease should create source .zip
> --
>
> Key: BEAM-4170
> URL: https://issues.apache.org/jira/browse/BEAM-4170
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Affects Versions: 2.5.0
>Reporter: Alan Myrvold
>Priority: Major
> Fix For: Not applicable
>
>
> When I published locally with gradle, I didn't see the source .zip but I see 
> it in 2.4.0.
> ./gradlew publish -Prelease -PdistMgmtSnapshotsUrl=http://someserver 
> --no-parallel
> In 2.4.0 it was 
> https://repo1.maven.org/maven2/org/apache/beam/beam-parent/2.4.0/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4044) Take advantage of Calcite DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4044?focusedWorklogId=97708=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97708
 ]

ASF GitHub Bot logged work on BEAM-4044:


Author: ASF GitHub Bot
Created on: 02/May/18 22:40
Start Date: 02/May/18 22:40
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5254: 
[BEAM-4044] [SQL] Simplify TableProvider interface
URL: https://github.com/apache/beam/pull/5254#discussion_r185659851
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProvider.java
 ##
 @@ -41,7 +39,7 @@
  * LOCATION '[PROJECT_ID]:[DATASET].[TABLE]'
  * }
  */
-public class BigQueryTableProvider implements TableProvider {
+public class BigQueryTableProvider extends InMemoryTableProvider {
 
 Review comment:
   Ok, I've renamed this to `InMemoryMetaTableProvider`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97708)
Time Spent: 9h 40m  (was: 9.5h)

> Take advantage of Calcite DDL
> -
>
> Key: BEAM-4044
> URL: https://issues.apache.org/jira/browse/BEAM-4044
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 9h 40m
>  Remaining Estimate: 0h
>
> In Calcite 1.15 support for abstract DDL moved into calcite core. We should 
> take advantage of that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (BEAM-4204) Python: PortableRunner - p.run() via given JobService

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov reopened BEAM-4204:

  Assignee: Eugene Kirpichov  (was: Robert Bradshaw)

universal_local_runner does more than just submit the job to a JobService, it 
also contains an actual JobService and starts and executes via it. Repurposing 
the current Jira to instead create a PortableRunner class that simply executes 
the job via a given JobService, presumably by refactoring 
universal_local_runner.

> Python: PortableRunner - p.run() via given JobService
> -
>
> Key: BEAM-4204
> URL: https://issues.apache.org/jira/browse/BEAM-4204
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Fix For: Not applicable
>
>
> Like BEAM-4071 but for Python. Is this fully encompassed by 
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/universal_local_runner.py]
>  ? 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4229) PostCommits broken due to SpannerReadIT tests

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-4229:
--

 Summary: PostCommits broken due to SpannerReadIT tests
 Key: BEAM-4229
 URL: https://issues.apache.org/jira/browse/BEAM-4229
 Project: Beam
  Issue Type: Bug
  Components: testing
Reporter: Scott Wegner
Assignee: Mairbek Khadikov


It appears that [PR#4946|https://github.com/apache/beam/pull/4946] broken Java 
post-commit tests: [https://scans.gradle.com/s/ajlgofucflem2/tests/failed] 

{quote}:beam-runners-google-cloud-dataflow-java:googleCloudPlatformIntegrationTestorg.apache.beam.sdk.io.gcp.spanner.SpannerReadIT
 » testQuery (0.134s)
java.lang.NoClassDefFoundError: 
com/google/api/gax/retrying/ExceptionRetryAlgorithm
Caused by: java.lang.ClassNotFoundException: 
com.google.api.gax.retrying.ExceptionRetryAlgorithm{quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3972) Flink runner translates batch pipelines directly by proto

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3972?focusedWorklogId=97705=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97705
 ]

ASF GitHub Bot logged work on BEAM-3972:


Author: ASF GitHub Bot
Created on: 02/May/18 22:04
Start Date: 02/May/18 22:04
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5226: 
[BEAM-3972] Translate portable batch pipelines by proto
URL: https://github.com/apache/beam/pull/5226#discussion_r185631769
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/wire/LengthPrefixUnknownCoders.java
 ##
 @@ -14,7 +14,7 @@
  * the License.
  */
 
-package org.apache.beam.runners.fnexecution.graph;
+package org.apache.beam.runners.fnexecution.wire;
 
 Review comment:
   The worker shouldn't depend on this in the staged SDK - so this will be fine.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97705)
Time Spent: 7h 50m  (was: 7h 40m)

> Flink runner translates batch pipelines directly by proto
> -
>
> Key: BEAM-3972
> URL: https://issues.apache.org/jira/browse/BEAM-3972
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Major
>  Time Spent: 7h 50m
>  Remaining Estimate: 0h
>
> The non-portable runner uses reydrated pipelines which lack necessary 
> information. The portable Flink runner needs to translate pipelines directly 
> by proto in order to wire components into individual executable stages 
> correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3972) Flink runner translates batch pipelines directly by proto

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3972?focusedWorklogId=97706=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97706
 ]

ASF GitHub Bot logged work on BEAM-3972:


Author: ASF GitHub Bot
Created on: 02/May/18 22:04
Start Date: 02/May/18 22:04
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5226: 
[BEAM-3972] Translate portable batch pipelines by proto
URL: https://github.com/apache/beam/pull/5226#discussion_r185652571
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
 ##
 @@ -0,0 +1,601 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.collect.BiMap;
+import com.google.common.collect.ImmutableBiMap;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.protobuf.InvalidProtocolBufferException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import 
org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.SideInputId;
+import org.apache.beam.runners.core.construction.CoderTranslation;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.RehydratedComponents;
+import org.apache.beam.runners.core.construction.WindowingStrategyTranslation;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.core.construction.graph.PipelineNode;
+import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
+import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
+import org.apache.beam.runners.flink.translation.functions.FlinkAssignWindows;
+import 
org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageFunction;
+import 
org.apache.beam.runners.flink.translation.functions.FlinkExecutableStagePruningFunction;
+import 
org.apache.beam.runners.flink.translation.functions.FlinkPartialReduceFunction;
+import org.apache.beam.runners.flink.translation.functions.FlinkReduceFunction;
+import org.apache.beam.runners.flink.translation.types.CoderTypeInformation;
+import org.apache.beam.runners.flink.translation.types.KvKeySelector;
+import org.apache.beam.runners.flink.translation.wrappers.ImpulseInputFormat;
+import org.apache.beam.runners.fnexecution.wire.WireCoders;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderRegistry;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.ListCoder;
+import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.join.RawUnionValue;
+import org.apache.beam.sdk.transforms.join.UnionCoder;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import 

[jira] [Created] (BEAM-4228) The FlinkRunner shouldn't require all of the values for a key to fit in memory

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-4228:
-

 Summary: The FlinkRunner shouldn't require all of the values for a 
key to fit in memory
 Key: BEAM-4228
 URL: https://issues.apache.org/jira/browse/BEAM-4228
 Project: Beam
  Issue Type: New Feature
  Components: runner-flink
Reporter: Thomas Groh


The use of a reducer that adds all of the elements that it consumes to a list 
is the primary way in which this occurs - if instead, we produce a filtered 
iterable, or a collection of filtered iterables, we can lazily iterate over all 
of the contained elements without having to buffer all of the elements.

 

For an example of where this occurs, see {{Concatenate}} in  
{{FlinkBatchPortablePipelineTranslator}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3972) Flink runner translates batch pipelines directly by proto

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3972?focusedWorklogId=97704=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97704
 ]

ASF GitHub Bot logged work on BEAM-3972:


Author: ASF GitHub Bot
Created on: 02/May/18 22:04
Start Date: 02/May/18 22:04
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5226: 
[BEAM-3972] Translate portable batch pipelines by proto
URL: https://github.com/apache/beam/pull/5226#discussion_r185632348
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
 ##
 @@ -0,0 +1,607 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.collect.BiMap;
+import com.google.common.collect.ImmutableBiMap;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.protobuf.InvalidProtocolBufferException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import 
org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.SideInputId;
+import org.apache.beam.runners.core.construction.CoderTranslation;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.RehydratedComponents;
+import org.apache.beam.runners.core.construction.WindowingStrategyTranslation;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.core.construction.graph.PipelineNode;
+import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
+import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
+import org.apache.beam.runners.flink.translation.functions.FlinkAssignWindows;
+import 
org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageFunction;
+import 
org.apache.beam.runners.flink.translation.functions.FlinkExecutableStagePruningFunction;
+import 
org.apache.beam.runners.flink.translation.functions.FlinkPartialReduceFunction;
+import org.apache.beam.runners.flink.translation.functions.FlinkReduceFunction;
+import org.apache.beam.runners.flink.translation.types.CoderTypeInformation;
+import org.apache.beam.runners.flink.translation.types.KvKeySelector;
+import org.apache.beam.runners.flink.translation.wrappers.ImpulseInputFormat;
+import org.apache.beam.runners.fnexecution.wire.WireCoders;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderRegistry;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.ListCoder;
+import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.join.RawUnionValue;
+import org.apache.beam.sdk.transforms.join.UnionCoder;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #262

[Apache Jenkins Server]

See 


Changes:

[mairbek] Wait for input collection

[mairbek] Expose DatabaseAdminClient

[mairbek] Consistently use lower case table name and column names

[mairbek] Addressed comments

--
[...truncated 18.92 MB...]
INFO: 2018-05-02T21:57:38.514Z: Fusing consumer 
PAssert$3/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous) into 
PAssert$3/CreateActual/FilterActuals/Window.Assign
May 02, 2018 9:57:42 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T21:57:38.535Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 02, 2018 9:57:42 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T21:57:38.550Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 02, 2018 9:57:42 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T21:57:38.563Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
May 02, 2018 9:57:42 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T21:57:38.580Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
May 02, 2018 9:57:42 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T21:57:38.593Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
May 02, 2018 9:57:42 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T21:57:38.616Z: Fusing consumer 
PAssert$3/CreateActual/RewindowActuals/Window.Assign into 
PAssert$3/CreateActual/Flatten.Iterables/FlattenIterables/FlatMap
May 02, 2018 9:57:42 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T21:57:38.630Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
May 02, 2018 9:57:42 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T21:57:38.651Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
May 02, 2018 9:57:42 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T21:57:38.667Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
May 02, 2018 9:57:42 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T21:57:38.689Z: Fusing consumer 

[jira] [Work logged] (BEAM-3977) Member classes of SdkHarnessClient should have their own files.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3977?focusedWorklogId=97702=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97702
 ]

ASF GitHub Bot logged work on BEAM-3977:


Author: ASF GitHub Bot
Created on: 02/May/18 21:44
Start Date: 02/May/18 21:44
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on issue #4988: [BEAM-3977] Move out 
nested classes from SdkHarnessClient.
URL: https://github.com/apache/beam/pull/4988#issuecomment-386131130
 
 
   @axelmagn Is this superseded by https://github.com/apache/beam/pull/5152?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97702)
Time Spent: 2h  (was: 1h 50m)

> Member classes of SdkHarnessClient should have their own files.
> ---
>
> Key: BEAM-3977
> URL: https://issues.apache.org/jira/browse/BEAM-3977
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Axel Magnuson
>Assignee: Axel Magnuson
>Priority: Minor
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> SdkHarnessClient contains quite a few nested classes that could be split out. 
>  of these, BundleProcessor and ActiveBundle have grown up to be first class 
> concepts that we interact with just as much as the SdkHarnessClient.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=97699=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97699
 ]

ASF GitHub Bot logged work on BEAM-4059:


Author: ASF GitHub Bot
Created on: 02/May/18 21:26
Start Date: 02/May/18 21:26
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #5193: [BEAM-4059] Reduce 
number of ValidatesRunner tests and reorganize them for better parallelization
URL: https://github.com/apache/beam/pull/5193#issuecomment-386126683
 
 
   @kennknowles please take a look. 
   
   Still waiting on Dataflow ValidatesRunner results, but if the current state 
is a clear improvement we should merge it since this test suite is currently 
executing at around 5 hours.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97699)
Time Spent: 5h  (was: 4h 50m)

> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Pablo Estrada
>Assignee: Scott Wegner
>Priority: Major
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> See 
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1509

[Apache Jenkins Server]

See 


Changes:

[mairbek] Wait for input collection

[mairbek] Expose DatabaseAdminClient

[mairbek] Consistently use lower case table name and column names

[mairbek] Addressed comments

--
Started by GitHub push by jkff
[EnvInject] - Loading node environment variables.
Building remotely on beam23 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 6ac67f75eceb54ebe9990bc77f93c09902c28e00 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6ac67f75eceb54ebe9990bc77f93c09902c28e00
Commit message: "Merge pull request #5259: SpannerIO fixes"
 > git rev-list --no-walk 48f9cce76cf87ffe7073c3ad9c34a9f2f6de4b76 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins6040247814120990170.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user ankurgoe...@gmail.com
Not sending mail to unregistered user mair...@google.com

[beam] branch master updated (48f9cce -> 6ac67f7)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 48f9cce  Merge pull request #5152: Harness Manager Interfaces
 add c1b4f4c  Wait for input collection
 add b99c67e  Expose DatabaseAdminClient
 add d338f1d  Consistently use lower case table name and column names
 add 969582c  Addressed comments
 new 6ac67f7  Merge pull request #5259: SpannerIO fixes

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/sdk/io/gcp/spanner/MutationGroupEncoder.java  |  4 ++--
 .../apache/beam/sdk/io/gcp/spanner/SpannerAccessor.java| 10 +-
 .../org/apache/beam/sdk/io/gcp/spanner/SpannerConfig.java  | 12 +++-
 .../java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 14 +++---
 .../org/apache/beam/sdk/io/gcp/spanner/SpannerSchema.java  |  6 +++---
 .../java/org/apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java |  4 
 .../apache/beam/sdk/io/gcp/spanner/SpannerIOWriteTest.java |  6 +++---
 7 files changed, 35 insertions(+), 21 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] 01/01: Merge pull request #5259: SpannerIO fixes

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 6ac67f75eceb54ebe9990bc77f93c09902c28e00
Merge: 48f9cce 969582c
Author: Eugene Kirpichov 
AuthorDate: Wed May 2 14:07:32 2018 -0700

Merge pull request #5259: SpannerIO fixes

SpannerIO fixes

 .../beam/sdk/io/gcp/spanner/MutationGroupEncoder.java  |  4 ++--
 .../apache/beam/sdk/io/gcp/spanner/SpannerAccessor.java| 10 +-
 .../org/apache/beam/sdk/io/gcp/spanner/SpannerConfig.java  | 12 +++-
 .../java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 14 +++---
 .../org/apache/beam/sdk/io/gcp/spanner/SpannerSchema.java  |  6 +++---
 .../java/org/apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java |  4 
 .../apache/beam/sdk/io/gcp/spanner/SpannerIOWriteTest.java |  6 +++---
 7 files changed, 35 insertions(+), 21 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1508

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam23 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 48f9cce76cf87ffe7073c3ad9c34a9f2f6de4b76 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 48f9cce76cf87ffe7073c3ad9c34a9f2f6de4b76
Commit message: "Merge pull request #5152: Harness Manager Interfaces"
 > git rev-list --no-walk 48f9cce76cf87ffe7073c3ad9c34a9f2f6de4b76 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins8204356684901283737.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user ankurgoe...@gmail.com

[jira] [Commented] (BEAM-2955) Create a Cloud Bigtable HBase connector

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461613#comment-16461613
 ] 

Ismaël Mejía commented on BEAM-2955:


Hi, recent work is around implementing the new way to write IOs on Beam 
(SplittableDoFn). See BEAM-4020 and the PR (still to be finished). However the 
public API is not changing at all, it is only internals, just a different 
implementation for the expand method. So I think you can go ahead and start, 
changes are easily mergeable / rebaseable (if that world even exists).
Also take a look at the new restirction tracker ByteKeyRangeTracker this for 
sure will be useful when you guys decid eto move the Bigtable implementation to 
SDF (probably once Dataflow supports it and it will be able to do liquid 
sharding there too).

> Create a Cloud Bigtable HBase connector
> ---
>
> Key: BEAM-2955
> URL: https://issues.apache.org/jira/browse/BEAM-2955
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-gcp
>Reporter: Solomon Duskis
>Assignee: Solomon Duskis
>Priority: Major
>
> The Cloud Bigtable (CBT) team has had a Dataflow connector maintained in a 
> different repo for awhile. Recently, we did some reworking of the Cloud 
> Bigtable client that would allow it to better coexist in the Beam ecosystem, 
> and we also released a Beam connector in our repository that exposes HBase 
> idioms rather than the Protobuf idioms of BigtableIO.  More information about 
> the customer experience of the HBase connector can be found here: 
> [https://cloud.google.com/bigtable/docs/dataflow-hbase].
> The Beam repo is a much better place to house a Cloud Bigtable HBase 
> connector.  There are a couple of ways we can implement this new connector:
> # The CBT connector depends on artifacts in the io/hbase maven project.  We 
> can create a new extend HBaseIO for the purposes of CBT.  We would have to 
> add some features to HBaseIO to make that work (dynamic rebalancing, and a 
> way for HBase and CBT's size estimation models to coexist)
> # The BigtableIO connector works well, and we can add an adapter layer on top 
> of it.  I have a proof of concept of it here: 
> [https://github.com/sduskis/cloud-bigtable-client/tree/add_beam/bigtable-dataflow-parent/bigtable-hbase-beam].
> # We can build a separate CBT HBase connector.
> I'm happy to do the work.  I would appreciate some guidance and discussion 
> about the right approach.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4227) Hidden annotation does not hide methods from help text

[Scott Wegner (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461586#comment-16461586
 ] 

Scott Wegner commented on BEAM-4227:


FYI, this was discussed a bit on 
[Slack|https://the-asf.slack.com/archives/C9H0YNP3P/p1525286101000561].

> Hidden annotation does not hide methods from help text
> --
>
> Key: BEAM-4227
> URL: https://issues.apache.org/jira/browse/BEAM-4227
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.4.0
>Reporter: Paul Gerver
>Assignee: Kenneth Knowles
>Priority: Minor
>
> The javadoc for *@Hidden* mentions that it hides methods and/or interfaces, 
> but the filtering of *@Hidden* for *--help* only works for interfaces while 
> methods with the annotation are still visible.
>  
> For example, in *PipelineOptions* the optionsId getter is set with the 
> *Hidden* annotation, but when *–help* is executed, it appears in the list:
> {noformat}
> org.apache.beam.sdk.options.PipelineOptions:
>   --jobName=
> Default: JobNameFactory
> Name of the pipeline execution.It must match the regular expression
> '[a-z]([-a-z0-9]{0,38}[a-z0-9])?'.It defaults to
> ApplicationName-UserName-Date-RandomInteger
>   --optionsId=
> Default: AtomicLongFactory
>   ...
>   --userAgent=
> Default: UserAgentFactory
>   ...{noformat}
> A proper fix should be skip methods marked with the Hidden class when 
> performing print help [1]. Additionally, a test should be added to 
> *PipelineOptionsReflectorTest* [2] or factory test.
> [1] 
> [https://github.com/apache/beam/blob/19bdad654550958f5749ebfcedc3981240127b05/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java#L581]
>  [2] 
> [https://github.com/apache/beam/blob/48f9cce76cf87ffe7073c3ad9c34a9f2f6de4b76/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsReflectorTest.java#L120]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #261

[Apache Jenkins Server]

See 


Changes:

[axelmagn] Create ArtifactSource Interface

[axelmagn] Create RemoteBundle interface

[axelmagn] Create JobInfo for job declaration by operators.

[axelmagn] Create BundleFactory chain.

--
[...truncated 18.87 MB...]
INFO: Uploading 
/home/jenkins/.m2/repository/org/json/json/20160810/json-20160810.jar to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0502201001-230dbe61/output/results/staging/json-20160810-L3-Jnwdm5lAXdEpMT8FNRg.jar
May 02, 2018 8:10:02 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0502201001-230dbe61/output/results/staging/commons-logging-1.2-BAtLTY6siG9rSio70vMbAA.jar
May 02, 2018 8:10:02 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/io/netty/netty-codec-http/4.1.8.Final/netty-codec-http-4.1.8.Final.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0502201001-230dbe61/output/results/staging/netty-codec-http-4.1.8.Final-uBsfxiZeL3IeVQNS0QTHUw.jar
May 02, 2018 8:10:02 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/org/threeten/threetenbp/1.3.3/threetenbp-1.3.3.jar 
to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0502201001-230dbe61/output/results/staging/threetenbp-1.3.3-bEXFSgaAYiXSdUtR-98IjQ.jar
May 02, 2018 8:10:02 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/org/mockito/mockito-core/1.9.5/mockito-core-1.9.5.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0502201001-230dbe61/output/results/staging/mockito-core-1.9.5-b3PPBKVutgqqmWUG58EPxw.jar
May 02, 2018 8:10:03 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/org/objenesis/objenesis/2.6/objenesis-2.6.jar to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0502201001-230dbe61/output/results/staging/objenesis-2.6-X_rD9RQFypspFZcKIks-jw.jar
May 02, 2018 8:10:03 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/commons-codec/commons-codec/1.3/commons-codec-1.3.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0502201001-230dbe61/output/results/staging/commons-codec-1.3-jhScEFN0HANzalLfg5dNzA.jar
May 02, 2018 8:10:03 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/io/netty/netty-buffer/4.1.8.Final/netty-buffer-4.1.8.Final.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0502201001-230dbe61/output/results/staging/netty-buffer-4.1.8.Final-iwny8IFMu0G1ln10Yx5VrA.jar
May 02, 2018 8:10:03 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/io/netty/netty-transport/4.1.8.Final/netty-transport-4.1.8.Final.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0502201001-230dbe61/output/results/staging/netty-transport-4.1.8.Final-PXWj5ZmqlznhCnqhkcOwDA.jar
May 02, 2018 8:10:03 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/io/netty/netty-codec/4.1.8.Final/netty-codec-4.1.8.Final.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0502201001-230dbe61/output/results/staging/netty-codec-4.1.8.Final-Og5DJ4MB-fn1oSzjKqVU_w.jar
May 02, 2018 8:10:03 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.gradle/caches/modules-2/files-2.1/com.esotericsoftware.kryo/kryo/2.21/9a4e69cff8d225729656f7e97e40893b23bffef/kryo-2.21.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0502201001-230dbe61/output/results/staging/kryo-2.21-olkSUBNMYGe-WXkdlwcytQ.jar
May 02, 2018 8:10:03 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/org/xerial/snappy/snappy-java/1.1.4/snappy-java-1.1.4.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0502201001-230dbe61/output/results/staging/snappy-java-1.1.4-SFNwbMuGq13aaoKVzeS1Tw.jar
May 02, 2018 8:10:03 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/io/netty/netty-resolver/4.1.8.Final/netty-resolver-4.1.8.Final.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0502201001-230dbe61/output/results/staging/netty-resolver-4.1.8.Final-ZBELwscrxRZgRR5DKH9siA.jar
May 02, 

[jira] [Created] (BEAM-4227) Hidden annotation does not hide methods from help text

[Paul Gerver (JIRA)]

Paul Gerver created BEAM-4227:
-

 Summary: Hidden annotation does not hide methods from help text
 Key: BEAM-4227
 URL: https://issues.apache.org/jira/browse/BEAM-4227
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Affects Versions: 2.4.0
Reporter: Paul Gerver
Assignee: Kenneth Knowles


The javadoc for *@Hidden* mentions that it hides methods and/or interfaces, but 
the filtering of *@Hidden* for *--help* only works for interfaces while methods 
with the annotation are still visible.

 

For example, in *PipelineOptions* the optionsId getter is set with the *Hidden* 
annotation, but when *–help* is executed, it appears in the list:
{noformat}
org.apache.beam.sdk.options.PipelineOptions:

  --jobName=
Default: JobNameFactory
Name of the pipeline execution.It must match the regular expression
'[a-z]([-a-z0-9]{0,38}[a-z0-9])?'.It defaults to
ApplicationName-UserName-Date-RandomInteger
  --optionsId=
Default: AtomicLongFactory
  ...
  --userAgent=
Default: UserAgentFactory
  ...{noformat}
A proper fix should be skip methods marked with the Hidden class when 
performing print help [1]. Additionally, a test should be added to 
*PipelineOptionsReflectorTest* [2] or factory test.

[1] 
[https://github.com/apache/beam/blob/19bdad654550958f5749ebfcedc3981240127b05/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java#L581]
 [2] 
[https://github.com/apache/beam/blob/48f9cce76cf87ffe7073c3ad9c34a9f2f6de4b76/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsReflectorTest.java#L120]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3972) Flink runner translates batch pipelines directly by proto

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3972?focusedWorklogId=97679=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97679
 ]

ASF GitHub Bot logged work on BEAM-3972:


Author: ASF GitHub Bot
Created on: 02/May/18 19:55
Start Date: 02/May/18 19:55
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on issue #5226: [BEAM-3972] Translate 
portable batch pipelines by proto
URL: https://github.com/apache/beam/pull/5226#issuecomment-386100660
 
 
   Run Dataflow ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97679)
Time Spent: 7h 40m  (was: 7.5h)

> Flink runner translates batch pipelines directly by proto
> -
>
> Key: BEAM-3972
> URL: https://issues.apache.org/jira/browse/BEAM-3972
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Major
>  Time Spent: 7h 40m
>  Remaining Estimate: 0h
>
> The non-portable runner uses reydrated pipelines which lack necessary 
> information. The portable Flink runner needs to translate pipelines directly 
> by proto in order to wire components into individual executable stages 
> correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #183

[Apache Jenkins Server]

See 


Changes:

[github] Count compressed records with a long to avoid overflow

[alan] [BEAM-4218] Fix failing javadoc build

--
[...truncated 23.62 MB...]
WARNING: Dataflow job 2018-05-02_12_40_42-11557599337390222415 did not 
output a success or failure metric.

org.apache.beam.sdk.transforms.ParDoTest > testParDoWithSideInputsIsCumulative 
FAILED
java.lang.RuntimeException: java.lang.AbstractMethodError: 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.element(Lorg/apache/beam/sdk/transforms/DoFn;)Ljava/lang/Object;
at 
org.apache.beam.sdk.transforms.ParDoTest$TestDoFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:138)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:200)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:383)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:355)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:286)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.lang.AbstractMethodError: 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.element(Lorg/apache/beam/sdk/transforms/DoFn;)Ljava/lang/Object;
at 
org.apache.beam.sdk.transforms.ParDoTest$TestDoFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:138)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:200)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:383)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:355)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:286)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.lang.AbstractMethodError: 

[jira] [Work logged] (BEAM-3327) Add abstractions to manage Environment Instance lifecycles.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3327?focusedWorklogId=97661=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97661
 ]

ASF GitHub Bot logged work on BEAM-3327:


Author: ASF GitHub Bot
Created on: 02/May/18 19:12
Start Date: 02/May/18 19:12
Worklog Time Spent: 10m 
  Work Description: tgroh closed pull request #5152: [BEAM-3327] Harness 
Manager Interfaces
URL: https://github.com/apache/beam/pull/5152
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ArtifactSource.java
 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ArtifactSource.java
new file mode 100644
index 000..a8b582640b5
--- /dev/null
+++ 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ArtifactSource.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.fnexecution.artifact;
+
+import io.grpc.stub.StreamObserver;
+import java.io.IOException;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactChunk;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.Manifest;
+
+/**
+ * Makes artifacts available to an ArtifactRetrievalService by
+ * encapsulating runner-specific resources.
+ */
+public interface ArtifactSource {
+
+  /**
+   * Get the artifact manifest available from this source.
+   */
+  Manifest getManifest() throws IOException;
+
+  /**
+   * Get an artifact by its name.
+   */
+  void getArtifact(String name, StreamObserver 
responseObserver);
+}
diff --git 
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/JobBundleFactory.java
 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/JobBundleFactory.java
new file mode 100644
index 000..96286fa0eef
--- /dev/null
+++ 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/JobBundleFactory.java
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.fnexecution.control;
+
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+
+/**
+ * A factory that has all job-scoped information, and can be combined with 
stage-scoped information
+ * to create a {@link StageBundleFactory}.
+ *
+ * Releases all job-scoped resources when closed.
+ */
+public interface JobBundleFactory extends AutoCloseable {
+  StageBundleFactory forStage(
+  ExecutableStage executableStage, StateRequestHandler 
stateRequestHandler);
+}
diff --git 
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/OutputReceiverFactory.java
 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/OutputReceiverFactory.java
new file mode 100644
index 000..ae818ca0726
--- /dev/null
+++ 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/OutputReceiverFactory.java
@@ 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1507

[Apache Jenkins Server]

See 


Changes:

[axelmagn] Create ArtifactSource Interface

[axelmagn] Create RemoteBundle interface

[axelmagn] Create JobInfo for job declaration by operators.

[axelmagn] Create BundleFactory chain.

--
Started by GitHub push by tgroh
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 48f9cce76cf87ffe7073c3ad9c34a9f2f6de4b76 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 48f9cce76cf87ffe7073c3ad9c34a9f2f6de4b76
Commit message: "Merge pull request #5152: Harness Manager Interfaces"
 > git rev-list --no-walk d19c3665a41978b6bdf3330dacc4ff4e8404fd79 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins6324569358630667435.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user ankurgoe...@gmail.com

[beam] 01/01: Merge pull request #5152: Harness Manager Interfaces

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 48f9cce76cf87ffe7073c3ad9c34a9f2f6de4b76
Merge: d19c366 c11c837
Author: Thomas Groh 
AuthorDate: Wed May 2 12:12:08 2018 -0700

Merge pull request #5152: Harness Manager Interfaces

[BEAM-3327]

 .../fnexecution/artifact/ArtifactSource.java   | 41 +
 .../fnexecution/control/JobBundleFactory.java  | 33 +
 .../fnexecution/control/OutputReceiverFactory.java | 31 
 .../runners/fnexecution/control/RemoteBundle.java  | 43 ++
 .../fnexecution/control/RemoteOutputReceiver.java  | 38 +++
 .../fnexecution/control/SdkHarnessClient.java  | 19 +-
 .../fnexecution/control/StageBundleFactory.java| 38 +++
 .../runners/fnexecution/provisioning/JobInfo.java  | 37 +++
 .../fnexecution/control/RemoteExecutionTest.java   |  1 -
 .../fnexecution/control/SdkHarnessClientTest.java  |  1 -
 10 files changed, 263 insertions(+), 19 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] branch master updated (d19c366 -> 48f9cce)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from d19c366  Merge pull request #5236
 add 74d6867  Create ArtifactSource Interface
 add 880d50e  Create RemoteBundle interface
 add 353be65  Create JobInfo for job declaration by operators.
 add c11c837  Create BundleFactory chain.
 new 48f9cce  Merge pull request #5152: Harness Manager Interfaces

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../{package-info.java => ArtifactSource.java} | 24 +++---
 .../{package-info.java => JobBundleFactory.java}   | 16 +---
 ...ackage-info.java => OutputReceiverFactory.java} | 14 ---
 .../runners/fnexecution/control/RemoteBundle.java} | 29 +++---
 .../RemoteOutputReceiver.java} | 17 +++--
 .../fnexecution/control/SdkHarnessClient.java  | 19 ++
 .../{package-info.java => StageBundleFactory.java} | 21 +---
 .../runners/fnexecution/provisioning/JobInfo.java  | 22 
 .../fnexecution/control/RemoteExecutionTest.java   |  1 -
 .../fnexecution/control/SdkHarnessClientTest.java  |  1 -
 10 files changed, 99 insertions(+), 65 deletions(-)
 copy 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/{package-info.java
 => ArtifactSource.java} (59%)
 copy 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/{package-info.java
 => JobBundleFactory.java} (62%)
 copy 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/{package-info.java
 => OutputReceiverFactory.java} (71%)
 copy 
runners/{core-java/src/main/java/org/apache/beam/runners/core/KeyedWorkItem.java
 => 
java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/RemoteBundle.java}
 (52%)
 copy 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/{data/RemoteInputDestination.java
 => control/RemoteOutputReceiver.java} (63%)
 copy 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/{package-info.java
 => StageBundleFactory.java} (53%)
 copy 
sdks/java/core/src/main/java/org/apache/beam/sdk/util/DoFnAndMainOutput.java => 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/provisioning/JobInfo.java
 (60%)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #260

[Apache Jenkins Server]

See 


Changes:

[tgroh] Remove unused Test Utilities, Test

--
[...truncated 18.83 MB...]
May 02, 2018 7:05:56 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T19:05:52.781Z: Fusing consumer 
PAssert$3/CreateActual/ParDo(Anonymous) into 
PAssert$3/CreateActual/RewindowActuals/Window.Assign
May 02, 2018 7:05:56 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T19:05:52.812Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into PAssert$3/CreateActual/ParDo(Anonymous)
May 02, 2018 7:05:56 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T19:05:52.847Z: Fusing consumer 
Combine.globally(Count)/ProduceDefault into 
Combine.globally(Count)/CreateVoid/Read(CreateSource)
May 02, 2018 7:05:56 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T19:05:52.880Z: Fusing consumer 
Combine.globally(Count)/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow) into 
Combine.globally(Count)/Values/Values/Map
May 02, 2018 7:05:56 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T19:05:52.914Z: Fusing consumer 
Combine.globally(Count)/Combine.perKey(Count)/Combine.GroupedValues into 
Combine.globally(Count)/Combine.perKey(Count)/GroupByKey/Read
May 02, 2018 7:05:56 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T19:05:52.949Z: Fusing consumer 
Combine.globally(Count)/Combine.perKey(Count)/GroupByKey+Combine.globally(Count)/Combine.perKey(Count)/Combine.GroupedValues/Partial
 into Combine.globally(Count)/WithKeys/AddKeys/Map
May 02, 2018 7:05:56 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T19:05:52.971Z: Fusing consumer 
DatastoreV1.Read/Reshuffle/Reshuffle/GroupByKey/GroupByWindow into 
DatastoreV1.Read/Reshuffle/Reshuffle/GroupByKey/Read
May 02, 2018 7:05:56 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T19:05:52.991Z: Fusing consumer 
Combine.globally(Count)/Combine.perKey(Count)/GroupByKey/Reify into 
Combine.globally(Count)/Combine.perKey(Count)/GroupByKey+Combine.globally(Count)/Combine.perKey(Count)/Combine.GroupedValues/Partial
May 02, 2018 7:05:56 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T19:05:53.008Z: Fusing consumer 
Combine.globally(Count)/WithKeys/AddKeys/Map into DatastoreV1.Read/Read
May 02, 2018 7:05:56 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T19:05:53.035Z: Fusing consumer 
DatastoreV1.Read/Reshuffle/Pair with random key into DatastoreV1.Read/Split
May 02, 2018 7:05:56 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T19:05:53.059Z: Fusing consumer 
DatastoreV1.Read/Reshuffle/Reshuffle/GroupByKey/Write into 
DatastoreV1.Read/Reshuffle/Reshuffle/GroupByKey/Reify
May 02, 2018 7:05:56 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T19:05:53.084Z: Fusing consumer 
DatastoreV1.Read/Reshuffle/Reshuffle/GroupByKey/Reify into 
DatastoreV1.Read/Reshuffle/Reshuffle/Window.Into()/Window.Assign
May 02, 2018 7:05:56 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T19:05:53.116Z: Fusing consumer 
Combine.globally(Count)/Values/Values/Map into 
Combine.globally(Count)/Combine.perKey(Count)/Combine.GroupedValues/Extract
May 02, 2018 7:05:56 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T19:05:53.149Z: Fusing consumer DatastoreV1.Read/Split into 
DatastoreV1.Read/ParDo(GqlQueryTranslate)
May 02, 2018 7:05:56 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T19:05:53.180Z: Fusing consumer 
DatastoreV1.Read/Reshuffle/Reshuffle/Window.Into()/Window.Assign into 
DatastoreV1.Read/Reshuffle/Pair with random key
May 02, 2018 7:05:56 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T19:05:53.209Z: Fusing consumer 
Combine.globally(Count)/Combine.perKey(Count)/GroupByKey/Write into 
Combine.globally(Count)/Combine.perKey(Count)/GroupByKey/Reify
May 02, 2018 7:05:56 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T19:05:53.234Z: Fusing consumer 
DatastoreV1.Read/ParDo(GqlQueryTranslate) into 
DatastoreV1.Read/Create.OfValueProvider/Create.Values/Read(CreateSource)
 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #259

[Apache Jenkins Server]

See 


--
[...truncated 19.59 MB...]
May 02, 2018 6:43:55 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.TriggerExampleTest > testExtractTotalFlow 
STANDARD_ERROR
May 02, 2018 6:43:57 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.FilterExamplesTest > 
testFilterSingleMonthDataFn STANDARD_ERROR
May 02, 2018 6:43:57 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.FilterExamplesTest > testProjectionFn 
STANDARD_ERROR
May 02, 2018 6:43:57 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.JoinExamplesTest > testExtractCountryInfoFn 
STANDARD_ERROR
May 02, 2018 6:43:57 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.JoinExamplesTest > testExtractEventDataFn 
STANDARD_ERROR
May 02, 2018 6:43:57 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.DebuggingWordCountTest > testDebuggingWordCount 
STANDARD_ERROR
May 02, 2018 6:43:57 PM org.apache.beam.sdk.io.FileBasedSource 
getEstimatedSizeBytes
INFO: Filepattern 
/tmp/junit7688379987344560127/junit4509727555484717318.tmp matched 1 files with 
total size 54
May 02, 2018 6:43:57 PM org.apache.beam.sdk.io.FileBasedSource split
INFO: Splitting filepattern 
/tmp/junit7688379987344560127/junit4509727555484717318.tmp into bundles of size 
3 took 0 ms and produced 1 files and 18 bundles

org.apache.beam.examples.WordCountTest > testExtractWordsFn STANDARD_ERROR
May 02, 2018 6:43:58 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.subprocess.ExampleEchoPipelineTest > 
testExampleEchoPipeline STANDARD_ERROR
May 02, 2018 6:43:59 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-Echo218432434358590604.sh 
May 02, 2018 6:43:59 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
May 02, 2018 6:43:59 PM org.apache.beam.examples.subprocess.utils.FileUtils 
copyFileFromGCSToWorker
INFO: Moving File /tmp/test-Echo218432434358590604.sh to 
/tmp/test-Echoo6570224116478865471/test-Echo218432434358590604.sh 
May 02, 2018 6:43:59 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-EchoAgain3779072099562518527.sh 
May 02, 2018 6:43:59 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
May 02, 2018 6:43:59 PM org.apache.beam.examples.subprocess.utils.FileUtils 
copyFileFromGCSToWorker
INFO: Moving File /tmp/test-EchoAgain3779072099562518527.sh to 
/tmp/test-Echoo6570224116478865471/test-EchoAgain3779072099562518527.sh 

org.apache.beam.examples.complete.game.HourlyTeamScoreTest > 
testUserScoresFilter STANDARD_OUT
GOT user18_BananaEmu,BananaEmu,1,144796569,2015-11-19 12:41:31.053
GOT user2_AmberCockatoo,AmberCockatoo,13,144796569,2015-11-19 
12:41:31.053
GOT user3_BananaEmu,BananaEmu,17,144796569,2015-11-19 12:41:31.053
GOT user18_BananaEmu,BananaEmu,7,144796569,2015-11-19 12:41:31.053
GOT user0_MagentaKangaroo,MagentaKangaroo,3,144795563,2015-11-19 
09:53:53.444
GOT user0_MagentaKangaroo,MagentaKangaroo,4,144796569,2015-11-19 
12:41:31.053
GOT user18_ApricotCaneToad,ApricotCaneToad,14,144796569,2015-11-19 
12:41:31.053
GOT 
user0_AndroidGreenEchidna,AndroidGreenEchidna,0,144796569,2015-11-19 
12:41:31.053
GOT user6_AmberNumbat,AmberNumbat,11,144795563,2015-11-19 09:53:53.444
GOT user19_BisqueBilby,BisqueBilby,6,144795563,2015-11-19 09:53:53.444
GOT user13_ApricotQuokka,ApricotQuokka,15,144795563,2015-11-19 
09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,11,144795563,2015-11-19 
09:53:53.444
GOT 

[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=97626=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97626
 ]

ASF GitHub Bot logged work on BEAM-4059:


Author: ASF GitHub Bot
Created on: 02/May/18 18:22
Start Date: 02/May/18 18:22
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #5193: [BEAM-4059] Reduce 
number of ValidatesRunner tests and reorganize them for better parallelization
URL: https://github.com/apache/beam/pull/5193#issuecomment-386073860
 
 
   Java precommit now [fails on only a single test 
case](https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/4838/), 
which is unrelated to this change.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97626)
Time Spent: 4h 40m  (was: 4.5h)

> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Pablo Estrada
>Assignee: Scott Wegner
>Priority: Major
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> See 
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=97628=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97628
 ]

ASF GitHub Bot logged work on BEAM-4059:


Author: ASF GitHub Bot
Created on: 02/May/18 18:23
Start Date: 02/May/18 18:23
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #5193: [BEAM-4059] Reduce 
number of ValidatesRunner tests and reorganize them for better parallelization
URL: https://github.com/apache/beam/pull/5193#issuecomment-386073895
 
 
   Run Dataflow ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97628)
Time Spent: 4h 50m  (was: 4h 40m)

> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Pablo Estrada
>Assignee: Scott Wegner
>Priority: Major
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> See 
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3327) Add abstractions to manage Environment Instance lifecycles.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3327?focusedWorklogId=97625=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97625
 ]

ASF GitHub Bot logged work on BEAM-3327:


Author: ASF GitHub Bot
Created on: 02/May/18 18:22
Start Date: 02/May/18 18:22
Worklog Time Spent: 10m 
  Work Description: axelmagn closed pull request #5152: [BEAM-3327] Harness 
Manager Interfaces
URL: https://github.com/apache/beam/pull/5152
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ArtifactSource.java
 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ArtifactSource.java
new file mode 100644
index 000..a8b582640b5
--- /dev/null
+++ 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ArtifactSource.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.fnexecution.artifact;
+
+import io.grpc.stub.StreamObserver;
+import java.io.IOException;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactChunk;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.Manifest;
+
+/**
+ * Makes artifacts available to an ArtifactRetrievalService by
+ * encapsulating runner-specific resources.
+ */
+public interface ArtifactSource {
+
+  /**
+   * Get the artifact manifest available from this source.
+   */
+  Manifest getManifest() throws IOException;
+
+  /**
+   * Get an artifact by its name.
+   */
+  void getArtifact(String name, StreamObserver 
responseObserver);
+}
diff --git 
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/JobBundleFactory.java
 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/JobBundleFactory.java
new file mode 100644
index 000..96286fa0eef
--- /dev/null
+++ 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/JobBundleFactory.java
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.fnexecution.control;
+
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+
+/**
+ * A factory that has all job-scoped information, and can be combined with 
stage-scoped information
+ * to create a {@link StageBundleFactory}.
+ *
+ * Releases all job-scoped resources when closed.
+ */
+public interface JobBundleFactory extends AutoCloseable {
+  StageBundleFactory forStage(
+  ExecutableStage executableStage, StateRequestHandler 
stateRequestHandler);
+}
diff --git 
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/OutputReceiverFactory.java
 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/OutputReceiverFactory.java
new file mode 100644
index 000..ae818ca0726
--- /dev/null
+++ 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/OutputReceiverFactory.java
@@ 

[jira] [Work logged] (BEAM-3327) Add abstractions to manage Environment Instance lifecycles.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3327?focusedWorklogId=97624=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97624
 ]

ASF GitHub Bot logged work on BEAM-3327:


Author: ASF GitHub Bot
Created on: 02/May/18 18:22
Start Date: 02/May/18 18:22
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on issue #5152: [BEAM-3327] Harness 
Manager Interfaces
URL: https://github.com/apache/beam/pull/5152#issuecomment-386073836
 
 
   Oops.  changed it but didn't push it.  should be fixed now.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97624)
Time Spent: 28h 40m  (was: 28.5h)

> Add abstractions to manage Environment Instance lifecycles.
> ---
>
> Key: BEAM-3327
> URL: https://issues.apache.org/jira/browse/BEAM-3327
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Axel Magnuson
>Priority: Major
>  Labels: portability
>  Time Spent: 28h 40m
>  Remaining Estimate: 0h
>
> This permits remote stage execution for arbitrary environments



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3327) Add abstractions to manage Environment Instance lifecycles.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3327?focusedWorklogId=97627=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97627
 ]

ASF GitHub Bot logged work on BEAM-3327:


Author: ASF GitHub Bot
Created on: 02/May/18 18:22
Start Date: 02/May/18 18:22
Worklog Time Spent: 10m 
  Work Description: axelmagn opened a new pull request #5152: [BEAM-3327] 
Harness Manager Interfaces
URL: https://github.com/apache/beam/pull/5152
 
 
   These are some interfaces that will be used on the worker to manage the 
lifetimes of remote environments and the related RPC services.  The key 
addition is of `SdkHarnessManager`, which is responsible for managing these 
resources and can provide a `RemoteEnvironment` to runner operators.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [x] Write a pull request description that is detailed enough to 
understand:
  - [x] What the pull request does
  - [x] Why it does it
  - [x] How it does it
  - [x] Why this approach
- [x] Each commit in the pull request should have a meaningful subject line 
and body.
- [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97627)
Time Spent: 29h  (was: 28h 50m)

> Add abstractions to manage Environment Instance lifecycles.
> ---
>
> Key: BEAM-3327
> URL: https://issues.apache.org/jira/browse/BEAM-3327
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Axel Magnuson
>Priority: Major
>  Labels: portability
>  Time Spent: 29h
>  Remaining Estimate: 0h
>
> This permits remote stage execution for arbitrary environments



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (971f1f1 -> d19c366)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 971f1f1  Fix DynamicDestinations documentation
 add 1d32930  Remove unused Test Utilities, Test
 new d19c366  Merge pull request #5236

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/runners/direct/StructuralKeyTest.java | 90 --
 .../runners/direct/portable/StructuralKeyTest.java | 89 -
 2 files changed, 179 deletions(-)
 delete mode 100644 
runners/direct-java/src/test/java/org/apache/beam/runners/direct/StructuralKeyTest.java
 delete mode 100644 
runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/StructuralKeyTest.java

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] 01/01: Merge pull request #5236

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d19c3665a41978b6bdf3330dacc4ff4e8404fd79
Merge: 971f1f1 1d32930
Author: Thomas Groh 
AuthorDate: Wed May 2 11:14:06 2018 -0700

Merge pull request #5236

Remove unused Test

 .../beam/runners/direct/StructuralKeyTest.java | 90 --
 .../runners/direct/portable/StructuralKeyTest.java | 89 -
 2 files changed, 179 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1506

[Apache Jenkins Server]

See 


Changes:

[tgroh] Remove unused Test Utilities, Test

--
Started by GitHub push by tgroh
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d19c3665a41978b6bdf3330dacc4ff4e8404fd79 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d19c3665a41978b6bdf3330dacc4ff4e8404fd79
Commit message: "Merge pull request #5236"
 > git rev-list --no-walk 971f1f11d09af8545b17eb23da3d3fd4a80055a2 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins5454284561726038269.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user ankurgoe...@gmail.com

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT_HDFS #121

[Apache Jenkins Server]

See 


Changes:

[github] Count compressed records with a long to avoid overflow

[alan] [BEAM-4218] Fix failing javadoc build

[github] Fix DynamicDestinations documentation

--
[...truncated 1005.85 KB...]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.net.ConnectException: Call From 
textioit0writethenreadall-05021104-754l-harness-93kh.c.apache-beam-testing.internal/10.128.15.194
 to 12.184.224.35.bc.googleusercontent.com:9000 failed on connection exception: 
java.net.ConnectException: Connection refused; For more details see:  
http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:923)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at 
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
at org.apache.hadoop.ipc.Client.call(Client.java:1451)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT_HDFS #127

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT #447

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3972) Flink runner translates batch pipelines directly by proto

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3972?focusedWorklogId=97620=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97620
 ]

ASF GitHub Bot logged work on BEAM-3972:


Author: ASF GitHub Bot
Created on: 02/May/18 18:07
Start Date: 02/May/18 18:07
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on issue #5226: [BEAM-3972] Translate 
portable batch pipelines by proto
URL: https://github.com/apache/beam/pull/5226#issuecomment-386069062
 
 
   Run Dataflow ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97620)
Time Spent: 7.5h  (was: 7h 20m)

> Flink runner translates batch pipelines directly by proto
> -
>
> Key: BEAM-3972
> URL: https://issues.apache.org/jira/browse/BEAM-3972
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Major
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>
> The non-portable runner uses reydrated pipelines which lack necessary 
> information. The portable Flink runner needs to translate pipelines directly 
> by proto in order to wire components into individual executable stages 
> correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_JDBC #524

[Apache Jenkins Server]

See 


Changes:

[github] Count compressed records with a long to avoid overflow

[alan] [BEAM-4218] Fix failing javadoc build

[github] Fix DynamicDestinations documentation

--
[...truncated 92.32 KB...]
:beam-sdks-java-io-google-cloud-platform:compileTestJava (Thread[Task worker 
for ':' Thread 11,5,main]) completed. Took 0.146 secs.
:beam-runners-google-cloud-dataflow-java:classes (Thread[Task worker for ':' 
Thread 8,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:classes UP-TO-DATE
Skipping task ':beam-runners-google-cloud-dataflow-java:classes' as it has no 
actions.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 14,5,main]) started.
:beam-runners-google-cloud-dataflow-java:classes (Thread[Task worker for ':' 
Thread 8,5,main]) completed. Took 0.0 secs.

> Task :beam-sdks-java-io-google-cloud-platform:testClasses UP-TO-DATE
Skipping task ':beam-sdks-java-io-google-cloud-platform:testClasses' as it has 
no actions.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 14,5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for ':' 
Thread 8,5,main]) started.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 14,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar' is 
04a3085b6bea6a879dc57d9ac5acee96
Caching disabled for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-sdks-java-io-google-cloud-platform:shadowTestJar' as it is 
up-to-date.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 14,5,main]) completed. Took 0.046 secs.

> Task :beam-runners-google-cloud-dataflow-java:shadowJar UP-TO-DATE
Build cache key for task ':beam-runners-google-cloud-dataflow-java:shadowJar' 
is ee2cdf0adb41f022aa18579c1dd94132
Caching disabled for task ':beam-runners-google-cloud-dataflow-java:shadowJar': 
Caching has not been enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for ':' 
Thread 8,5,main]) completed. Took 0.118 secs.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 8,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:compileTestJava' is 
637b686cd35928bb6a91d562264c47dd
Skipping task ':beam-runners-google-cloud-dataflow-java:compileTestJava' as it 
is up-to-date.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 8,5,main]) completed. Took 0.073 secs.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 8,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:testClasses UP-TO-DATE
Skipping task ':beam-runners-google-cloud-dataflow-java:testClasses' as it has 
no actions.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 8,5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 8,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar' is 
f750490a8eb31879a3ad219f9056a199
Caching disabled for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowTestJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 8,5,main]) completed. Took 0.045 secs.
:beam-sdks-java-io-jdbc:compileTestJava (Thread[Task worker for ':' Thread 
8,5,main]) started.

> Task :beam-sdks-java-io-jdbc:compileTestJava UP-TO-DATE
Build cache key for task ':beam-sdks-java-io-jdbc:compileTestJava' is 
c449a6f450f06d2c5bc21e45bbd3
Skipping task ':beam-sdks-java-io-jdbc:compileTestJava' as it is up-to-date.
:beam-sdks-java-io-jdbc:compileTestJava (Thread[Task worker for ':' Thread 
8,5,main]) completed. Took 0.056 secs.
:beam-sdks-java-io-jdbc:testClasses (Thread[Task worker for ':' Thread 
8,5,main]) started.

> Task :beam-sdks-java-io-jdbc:testClasses UP-TO-DATE
Skipping task ':beam-sdks-java-io-jdbc:testClasses' as it has no actions.
:beam-sdks-java-io-jdbc:testClasses (Thread[Task worker for ':' Thread 
8,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-io-jdbc:integrationTest (Thread[Task worker for ':' Thread 

[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=97619=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97619
 ]

ASF GitHub Bot logged work on BEAM-3981:


Author: ASF GitHub Bot
Created on: 02/May/18 18:06
Start Date: 02/May/18 18:06
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #5053: [BEAM-3981] 
Futurize coders subpackage
URL: https://github.com/apache/beam/pull/5053#issuecomment-386068489
 
 
   Based on the information I currently have, my best guess is that regression 
is caused by changes in Cython's interpretation of Py3 implementations of 
range() and iteritems() in coder_impl.py. We can see a difference in code 
generated by Cython here:
   
   
https://docs.google.com/document/d/1S-oeqJGiMHt_L3iCgr9dYfQdR0_ukQE25mcvK-BqudU/edit#heading=h.xkt18qn9ofn1
   
   I am seeing 7% regression in the median on the benchmarks I've been running, 
and I was able to see the regression go away with these changes reverted. 
   
   We can also try to benchmark these methods in isolation to confirm and rule 
out the variance of test setup and runner componets. I might not have time to 
do it this week.
   
   @robertwb , do you  think anything can be done to improve Cython translation 
here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97619)
Time Spent: 15h 40m  (was: 15.5h)

> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 15h 40m
>  Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix 
> python 2 compatibility. This prepares the subpackage for python 3 support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=97618=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97618
 ]

ASF GitHub Bot logged work on BEAM-3981:


Author: ASF GitHub Bot
Created on: 02/May/18 18:05
Start Date: 02/May/18 18:05
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #5053: [BEAM-3981] 
Futurize coders subpackage
URL: https://github.com/apache/beam/pull/5053#issuecomment-386068489
 
 
   Based on the information I currently have, my best guess is that regression 
is caused by changes in Cython's interpretation of Py3 implementations of 
range() and iteritems() in coder_impl.py. We can see a difference in code 
generated by Cython here:
   
   
https://docs.google.com/document/d/1S-oeqJGiMHt_L3iCgr9dYfQdR0_ukQE25mcvK-BqudU/edit#heading=h.xkt18qn9ofn1
   
   I am seeing 7% regression in the median on the benchmarks I've been running, 
and I was able to see the regression go away with these changes reverted. 
   
   We can also try to benchmark these methods in isolation to confirm and rule 
out the variance of test setup. I might not have time to do it this week.
   
   @robertwb , do you  think anything can be done to improve Cython translation 
here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97618)
Time Spent: 15.5h  (was: 15h 20m)

> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 15.5h
>  Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix 
> python 2 compatibility. This prepares the subpackage for python 3 support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Python #1220

[Apache Jenkins Server]

See 


Changes:

[github] Count compressed records with a long to avoid overflow

[alan] [BEAM-4218] Fix failing javadoc build

[github] Fix DynamicDestinations documentation

--
[...truncated 4.42 KB...]
  Using cached 
https://files.pythonhosted.org/packages/a2/71/8273a7eeed0aff6a854237ab5453bc9aa67deb49df4832801c21f0ff3782/contextlib2-0.5.5-py2.py3-none-any.whl
Collecting pywinrm (from -r PerfKitBenchmarker/requirements.txt (line 25))
  Using cached 
https://files.pythonhosted.org/packages/0d/12/13a3117bbd2230043aa32dcfa2198c33269665eaa1a8fa26174ce49b338f/pywinrm-0.3.0-py2.py3-none-any.whl
Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages 
(from absl-py->-r PerfKitBenchmarker/requirements.txt (line 14)) (1.11.0)
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15)) (1.0)
Collecting colorama; extra == "windows" (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/db/c8/7dcf9dbcb22429512708fe3a547f8b6101c0d02137acbd892505aee57adf/colorama-0.3.9-py2.py3-none-any.whl
Collecting requests-ntlm>=0.3.0 (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
  Using cached 
https://files.pythonhosted.org/packages/03/4b/8b9a1afde8072c4d5710d9fa91433d504325821b038e00237dc8d6d833dc/requests_ntlm-1.1.0-py2.py3-none-any.whl
Requirement already satisfied: requests>=2.9.1 in 
/usr/local/lib/python2.7/dist-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.18.4)
Collecting xmltodict (from pywinrm->-r PerfKitBenchmarker/requirements.txt 
(line 25))
  Using cached 
https://files.pythonhosted.org/packages/42/a9/7e99652c6bc619d19d58cdd8c47560730eb5825d43a7e25db2e1d776ceb7/xmltodict-0.11.0-py2.py3-none-any.whl
Requirement already satisfied: cryptography>=1.3 in 
/usr/local/lib/python2.7/dist-packages (from requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.2.2)
Collecting ntlm-auth>=1.0.2 (from requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
  Using cached 
https://files.pythonhosted.org/packages/69/bc/230987c0dc22c763529330b2e669dbdba374d6a10c1f61232274184731be/ntlm_auth-1.1.0-py2.py3-none-any.whl
Requirement already satisfied: certifi>=2017.4.17 in 
/usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2018.4.16)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in 
/usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (3.0.4)
Requirement already satisfied: idna<2.7,>=2.5 in 
/usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.6)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in 
/usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (1.22)
Requirement already satisfied: cffi>=1.7; platform_python_implementation != 
"PyPy" in /usr/local/lib/python2.7/dist-packages (from 
cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (1.11.5)
Requirement already satisfied: enum34; python_version < "3" in 
/usr/local/lib/python2.7/dist-packages (from 
cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (1.1.6)
Requirement already satisfied: asn1crypto>=0.21.0 in 
/usr/local/lib/python2.7/dist-packages (from 
cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (0.24.0)
Requirement already satisfied: ipaddress; python_version < "3" in 
/usr/local/lib/python2.7/dist-packages (from 
cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (1.0.22)
Requirement already satisfied: pycparser in 
/usr/local/lib/python2.7/dist-packages (from cffi>=1.7; 
platform_python_implementation != 
"PyPy"->cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.18)
Installing collected packages: absl-py, colorama, colorlog, blinker, futures, 
pint, numpy, contextlib2, ntlm-auth, requests-ntlm, xmltodict, pywinrm
Successfully installed absl-py-0.2.0 blinker-1.4 colorama-0.3.9 colorlog-2.6.0 
contextlib2-0.5.5 futures-3.2.0 ntlm-auth-1.1.0 numpy-1.13.3 pint-0.8.1 
pywinrm-0.3.0 requests-ntlm-1.1.0 xmltodict-0.11.0
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins998352039491693429.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
Collecting avro<2.0.0,>=1.8.1 (from 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #120

[Apache Jenkins Server]

See 


Changes:

[github] Count compressed records with a long to avoid overflow

[alan] [BEAM-4218] Fix failing javadoc build

[github] Fix DynamicDestinations documentation

--
[...truncated 91.68 KB...]
:86:
 warning: [TypeParameterUnusedInFormals] Declaring a type parameter that is 
only used in the return type is a misuse of generics: operations on the type 
parameter are unchecked, it hides unsafe casts at invocations of the method, 
and it interacts badly with method overload resolution.
  > TrackerT 
invokeNewTracker(

^
(see http://errorprone.info/bugpattern/TypeParameterUnusedInFormals)
:285:
 warning: [MissingOverride] restrictionTracker implements method in 
ArgumentProvider; expected @Override
public RestrictionTracker restrictionTracker() {
^
(see http://errorprone.info/bugpattern/MissingOverride)
  Did you mean '@Override public RestrictionTracker restrictionTracker() 
{'?
:428:
 warning: [MissingOverride] expand implements method in PTransform; expected 
@Override
public PCollection expand(PCollection in) {
   ^
(see http://errorprone.info/bugpattern/MissingOverride)
  Did you mean '@Override public PCollection expand(PCollection 
in) {'?
:470:
 warning: [MissingOverride] expand implements method in PTransform; expected 
@Override
public PCollection expand(PCollection in) {
   ^
(see http://errorprone.info/bugpattern/MissingOverride)
  Did you mean '@Override public PCollection expand(PCollection 
in) {'?
:512:
 warning: [MissingOverride] expand implements method in PTransform; expected 
@Override
public PCollection expand(PCollection in) {
 ^
(see http://errorprone.info/bugpattern/MissingOverride)
  Did you mean '@Override public PCollection 
expand(PCollection in) {'?
:564:
 warning: [MissingOverride] expand implements method in PTransform; expected 
@Override
public PCollection> expand(PCollection in) {
   ^
(see http://errorprone.info/bugpattern/MissingOverride)
  Did you mean '@Override public PCollection> 
expand(PCollection in) {'?
:610:
 warning: [MissingOverride] expand implements method in PTransform; expected 
@Override
public PCollection> expand(PCollection in) {
   ^
(see http://errorprone.info/bugpattern/MissingOverride)
  Did you mean '@Override public PCollection> 
expand(PCollection in) {'?
:652:
 warning: [MissingOverride] expand implements method in PTransform; expected 
@Override
public PCollection expand(PCollection in) {
   ^
(see http://errorprone.info/bugpattern/MissingOverride)
  Did you mean '@Override public PCollection expand(PCollection 
in) {'?
:694:
 warning: [MissingOverride] expand implements method in PTransform; expected 
@Override
public PCollection expand(PCollection in) {
   ^
(see http://errorprone.info/bugpattern/MissingOverride)
  Did you mean '@Override public PCollection expand(PCollection 
in) {'?

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #122

[Apache Jenkins Server]

See 


Changes:

[github] Count compressed records with a long to avoid overflow

[alan] [BEAM-4218] Fix failing javadoc build

[github] Fix DynamicDestinations documentation

--
[...truncated 90.54 KB...]
  ^
(see http://errorprone.info/bugpattern/TypeParameterUnusedInFormals)
:86:
 warning: [TypeParameterUnusedInFormals] Declaring a type parameter that is 
only used in the return type is a misuse of generics: operations on the type 
parameter are unchecked, it hides unsafe casts at invocations of the method, 
and it interacts badly with method overload resolution.
  > TrackerT 
invokeNewTracker(

^
(see http://errorprone.info/bugpattern/TypeParameterUnusedInFormals)
:285:
 warning: [MissingOverride] restrictionTracker implements method in 
ArgumentProvider; expected @Override
public RestrictionTracker restrictionTracker() {
^
(see http://errorprone.info/bugpattern/MissingOverride)
  Did you mean '@Override public RestrictionTracker restrictionTracker() 
{'?
:428:
 warning: [MissingOverride] expand implements method in PTransform; expected 
@Override
public PCollection expand(PCollection in) {
   ^
(see http://errorprone.info/bugpattern/MissingOverride)
  Did you mean '@Override public PCollection expand(PCollection 
in) {'?
:470:
 warning: [MissingOverride] expand implements method in PTransform; expected 
@Override
public PCollection expand(PCollection in) {
   ^
(see http://errorprone.info/bugpattern/MissingOverride)
  Did you mean '@Override public PCollection expand(PCollection 
in) {'?
:512:
 warning: [MissingOverride] expand implements method in PTransform; expected 
@Override
public PCollection expand(PCollection in) {
 ^
(see http://errorprone.info/bugpattern/MissingOverride)
  Did you mean '@Override public PCollection 
expand(PCollection in) {'?
:564:
 warning: [MissingOverride] expand implements method in PTransform; expected 
@Override
public PCollection> expand(PCollection in) {
   ^
(see http://errorprone.info/bugpattern/MissingOverride)
  Did you mean '@Override public PCollection> 
expand(PCollection in) {'?
:610:
 warning: [MissingOverride] expand implements method in PTransform; expected 
@Override
public PCollection> expand(PCollection in) {
   ^
(see http://errorprone.info/bugpattern/MissingOverride)
  Did you mean '@Override public PCollection> 
expand(PCollection in) {'?
:652:
 warning: [MissingOverride] expand implements method in PTransform; expected 
@Override
public PCollection expand(PCollection in) {
   ^
(see http://errorprone.info/bugpattern/MissingOverride)
  Did you mean '@Override public PCollection expand(PCollection 
in) {'?
:694:
 warning: [MissingOverride] expand implements method in PTransform; expected 
@Override
public PCollection expand(PCollection in) {
   ^
(see 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #258

[Apache Jenkins Server]

See 


Changes:

[github] Fix DynamicDestinations documentation

--
[...truncated 17.90 MB...]
May 02, 2018 6:02:24 PM org.apache.beam.sdk.io.FileBasedSource split
INFO: Splitting filepattern 
/tmp/junit147560876988403/junit9128274499579332023.tmp into bundles of size 
3 took 0 ms and produced 1 files and 18 bundles

org.apache.beam.examples.WordCountTest > testExtractWordsFn STANDARD_ERROR
May 02, 2018 6:02:24 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.subprocess.ExampleEchoPipelineTest > 
testExampleEchoPipeline STANDARD_ERROR
May 02, 2018 6:02:25 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-Echo2923726372027294620.sh 
May 02, 2018 6:02:25 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
May 02, 2018 6:02:26 PM org.apache.beam.examples.subprocess.utils.FileUtils 
copyFileFromGCSToWorker
INFO: Moving File /tmp/test-Echo2923726372027294620.sh to 
/tmp/test-Echoo574535324822520/test-Echo2923726372027294620.sh 
May 02, 2018 6:02:26 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-EchoAgain9207570007618915658.sh 
May 02, 2018 6:02:26 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
May 02, 2018 6:02:26 PM org.apache.beam.examples.subprocess.utils.FileUtils 
copyFileFromGCSToWorker
INFO: Moving File /tmp/test-EchoAgain9207570007618915658.sh to 
/tmp/test-Echoo574535324822520/test-EchoAgain9207570007618915658.sh 

org.apache.beam.examples.complete.game.HourlyTeamScoreTest > 
testUserScoresFilter STANDARD_OUT
GOT user18_ApricotCaneToad,ApricotCaneToad,14,144796569,2015-11-19 
12:41:31.053
GOT user18_BananaEmu,BananaEmu,1,144796569,2015-11-19 12:41:31.053
GOT user7_AlmondWallaby,AlmondWallaby,15,144795563,2015-11-19 
09:53:53.444
GOT user18_BananaEmu,BananaEmu,7,144796569,2015-11-19 12:41:31.053
GOT user0_MagentaKangaroo,MagentaKangaroo,3,144795563,2015-11-19 
09:53:53.444
GOT user0_MagentaKangaroo,MagentaKangaroo,4,144796569,2015-11-19 
12:41:31.053
GOT user2_AmberCockatoo,AmberCockatoo,13,144796569,2015-11-19 
12:41:31.053
GOT 
user0_AndroidGreenEchidna,AndroidGreenEchidna,0,144796569,2015-11-19 
12:41:31.053
GOT user13_ApricotQuokka,ApricotQuokka,15,144795563,2015-11-19 
09:53:53.444
GOT user19_BisqueBilby,BisqueBilby,6,144795563,2015-11-19 09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,11,144795563,2015-11-19 
09:53:53.444
GOT user19_BisqueBilby,BisqueBilby,8,144795563,2015-11-19 09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,12,144795563,2015-11-19 
09:53:53.444
GOT user3_BananaEmu,BananaEmu,17,144796569,2015-11-19 12:41:31.053
GOT user6_AmberNumbat,AmberNumbat,11,144795563,2015-11-19 09:53:53.444

org.apache.beam.examples.complete.game.UserScoreTest > testTeamScoreSums 
STANDARD_OUT
GOT user7_AlmondWallaby,AlmondWallaby,15,144795563,2015-11-19 
09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,11,144795563,2015-11-19 
09:53:53.444
GOT user0_MagentaKangaroo,MagentaKangaroo,3,144795563,2015-11-19 
09:53:53.444
GOT THIS IS A PARSE ERROR,2015-11-19 09:53:53.444
GOT user6_AmberNumbat,AmberNumbat,11,144795563,2015-11-19 09:53:53.444
GOT user13_ApricotQuokka,ApricotQuokka,15,144795563,2015-11-19 
09:53:53.444
GOT user19_BisqueBilby,BisqueBilby,6,144795563,2015-11-19 09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,12,144795563,2015-11-19 
09:53:53.444
GOT user19_BisqueBilby,BisqueBilby,8,144795563,2015-11-19 09:53:53.444

org.apache.beam.examples.complete.game.UserScoreTest > testTeamScoreSums 
STANDARD_ERROR
May 02, 2018 6:02:27 PM 
org.apache.beam.examples.complete.game.UserScore$ParseEventFn processElement
INFO: Parse error on THIS IS A PARSE ERROR,2015-11-19 09:53:53.444, 2

org.apache.beam.examples.complete.game.UserScoreTest > testTeamScoreSums 
STANDARD_OUT
GOT user6_AliceBlueDingo,AliceBlueDingo,4,xxx,2015-11-19 09:53:53.444

org.apache.beam.examples.complete.game.UserScoreTest > testTeamScoreSums 
STANDARD_ERROR
May 02, 2018 6:02:27 PM 
org.apache.beam.examples.complete.game.UserScore$ParseEventFn processElement
INFO: Parse error on 
user6_AliceBlueDingo,AliceBlueDingo,4,xxx,2015-11-19 09:53:53.444, For 
input string: "xxx"

[jira] [Work logged] (BEAM-3327) Add abstractions to manage Environment Instance lifecycles.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3327?focusedWorklogId=97612=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97612
 ]

ASF GitHub Bot logged work on BEAM-3327:


Author: ASF GitHub Bot
Created on: 02/May/18 17:57
Start Date: 02/May/18 17:57
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on a change in pull request #5152: 
[BEAM-3327] Harness Manager Interfaces
URL: https://github.com/apache/beam/pull/5152#discussion_r185584927
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/OutputReceiverFactory.java
 ##
 @@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.fnexecution.control;
+
+/**
+ * A factory that can create output receivers during an executable stage.
+ */
+public interface OutputReceiverFactory {
+  /**
+   * Get a new {@link RemoteOutputReceiver} for an output PCollection.
+   */
+   RemoteOutputReceiver create(String pCollectionId);
 
 Review comment:
   fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97612)
Time Spent: 28.5h  (was: 28h 20m)

> Add abstractions to manage Environment Instance lifecycles.
> ---
>
> Key: BEAM-3327
> URL: https://issues.apache.org/jira/browse/BEAM-3327
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Axel Magnuson
>Priority: Major
>  Labels: portability
>  Time Spent: 28.5h
>  Remaining Estimate: 0h
>
> This permits remote stage execution for arbitrary environments



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3658) Port SpannerIOReadTest off DoFnTester

[Almas Shaikh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461398#comment-16461398
 ] 

Almas Shaikh commented on BEAM-3658:


[~kenn] What needs to be done here please?

> Port SpannerIOReadTest off DoFnTester
> -
>
> Key: BEAM-3658
> URL: https://issues.apache.org/jira/browse/BEAM-3658
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-gcp
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3327) Add abstractions to manage Environment Instance lifecycles.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3327?focusedWorklogId=97608=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97608
 ]

ASF GitHub Bot logged work on BEAM-3327:


Author: ASF GitHub Bot
Created on: 02/May/18 17:40
Start Date: 02/May/18 17:40
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5152: 
[BEAM-3327] Harness Manager Interfaces
URL: https://github.com/apache/beam/pull/5152#discussion_r185579308
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/OutputReceiverFactory.java
 ##
 @@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.fnexecution.control;
+
+/**
+ * A factory that can create output receivers during an executable stage.
+ */
+public interface OutputReceiverFactory {
+  /**
+   * Get a new {@link RemoteOutputReceiver} for an output PCollection.
+   */
+   RemoteOutputReceiver create(String pCollectionId);
 
 Review comment:
   This looks like it should be stateless, but as discussed with @bsidhom and 
myself, it's probable that within a single instance the caller should only ever 
invoke it once per `PCollection` - that ability should be commented explicitly, 
just so other integrators can understand permitted scoping effectively.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97608)
Time Spent: 28h 20m  (was: 28h 10m)

> Add abstractions to manage Environment Instance lifecycles.
> ---
>
> Key: BEAM-3327
> URL: https://issues.apache.org/jira/browse/BEAM-3327
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Axel Magnuson
>Priority: Major
>  Labels: portability
>  Time Spent: 28h 20m
>  Remaining Estimate: 0h
>
> This permits remote stage execution for arbitrary environments



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3327) Add abstractions to manage Environment Instance lifecycles.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3327?focusedWorklogId=97607=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97607
 ]

ASF GitHub Bot logged work on BEAM-3327:


Author: ASF GitHub Bot
Created on: 02/May/18 17:37
Start Date: 02/May/18 17:37
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on a change in pull request #5152: 
[BEAM-3327] Harness Manager Interfaces
URL: https://github.com/apache/beam/pull/5152#discussion_r185578236
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/StageBundleFactory.java
 ##
 @@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.fnexecution.control;
+
+/**
+ * A bundle factory scoped to a particular
+ * {@link org.apache.beam.runners.core.construction.graph.ExecutableStage}, 
which has all of the
+ * resources it needs to provide new {@link RemoteBundle RemoteBundles}.
+ *
+ * Closing a StageBundleFactory signals that a a stage has completed and 
any resources bound to
+ * its lifetime can be cleaned up.
+ */
+public interface StageBundleFactory extends AutoCloseable {
+  /**
+   * Get a new {@link RemoteBundle bundle} for processing the data in an 
executable stage.
 
 Review comment:
   patched my commit to implement this.  Let me know if you had something 
different in mind.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97607)
Time Spent: 28h 10m  (was: 28h)

> Add abstractions to manage Environment Instance lifecycles.
> ---
>
> Key: BEAM-3327
> URL: https://issues.apache.org/jira/browse/BEAM-3327
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Axel Magnuson
>Priority: Major
>  Labels: portability
>  Time Spent: 28h 10m
>  Remaining Estimate: 0h
>
> This permits remote stage execution for arbitrary environments



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #257

[Apache Jenkins Server]

See 


Changes:

[github] Count compressed records with a long to avoid overflow

--
[...truncated 18.00 MB...]
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.subprocess.ExampleEchoPipelineTest > 
testExampleEchoPipeline STANDARD_ERROR
May 02, 2018 5:30:55 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-Echo2336041064492455357.sh 
May 02, 2018 5:30:55 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
May 02, 2018 5:30:55 PM org.apache.beam.examples.subprocess.utils.FileUtils 
copyFileFromGCSToWorker
INFO: Moving File /tmp/test-Echo2336041064492455357.sh to 
/tmp/test-Echoo9205932006238815312/test-Echo2336041064492455357.sh 
May 02, 2018 5:30:55 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-EchoAgain5511295495971032017.sh 
May 02, 2018 5:30:55 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
May 02, 2018 5:30:55 PM org.apache.beam.examples.subprocess.utils.FileUtils 
copyFileFromGCSToWorker
INFO: Moving File /tmp/test-EchoAgain5511295495971032017.sh to 
/tmp/test-Echoo9205932006238815312/test-EchoAgain5511295495971032017.sh 

org.apache.beam.examples.complete.game.HourlyTeamScoreTest > 
testUserScoresFilter STANDARD_OUT
GOT user0_MagentaKangaroo,MagentaKangaroo,3,144795563,2015-11-19 
09:53:53.444
GOT user18_ApricotCaneToad,ApricotCaneToad,14,144796569,2015-11-19 
12:41:31.053
GOT user18_BananaEmu,BananaEmu,7,144796569,2015-11-19 12:41:31.053
GOT user7_AlmondWallaby,AlmondWallaby,15,144795563,2015-11-19 
09:53:53.444
GOT user0_MagentaKangaroo,MagentaKangaroo,4,144796569,2015-11-19 
12:41:31.053
GOT user18_BananaEmu,BananaEmu,1,144796569,2015-11-19 12:41:31.053
GOT user2_AmberCockatoo,AmberCockatoo,13,144796569,2015-11-19 
12:41:31.053
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,12,144795563,2015-11-19 
09:53:53.444
GOT user3_BananaEmu,BananaEmu,17,144796569,2015-11-19 12:41:31.053
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,11,144795563,2015-11-19 
09:53:53.444
GOT user6_AmberNumbat,AmberNumbat,11,144795563,2015-11-19 09:53:53.444
GOT user19_BisqueBilby,BisqueBilby,8,144795563,2015-11-19 09:53:53.444
GOT user19_BisqueBilby,BisqueBilby,6,144795563,2015-11-19 09:53:53.444
GOT 
user0_AndroidGreenEchidna,AndroidGreenEchidna,0,144796569,2015-11-19 
12:41:31.053
GOT user13_ApricotQuokka,ApricotQuokka,15,144795563,2015-11-19 
09:53:53.444

org.apache.beam.examples.complete.game.UserScoreTest > testTeamScoreSums 
STANDARD_OUT
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,12,144795563,2015-11-19 
09:53:53.444
GOT user19_BisqueBilby,BisqueBilby,8,144795563,2015-11-19 09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,11,144795563,2015-11-19 
09:53:53.444
GOT user7_AlmondWallaby,AlmondWallaby,15,144795563,2015-11-19 
09:53:53.444
GOT user13_ApricotQuokka,ApricotQuokka,15,144795563,2015-11-19 
09:53:53.444
GOT user19_BisqueBilby,BisqueBilby,6,144795563,2015-11-19 09:53:53.444
GOT user0_MagentaKangaroo,MagentaKangaroo,3,144795563,2015-11-19 
09:53:53.444
GOT user6_AmberNumbat,AmberNumbat,11,144795563,2015-11-19 09:53:53.444
GOT user6_AliceBlueDingo,AliceBlueDingo,4,xxx,2015-11-19 09:53:53.444
GOT THIS IS A PARSE ERROR,2015-11-19 09:53:53.444

org.apache.beam.examples.complete.game.UserScoreTest > testTeamScoreSums 
STANDARD_ERROR
May 02, 2018 5:30:56 PM 
org.apache.beam.examples.complete.game.UserScore$ParseEventFn processElement
INFO: Parse error on THIS IS A PARSE ERROR,2015-11-19 09:53:53.444, 2
May 02, 2018 5:30:56 PM 
org.apache.beam.examples.complete.game.UserScore$ParseEventFn processElement
INFO: Parse error on 
user6_AliceBlueDingo,AliceBlueDingo,4,xxx,2015-11-19 09:53:53.444, For 
input string: "xxx"

org.apache.beam.examples.complete.game.UserScoreTest > testUserScoresBadInput 
STANDARD_OUT
GOT user6_AliceBlueDingo,AliceBlueDingo,4,xxx,2015-11-19 09:53:53.444
GOT THIS IS A PARSE ERROR,2015-11-19 09:53:53.444
GOT user13_BisqueBilby,BisqueBilby,xxx,144795563,2015-11-19 09:53:53.444

org.apache.beam.examples.complete.game.UserScoreTest > testUserScoresBadInput 
STANDARD_ERROR
May 02, 2018 5:30:57 PM 
org.apache.beam.examples.complete.game.UserScore$ParseEventFn processElement
INFO: Parse error on THIS IS A PARSE 

[jira] [Work logged] (BEAM-3327) Add abstractions to manage Environment Instance lifecycles.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3327?focusedWorklogId=97597=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97597
 ]

ASF GitHub Bot logged work on BEAM-3327:


Author: ASF GitHub Bot
Created on: 02/May/18 17:21
Start Date: 02/May/18 17:21
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5152: 
[BEAM-3327] Harness Manager Interfaces
URL: https://github.com/apache/beam/pull/5152#discussion_r185572115
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/StageBundleFactory.java
 ##
 @@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.fnexecution.control;
+
+/**
+ * A bundle factory scoped to a particular
+ * {@link org.apache.beam.runners.core.construction.graph.ExecutableStage}, 
which has all of the
+ * resources it needs to provide new {@link RemoteBundle RemoteBundles}.
+ *
+ * Closing a StageBundleFactory signals that a a stage has completed and 
any resources bound to
+ * its lifetime can be cleaned up.
+ */
+public interface StageBundleFactory extends AutoCloseable {
+  /**
+   * Get a new {@link RemoteBundle bundle} for processing the data in an 
executable stage.
 
 Review comment:
   Instead of going for a map of `target -> output receiver`, I'd go for a 
factory that consumes `PCollectionId` which produces the appropriate 
`RemoteOutputReceiver` - this may be extended into consuming the `PTransformId` 
which produced the partial `PCollection` that the target represents.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97597)
Time Spent: 28h  (was: 27h 50m)

> Add abstractions to manage Environment Instance lifecycles.
> ---
>
> Key: BEAM-3327
> URL: https://issues.apache.org/jira/browse/BEAM-3327
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Axel Magnuson
>Priority: Major
>  Labels: portability
>  Time Spent: 28h
>  Remaining Estimate: 0h
>
> This permits remote stage execution for arbitrary environments



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3327) Add abstractions to manage Environment Instance lifecycles.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3327?focusedWorklogId=97596=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97596
 ]

ASF GitHub Bot logged work on BEAM-3327:


Author: ASF GitHub Bot
Created on: 02/May/18 17:20
Start Date: 02/May/18 17:20
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5152: 
[BEAM-3327] Harness Manager Interfaces
URL: https://github.com/apache/beam/pull/5152#discussion_r185572115
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/StageBundleFactory.java
 ##
 @@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.fnexecution.control;
+
+/**
+ * A bundle factory scoped to a particular
+ * {@link org.apache.beam.runners.core.construction.graph.ExecutableStage}, 
which has all of the
+ * resources it needs to provide new {@link RemoteBundle RemoteBundles}.
+ *
+ * Closing a StageBundleFactory signals that a a stage has completed and 
any resources bound to
+ * its lifetime can be cleaned up.
+ */
+public interface StageBundleFactory extends AutoCloseable {
+  /**
+   * Get a new {@link RemoteBundle bundle} for processing the data in an 
executable stage.
 
 Review comment:
   Instead of going for a map of `target -> output receiver`, I'd go for a 
factory that consumes `PTransformId` and `PCollectionId` which produces the 
appropriate `RemoteOutputReceiver`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97596)
Time Spent: 27h 50m  (was: 27h 40m)

> Add abstractions to manage Environment Instance lifecycles.
> ---
>
> Key: BEAM-3327
> URL: https://issues.apache.org/jira/browse/BEAM-3327
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Axel Magnuson
>Priority: Major
>  Labels: portability
>  Time Spent: 27h 50m
>  Remaining Estimate: 0h
>
> This permits remote stage execution for arbitrary environments



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1505

[Apache Jenkins Server]

See 


Changes:

[github] Fix DynamicDestinations documentation

--
Started by GitHub push by lukecwik
[EnvInject] - Loading node environment variables.
Building remotely on beam17 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 971f1f11d09af8545b17eb23da3d3fd4a80055a2 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 971f1f11d09af8545b17eb23da3d3fd4a80055a2
Commit message: "Fix DynamicDestinations documentation"
 > git rev-list --no-walk 69293a7c5a75187f38ead3ae9ef69b867ee62993 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins6839563411572941403.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user ankurgoe...@gmail.com

[beam] 01/01: Fix DynamicDestinations documentation

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 971f1f11d09af8545b17eb23da3d3fd4a80055a2
Merge: 69293a7 8dbf251
Author: Lukasz Cwik 
AuthorDate: Wed May 2 10:19:37 2018 -0700

Fix DynamicDestinations documentation

 .../java/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinations.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] branch master updated (69293a7 -> 971f1f1)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 69293a7  Count compressed records with a long to avoid overflow
 add 8dbf251  Fix DynamicDestinations documentation
 new 971f1f1  Fix DynamicDestinations documentation

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../java/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinations.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Work logged] (BEAM-3119) direct-metrics-counter-committer threads are leaking

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3119?focusedWorklogId=97594=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97594
 ]

ASF GitHub Bot logged work on BEAM-3119:


Author: ASF GitHub Bot
Created on: 02/May/18 17:17
Start Date: 02/May/18 17:17
Worklog Time Spent: 10m 
  Work Description: rmannibucau commented on issue #4965: BEAM-3119 ensure 
the metrics thread pool is related to an execution
URL: https://github.com/apache/beam/pull/4965#issuecomment-386053451
 
 
   Will not help but the timeout should be infinite for this one since it must 
stop before exiting. Also the build env should probably guarantee the machine 
is not overloaded, 10s for that test is likely more than 9s of a blocked 
machine cause of a lack of resource.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97594)
Time Spent: 5h  (was: 4h 50m)

> direct-metrics-counter-committer threads are leaking
> 
>
> Key: BEAM-3119
> URL: https://issues.apache.org/jira/browse/BEAM-3119
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Etienne Chauchot
>Assignee: Romain Manni-Bucau
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> When I run ElasticsearchIOTests using ESv5, there is a thread leak control 
> mechanism ({{com.carrotsearch.randomizedtesting.ThreadLeakControl}}). It 
> waits for 5s for non-terminated threads at the end of a test. It detects 
> leaked {{direct-metrics-counter-committer}} thread.
> {code}
> com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
> threads that couldn't be terminated:
>1) Thread[id=296, name=direct-metrics-counter-committer, 
> state=TIMED_WAITING, group=TGRP-ElasticsearchIOTest]
> at sun.misc.Unsafe.park(Native Method)
> at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>   at __randomizedtesting.SeedInfo.seed([59E504CA1B0DD6A8]:0){code}
> I tried to increase the timeout to 30s (by patching 
> randomizedtesting-runner-2.5.0.jar) but still gets a zombie thread.
> To reproduce, just comment 
> {code}
> @ThreadLeakScope(ThreadLeakScope.Scope.NONE)
> {code}
>  in 
> {code}
> beam/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java
> {code}
> and run 
> {code}
> testRead()
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3327) Add abstractions to manage Environment Instance lifecycles.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3327?focusedWorklogId=97592=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97592
 ]

ASF GitHub Bot logged work on BEAM-3327:


Author: ASF GitHub Bot
Created on: 02/May/18 17:16
Start Date: 02/May/18 17:16
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5152: 
[BEAM-3327] Harness Manager Interfaces
URL: https://github.com/apache/beam/pull/5152#discussion_r185572115
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/StageBundleFactory.java
 ##
 @@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.fnexecution.control;
+
+/**
+ * A bundle factory scoped to a particular
+ * {@link org.apache.beam.runners.core.construction.graph.ExecutableStage}, 
which has all of the
+ * resources it needs to provide new {@link RemoteBundle RemoteBundles}.
+ *
+ * Closing a StageBundleFactory signals that a a stage has completed and 
any resources bound to
+ * its lifetime can be cleaned up.
+ */
+public interface StageBundleFactory extends AutoCloseable {
+  /**
+   * Get a new {@link RemoteBundle bundle} for processing the data in an 
executable stage.
 
 Review comment:
   Instead of going for a map of `target -> output receiver`, I'd go for a 
factory that consumes `ProcessBundleDescriptor` and `Target` which produces the 
appropriate `RemoteOutputReceiver`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97592)
Time Spent: 27h 40m  (was: 27.5h)

> Add abstractions to manage Environment Instance lifecycles.
> ---
>
> Key: BEAM-3327
> URL: https://issues.apache.org/jira/browse/BEAM-3327
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Axel Magnuson
>Priority: Major
>  Labels: portability
>  Time Spent: 27h 40m
>  Remaining Estimate: 0h
>
> This permits remote stage execution for arbitrary environments



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #256

[Apache Jenkins Server]

See 


Changes:

[alan] [BEAM-4218] Fix failing javadoc build

--
[...truncated 19.03 MB...]
INFO: 2018-05-02T17:00:57.532Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 02, 2018 5:00:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T17:00:57.554Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 02, 2018 5:00:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T17:00:57.578Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
May 02, 2018 5:00:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T17:00:57.597Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
May 02, 2018 5:00:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T17:00:57.618Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
May 02, 2018 5:00:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T17:00:57.643Z: Fusing consumer 
PAssert$3/CreateActual/RewindowActuals/Window.Assign into 
PAssert$3/CreateActual/Flatten.Iterables/FlattenIterables/FlatMap
May 02, 2018 5:00:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T17:00:57.664Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
May 02, 2018 5:00:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T17:00:57.677Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
May 02, 2018 5:00:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T17:00:57.701Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
May 02, 2018 5:00:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T17:00:57.723Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
May 02, 2018 5:00:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-02T17:00:57.745Z: Fusing 

[jira] [Work logged] (BEAM-3119) direct-metrics-counter-committer threads are leaking

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3119?focusedWorklogId=97580=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97580
 ]

ASF GitHub Bot logged work on BEAM-3119:


Author: ASF GitHub Bot
Created on: 02/May/18 16:56
Start Date: 02/May/18 16:56
Worklog Time Spent: 10m 
  Work Description: echauchot commented on a change in pull request #4965: 
BEAM-3119 ensure the metrics thread pool is related to an execution
URL: https://github.com/apache/beam/pull/4965#discussion_r185566321
 
 

 ##
 File path: 
runners/direct-java/src/test/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutorTest.java
 ##
 @@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.direct;
+
+import static java.util.Collections.emptyMap;
+import static java.util.Collections.emptySet;
+
+import com.google.common.collect.LinkedListMultimap;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.ThreadLeakTracker;
+import org.joda.time.Instant;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TestName;
+import org.junit.rules.TestRule;
+
+/**
+ * Validates some basic behavior for the ExecutorServiceParallelExecutor.
+ */
+public class ExecutorServiceParallelExecutorTest {
+
+  @Rule
+  public final TestRule threadTracker = new ThreadLeakTracker();
+
+  @Rule
+  public final TestName testName = new TestName();
+
+  @Test
+  public void ensureMetricsThreadDoesntLeak() {
 
 Review comment:
   @swegner Thanks for pointing out. The test failure shows that the thread 
created to fake a metrics call in 
ExecutorServiceParallelExecutorTest#ensureMetricsThreadDoesntLeak sometimes 
leak. This can happen if this thread is not stopped before the timeout of 10s 
after the call of shutdown() on its metricsExecutorService. The call of 
shutdown() is done inside the test itself via the call of stop() on 
ExecutorServiceParallelExecutor so I don't think #5096 could fix it.  
@rmannibucau any comment on #5096?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97580)
Time Spent: 4h 50m  (was: 4h 40m)

> direct-metrics-counter-committer threads are leaking
> 
>
> Key: BEAM-3119
> URL: https://issues.apache.org/jira/browse/BEAM-3119
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Etienne Chauchot
>Assignee: Romain Manni-Bucau
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> When I run ElasticsearchIOTests using ESv5, there is a thread leak control 
> mechanism ({{com.carrotsearch.randomizedtesting.ThreadLeakControl}}). It 
> waits for 5s for non-terminated threads at the end of a test. It detects 
> leaked {{direct-metrics-counter-committer}} thread.
> {code}
> com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
> threads that couldn't be terminated:
>1) Thread[id=296, name=direct-metrics-counter-committer, 
> state=TIMED_WAITING, group=TGRP-ElasticsearchIOTest]
> at sun.misc.Unsafe.park(Native Method)
> at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at 
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at 
> 

[jira] [Comment Edited] (BEAM-4222) Python minimal wordcount needs cleanup

[Melissa Pashniak (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460240#comment-16460240
 ] 

Melissa Pashniak edited comment on BEAM-4222 at 5/2/18 4:54 PM:


Additional notes:
 * Unlike Java, Python needs to install the gcp specific package to get the 
shakespeare GCS file, so --input may be required unless we require users to 
install gcp package to run it (probably not a good idea)
 * We could set --output to default to "wordcounts" like Java, but then the 
user would need to only specify --input and not --output which might be 
confusing?
 * Either way, source file needs a lot of cleanup (references long removed 
things like BlockingDataflowRunner)


was (Author: melap):
Additional notes:
 * Unlike Java, Python needs to install the gcp specific package to get the 
shakespeare GCS file, so --input may be required unless we require users to 
install gcp package to run it
 * We could set --output to default to "wordcounts" like Java, but then the 
user would need to specify --input and not --output which might be confusing?
 * Either way, source file needs a lot of cleanup (references long removed 
things like BlockingDataflowRunner)

> Python minimal wordcount needs cleanup
> --
>
> Key: BEAM-4222
> URL: https://issues.apache.org/jira/browse/BEAM-4222
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>Priority: Minor
>
> The minimal wordcount example is meant to be very simple and not require the 
> user to set a bunch of options. However, the Python minimal wordcount 
> requires setting options (including output). Ideally this should match the 
> Java version (runs on Direct runner without needing many options) as closely 
> as possible



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4222) Python minimal wordcount needs cleanup

[Melissa Pashniak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Melissa Pashniak updated BEAM-4222:
---
Description: The minimal wordcount example is meant to be very simple and 
not require the user to set a bunch of options. However, the Python minimal 
wordcount requires setting options (including output). Ideally this should 
match the Java version (runs on Direct runner without needing many options) as 
closely as possible  (was: The minimal wordcount example is meant to be very 
simple and not require the user to set a bunch of options. However, the Python 
minimal wordcount requires setting options (including output). Ideally this 
should match the Java version (runs on Direct runner without needing many 
options))

> Python minimal wordcount needs cleanup
> --
>
> Key: BEAM-4222
> URL: https://issues.apache.org/jira/browse/BEAM-4222
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>Priority: Minor
>
> The minimal wordcount example is meant to be very simple and not require the 
> user to set a bunch of options. However, the Python minimal wordcount 
> requires setting options (including output). Ideally this should match the 
> Java version (runs on Direct runner without needing many options) as closely 
> as possible



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4222) Python minimal wordcount needs cleanup

[Melissa Pashniak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Melissa Pashniak updated BEAM-4222:
---
Summary: Python minimal wordcount needs cleanup  (was: Python minimal 
wordcount is not minimal)

> Python minimal wordcount needs cleanup
> --
>
> Key: BEAM-4222
> URL: https://issues.apache.org/jira/browse/BEAM-4222
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>Priority: Minor
>
> The minimal wordcount example is meant to be very simple and not require the 
> user to set a bunch of options. However, the Python minimal wordcount 
> requires setting options (including output). Ideally this should match the 
> Java version (runs on Direct runner without needing many options)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)