Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #279

[Apache Jenkins Server]

See 


Changes:

[apilloud] [SQL] Use the same javacc and fmpp as calcite

[apilloud] [SQL] BeamQueryPlanner uses BeamSqlParserImpl

--
[...truncated 19.63 MB...]
May 04, 2018 5:26:23 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T05:26:19.400Z: Fusing unzipped copy of 
PAssert$3/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous), through 
flatten s18-u63, into producer 
PAssert$3/CreateActual/FilterActuals/Window.Assign
May 04, 2018 5:26:23 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T05:26:19.437Z: Fusing consumer 
PAssert$3/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous) into 
PAssert$3/CreateActual/FilterActuals/Window.Assign
May 04, 2018 5:26:23 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T05:26:19.474Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 04, 2018 5:26:23 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T05:26:19.509Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 04, 2018 5:26:23 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T05:26:19.550Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
May 04, 2018 5:26:23 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T05:26:19.595Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
May 04, 2018 5:26:23 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T05:26:19.630Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
May 04, 2018 5:26:23 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T05:26:19.664Z: Fusing consumer 
PAssert$3/CreateActual/RewindowActuals/Window.Assign into 
PAssert$3/CreateActual/Flatten.Iterables/FlattenIterables/FlatMap
May 04, 2018 5:26:23 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T05:26:19.709Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
May 04, 2018 5:26:23 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T05:26:19.748Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
May 04, 2018 5:26:23 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T05:26:19.780Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
 into 

[jira] [Commented] (BEAM-3658) Port SpannerIOReadTest off DoFnTester

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463371#comment-16463371
 ] 

Kenneth Knowles commented on BEAM-3658:
---

[~eralmas7] DoFnTester is deprecated and tests that use it are probably 
overspecified (and maybe broken anyhow). So any use of DoFnTester in the tests 
should be replaced by use of TestPipeline and maybe TestStream if you need it. 
Is that enough information to get started?

> Port SpannerIOReadTest off DoFnTester
> -
>
> Key: BEAM-3658
> URL: https://issues.apache.org/jira/browse/BEAM-3658
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-gcp
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3489) Expose the message id of received messages within PubsubMessage

[Justin Tumale (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justin Tumale reassigned BEAM-3489:
---

Assignee: Justin Tumale

> Expose the message id of received messages within PubsubMessage
> ---
>
> Key: BEAM-3489
> URL: https://issues.apache.org/jira/browse/BEAM-3489
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-gcp
>Reporter: Luke Cwik
>Assignee: Justin Tumale
>Priority: Minor
>  Labels: newbie, starter
>
> This task is about passing forward the message id from the pubsub proto to 
> the java PubsubMessage.
> Add a message id field to PubsubMessage.
> Update the coder for PubsubMessage to encode the message id.
> Update the translation from the Pubsub proto message to the Dataflow message:
> https://github.com/apache/beam/blob/2e275264b21db45787833502e5e42907b05e28b8/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java#L976



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1526

[Apache Jenkins Server]

See 


Changes:

[apilloud] [SQL] Use the same javacc and fmpp as calcite

[apilloud] [SQL] BeamQueryPlanner uses BeamSqlParserImpl

--
Started by GitHub push by kennknowles
[EnvInject] - Loading node environment variables.
Building remotely on beam23 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 8ef71b6eb1d2d5c63974ec506a01faf3813efe74 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8ef71b6eb1d2d5c63974ec506a01faf3813efe74
Commit message: "Merge pull request #5275: [BEAM-4233] [SQL] Use the same 
javacc and fmpp as calcite"
 > git rev-list --no-walk 86fdb74fd7cbc016065cfe6dea530155051a6f39 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins329871054207621139.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user mair...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user jasonkus...@google.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user ankurgoe...@gmail.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com

[jira] [Work logged] (BEAM-4233) SQL generated code don't match between Maven and Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4233?focusedWorklogId=98112=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98112
 ]

ASF GitHub Bot logged work on BEAM-4233:


Author: ASF GitHub Bot
Created on: 04/May/18 04:35
Start Date: 04/May/18 04:35
Worklog Time Spent: 10m 
  Work Description: kennknowles closed pull request #5275: [BEAM-4233] 
[SQL] Use the same javacc and fmpp as calcite
URL: https://github.com/apache/beam/pull/5275
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/java/extensions/sql/build.gradle 
b/sdks/java/extensions/sql/build.gradle
index a286cf437f2..dcf98586a14 100644
--- a/sdks/java/extensions/sql/build.gradle
+++ b/sdks/java/extensions/sql/build.gradle
@@ -48,7 +48,8 @@ def calcite_version = "1.16.0"
 def avatica_version = "1.11.0"
 
 dependencies {
-  fmppTask "net.sourceforge.fmpp:fmpp:0.9.15"
+  javacc "net.java.dev.javacc:javacc:4.0"
+  fmppTask "com.googlecode.fmpp-maven-plugin:fmpp-maven-plugin:1.0"
   fmppTask "org.freemarker:freemarker:2.3.25-incubating"
   fmppTemplates "org.apache.calcite:calcite-core:$calcite_version"
   compile library.java.guava
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamQueryPlanner.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamQueryPlanner.java
index 348223e3851..36603f1e71b 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamQueryPlanner.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamQueryPlanner.java
@@ -22,6 +22,7 @@
 import java.util.List;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.impl.parser.impl.BeamSqlParserImpl;
 import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
 import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
 import org.apache.beam.sdk.values.PCollection;
@@ -91,7 +92,10 @@ public BeamQueryPlanner(BeamSqlEnv sqlEnv, SchemaPlus 
schema) {
 
 FrameworkConfig config =
 Frameworks.newConfigBuilder()
-.parserConfig(SqlParser.configBuilder().setLex(Lex.MYSQL).build())
+.parserConfig(SqlParser.configBuilder()
+.setLex(Lex.MYSQL)
+.setParserFactory(BeamSqlParserImpl.FACTORY)
+.build())
 .defaultSchema(schema)
 .traitDefs(traitDefs)
 .context(Contexts.EMPTY_CONTEXT)


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98112)
Time Spent: 50m  (was: 40m)

> SQL generated code don't match between Maven and Gradle 
> 
>
> Key: BEAM-4233
> URL: https://issues.apache.org/jira/browse/BEAM-4233
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> The generated code between maven and gradle don't match. Particularly in the 
> SQL parser. This causes the behavior of our generated SQL parser to be 
> incorrect when built from gradle.
> The Maven plugin is using JavaCC version 4.0 while the Gradle plugin is using 
> version 6.1.2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #121

[Apache Jenkins Server]

See 


Changes:

[axelmagn] Create ArtifactSource Interface

[echauchot] [BEAM-4138] Support runners that do not support committed metrics in

[tgroh] Add KV and Window Coder Utilities to ModelCoders

[tgroh] Use ModelCoders in ProcessBundleDescriptors

[apilloud] [SQL] Add BeamEnumerableConverter

[tgroh] Remove unused Test Utilities, Test

[github] Count compressed records with a long to avoid overflow

[axelmagn] Create RemoteBundle interface

[axelmagn] Create JobInfo for job declaration by operators.

[mairbek] Wait for input collection

[mairbek] Expose DatabaseAdminClient

[mairbek] Consistently use lower case table name and column names

[alan] [BEAM-4218] Fix failing javadoc build

[github] Fix DynamicDestinations documentation

[axelmagn] Create BundleFactory chain.

[mairbek] Addressed comments

[sidhom] Remove StateRequestHandler from JobBundleFactory

[mairbek] Templatize host name in SpannerIO

[coheigea] A few small fixes to catch a few regressions - put String literals

[sidhom] Move SyntheticNodes to top-level construction package and rename

[sidhom] Pull Flink execution environment utilities into static methods

[sidhom] Refactor Runner/SDK wire coder creation into a shared utility

[sidhom] [BEAM-3972] Translate portable batch pipelines by proto

[sidhom] FIXUP: Use ModelCoders to create proto coders and use WireCoders to

[sidhom] FIXUP: Use RunnerApi in WireCoders to prepare for coder instantiation

[tgroh] Add additional methods to QueryablePipeline

[tgroh] Expand the ExecutableGraph interface

[tgroh] Add ProtoOverrides

[jasonkuster] Add labels to Go SDK Dataflow Runner.

[jasonkuster] Add verbose error message if unmarshaling fails for label flag.

[tgroh] Add an Unsupported StateRequestHandler

--
[...truncated 477 B...]
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 86fdb74fd7cbc016065cfe6dea530155051a6f39 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 86fdb74fd7cbc016065cfe6dea530155051a6f39
Commit message: "Merge pull request #5269: Add an Unsupported 
StateRequestHandler"
 > git rev-list --no-walk 5e995f78ffcbde61d61bcfcd662ec41f2218dbbc # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesContainer_Dataflow] $ /bin/bash -xe 
/tmp/jenkins2980714578009329173.sh
+ cd src
+ bash sdks/python/container/run_validatescontainer.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# Where to store integration test outputs.
GCS_LOCATION=gs://temp-storage-for-end-to-end-tests

# Project for the container and integration test
PROJECT=apache-beam-testing

# Verify in the root of the repository
test -d sdks/python/container

# Verify docker and gcloud commands exist
command -v docker
/usr/bin/docker
command -v gcloud
/usr/bin/gcloud
docker -v
Docker version 17.05.0-ce, build 89658be
gcloud -v
Google Cloud SDK 191.0.0
alpha 2018.02.23
beta 2018.02.23
bq 2.0.29
core 2018.02.23
gsutil 4.28

# ensure gcloud is version 186 or above
TMPDIR=$(mktemp -d)
mktemp -d
gcloud_ver=$(gcloud -v | head -1 | awk '{print $4}')
gcloud -v | head -1 | awk '{print $4}'
if [[ "$gcloud_ver" < "186" ]]
then
  pushd $TMPDIR
  curl 
https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-186.0.0-linux-x86_64.tar.gz
 --output gcloud.tar.gz
  tar xf gcloud.tar.gz
  ./google-cloud-sdk/install.sh --quiet
  . ./google-cloud-sdk/path.bash.inc
  popd
  gcloud components update --quiet || echo 'gcloud components update failed'
  gcloud -v
fi

# Build the container
TAG=$(date +%Y%m%d-%H%M%S)
date +%Y%m%d-%H%M%S
CONTAINER=us.gcr.io/$PROJECT/$USER/python
echo "Using container $CONTAINER"
Using container us.gcr.io/apache-beam-testing/jenkins/python
./gradlew :beam-sdks-python-container:docker 
-Pdocker-repository-root=us.gcr.io/$PROJECT/$USER -Pdocker-tag=$TAG
Parallel execution with configuration on demand is an incubating feature.

> Configure project :
Applying build_rules.gradle to beam
createPerformanceTestHarness with default configuration for project beam
Adding 47 .gitignore exclusions to Apache 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1525

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam23 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 86fdb74fd7cbc016065cfe6dea530155051a6f39 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 86fdb74fd7cbc016065cfe6dea530155051a6f39
Commit message: "Merge pull request #5269: Add an Unsupported 
StateRequestHandler"
 > git rev-list --no-walk 86fdb74fd7cbc016065cfe6dea530155051a6f39 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins4711700608973881502.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user mair...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user jasonkus...@google.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user ankurgoe...@gmail.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #278

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add an Unsupported StateRequestHandler

--
[...truncated 18.87 MB...]
May 04, 2018 2:43:08 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T02:43:00.022Z: Fusing unzipped copy of 
PAssert$3/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous), through 
flatten s18-u63, into producer 
PAssert$3/CreateActual/FilterActuals/Window.Assign
May 04, 2018 2:43:08 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T02:43:00.049Z: Fusing consumer 
PAssert$3/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous) into 
PAssert$3/CreateActual/FilterActuals/Window.Assign
May 04, 2018 2:43:08 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T02:43:00.077Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 04, 2018 2:43:08 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T02:43:00.107Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 04, 2018 2:43:08 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T02:43:00.144Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
May 04, 2018 2:43:08 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T02:43:00.199Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
May 04, 2018 2:43:08 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T02:43:00.240Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
May 04, 2018 2:43:08 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T02:43:00.279Z: Fusing consumer 
PAssert$3/CreateActual/RewindowActuals/Window.Assign into 
PAssert$3/CreateActual/Flatten.Iterables/FlattenIterables/FlatMap
May 04, 2018 2:43:08 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T02:43:00.315Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
May 04, 2018 2:43:08 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T02:43:00.355Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
May 04, 2018 2:43:08 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-04T02:43:00.392Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
May 04, 

[jira] [Work logged] (BEAM-4233) SQL generated code don't match between Maven and Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4233?focusedWorklogId=98103=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98103
 ]

ASF GitHub Bot logged work on BEAM-4233:


Author: ASF GitHub Bot
Created on: 04/May/18 02:43
Start Date: 04/May/18 02:43
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5275: [BEAM-4233] [SQL] 
Use the same javacc and fmpp as calcite
URL: https://github.com/apache/beam/pull/5275#issuecomment-386493053
 
 
   FYI this is green.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98103)
Time Spent: 40m  (was: 0.5h)

> SQL generated code don't match between Maven and Gradle 
> 
>
> Key: BEAM-4233
> URL: https://issues.apache.org/jira/browse/BEAM-4233
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The generated code between maven and gradle don't match. Particularly in the 
> SQL parser. This causes the behavior of our generated SQL parser to be 
> incorrect when built from gradle.
> The Maven plugin is using JavaCC version 4.0 while the Gradle plugin is using 
> version 6.1.2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4196) [SQL] Support Complex Types in DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4196?focusedWorklogId=98102=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98102
 ]

ASF GitHub Bot logged work on BEAM-4196:


Author: ASF GitHub Bot
Created on: 04/May/18 02:42
Start Date: 04/May/18 02:42
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5276: 
[BEAM-4196][SQL] Support complex types in DDL
URL: https://github.com/apache/beam/pull/5276#discussion_r185980780
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDdlNodes.java
 ##
 @@ -28,14 +27,6 @@
 public class SqlDdlNodes {
   private SqlDdlNodes() {}
 
-  /** Creates a CREATE TABLE. */
 
 Review comment:
   This annoys me. I don't like this extra indirection in calcite but I also 
don't like deviating from calcite unnecessaraly. Not much you can do about that.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98102)
Time Spent: 0.5h  (was: 20m)

> [SQL] Support Complex Types in DDL
> --
>
> Key: BEAM-4196
> URL: https://issues.apache.org/jira/browse/BEAM-4196
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Neither our DDL parser we copied from calcite-server or the calcite-server 
> don't support complex types in DDL. If we want to model something like JSON 
> objects we need to support at least Arrays and nested Rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #192

[Apache Jenkins Server]

See 


Changes:

[apilloud] [SQL] Add BeamEnumerableConverter

[mairbek] Templatize host name in SpannerIO

[coheigea] A few small fixes to catch a few regressions - put String literals

[sidhom] Move SyntheticNodes to top-level construction package and rename

[sidhom] Pull Flink execution environment utilities into static methods

[sidhom] Refactor Runner/SDK wire coder creation into a shared utility

[sidhom] [BEAM-3972] Translate portable batch pipelines by proto

[sidhom] FIXUP: Use ModelCoders to create proto coders and use WireCoders to

[sidhom] FIXUP: Use RunnerApi in WireCoders to prepare for coder instantiation

[tgroh] Add additional methods to QueryablePipeline

[tgroh] Expand the ExecutableGraph interface

[tgroh] Add ProtoOverrides

[jasonkuster] Add labels to Go SDK Dataflow Runner.

[jasonkuster] Add verbose error message if unmarshaling fails for label flag.

--
[...truncated 23.36 MB...]
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:286)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

May 04, 2018 2:23:40 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2018-05-04T02:23:37.726Z: java.lang.AbstractMethodError: 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.element(Lorg/apache/beam/sdk/transforms/DoFn;)Ljava/lang/Object;
at 
org.apache.beam.sdk.transforms.ParDoTest$TestDoFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:138)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:200)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:383)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:355)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:286)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

May 04, 2018 2:23:40 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2018-05-04T02:23:37.814Z: Workflow failed. Causes: 
S28:Create.Values/Read(CreateSource)+ParDo(Test)+PAssert$53/GroupGlobally/Window.Into()/Window.Assign+PAssert$53/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)+PAssert$53/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map+PAssert$53/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign+PAssert$53/GroupGlobally/GatherAllOutputs/GroupByKey/Reify+PAssert$53/GroupGlobally/GatherAllOutputs/GroupByKey/Write
 failed., A work item was attempted 4 times without success. Each time the 
worker eventually lost contact with the service. The work item was attempted 
on: 
  pardotest0testpardowithsi-05031920-jskb-harness-68k5,
  

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=98097=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98097
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 04/May/18 01:53
Start Date: 04/May/18 01:53
Worklog Time Spent: 10m 
  Work Description: tgroh closed pull request #5269: [BEAM-3326] Add an 
Unsupported StateRequestHandler
URL: https://github.com/apache/beam/pull/5269
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandler.java
 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandler.java
index cfa86db1b04..d0858935d73 100644
--- 
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandler.java
+++ 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandler.java
@@ -18,7 +18,8 @@
 package org.apache.beam.runners.fnexecution.state;
 
 import java.util.concurrent.CompletionStage;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateResponse;
 
 /**
  * Handler for {@link 
org.apache.beam.model.fnexecution.v1.BeamFnApi.StateRequest StateRequests}.
@@ -34,6 +35,12 @@
* Throwing an error during handling will complete the handler result 
{@link CompletionStage}
* exceptionally.
*/
-  CompletionStage 
handle(BeamFnApi.StateRequest request)
-  throws Exception;
+  CompletionStage handle(StateRequest request) throws 
Exception;
+
+  static StateRequestHandler unsupported() {
+return request -> {
+  throw new UnsupportedOperationException(
+  String.format("Cannot use an empty %s", 
StateRequestHandler.class.getSimpleName()));
+};
+  }
 }


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98097)
Time Spent: 8h  (was: 7h 50m)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 8h
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1524

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add an Unsupported StateRequestHandler

--
Started by GitHub push by tgroh
[EnvInject] - Loading node environment variables.
Building remotely on beam23 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 86fdb74fd7cbc016065cfe6dea530155051a6f39 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 86fdb74fd7cbc016065cfe6dea530155051a6f39
Commit message: "Merge pull request #5269: Add an Unsupported 
StateRequestHandler"
 > git rev-list --no-walk bf94e36f67a8bc5d24c795e40697ad2504c8594c # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins255961844494983.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user mair...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user jasonkus...@google.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user ankurgoe...@gmail.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com

[beam] branch master updated (bf94e36 -> 86fdb74)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from bf94e36  Merge pull request #5173: [BEAM-3773][SQL] Add 
EnumerableConverter for JDBC support
 add 1af8537  Add an Unsupported StateRequestHandler
 add 86fdb74  Merge pull request #5269: Add an Unsupported 
StateRequestHandler

No new revisions were added by this update.

Summary of changes:
 .../beam/runners/fnexecution/state/StateRequestHandler.java | 13 ++---
 1 file changed, 10 insertions(+), 3 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[jira] [Work logged] (BEAM-3515) Use portable ParDoPayload for SDF in DataflowRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3515?focusedWorklogId=98095=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98095
 ]

ASF GitHub Bot logged work on BEAM-3515:


Author: ASF GitHub Bot
Created on: 04/May/18 01:49
Start Date: 04/May/18 01:49
Worklog Time Spent: 10m 
  Work Description: jkff opened a new pull request #5277: [BEAM-3515] 
Portable translation of SplittableProcessKeyed
URL: https://github.com/apache/beam/pull/5277
 
 
   Introduces a new Payload for this primitive transform, consisting of the 
DoFn and restriction coder id.
   Expansion on the Dataflow backend side will proceed as usual. Next steps 
will be mostly in the worker, which will get a ParDo instruction now properly 
referencing the DoFn and coder in the portable pipeline proto.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98095)
Time Spent: 10m
Remaining Estimate: 0h

> Use portable ParDoPayload for SDF in DataflowRunner
> ---
>
> Key: BEAM-3515
> URL: https://issues.apache.org/jira/browse/BEAM-3515
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Eugene Kirpichov
>Priority: Major
>  Labels: portability
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The Java-specific blobs transmitted to Dataflow need more context, in the 
> form of portability framework protos.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #277

[Apache Jenkins Server]

See 


--
[...truncated 18.85 MB...]

org.apache.beam.examples.cookbook.JoinExamplesTest > testExtractCountryInfoFn 
STANDARD_ERROR
May 04, 2018 12:39:13 AM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.JoinExamplesTest > testExtractEventDataFn 
STANDARD_ERROR
May 04, 2018 12:39:13 AM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.DebuggingWordCountTest > testDebuggingWordCount 
STANDARD_ERROR
May 04, 2018 12:39:14 AM org.apache.beam.sdk.io.FileBasedSource 
getEstimatedSizeBytes
INFO: Filepattern 
/tmp/junit2802740750844789913/junit7670987741804310477.tmp matched 1 files with 
total size 54
May 04, 2018 12:39:14 AM org.apache.beam.sdk.io.FileBasedSource split
INFO: Splitting filepattern 
/tmp/junit2802740750844789913/junit7670987741804310477.tmp into bundles of size 
3 took 1 ms and produced 1 files and 18 bundles

org.apache.beam.examples.WordCountTest > testExtractWordsFn STANDARD_ERROR
May 04, 2018 12:39:14 AM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.subprocess.ExampleEchoPipelineTest > 
testExampleEchoPipeline STANDARD_ERROR
May 04, 2018 12:39:15 AM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-Echo4661759351989372341.sh 
May 04, 2018 12:39:15 AM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
May 04, 2018 12:39:15 AM 
org.apache.beam.examples.subprocess.utils.FileUtils copyFileFromGCSToWorker
INFO: Moving File /tmp/test-Echo4661759351989372341.sh to 
/tmp/test-Echoo1560119465224277674/test-Echo4661759351989372341.sh 
May 04, 2018 12:39:16 AM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-EchoAgain6246923428479559223.sh 
May 04, 2018 12:39:16 AM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
May 04, 2018 12:39:16 AM 
org.apache.beam.examples.subprocess.utils.FileUtils copyFileFromGCSToWorker
INFO: Moving File /tmp/test-EchoAgain6246923428479559223.sh to 
/tmp/test-Echoo1560119465224277674/test-EchoAgain6246923428479559223.sh 

org.apache.beam.examples.complete.game.HourlyTeamScoreTest > 
testUserScoresFilter STANDARD_OUT
GOT user18_BananaEmu,BananaEmu,1,144796569,2015-11-19 12:41:31.053
GOT user3_BananaEmu,BananaEmu,17,144796569,2015-11-19 12:41:31.053
GOT user7_AlmondWallaby,AlmondWallaby,15,144795563,2015-11-19 
09:53:53.444
GOT user18_BananaEmu,BananaEmu,7,144796569,2015-11-19 12:41:31.053
GOT user18_ApricotCaneToad,ApricotCaneToad,14,144796569,2015-11-19 
12:41:31.053
GOT user0_MagentaKangaroo,MagentaKangaroo,4,144796569,2015-11-19 
12:41:31.053
GOT 
user0_AndroidGreenEchidna,AndroidGreenEchidna,0,144796569,2015-11-19 
12:41:31.053
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,11,144795563,2015-11-19 
09:53:53.444
GOT user19_BisqueBilby,BisqueBilby,8,144795563,2015-11-19 09:53:53.444
GOT user13_ApricotQuokka,ApricotQuokka,15,144795563,2015-11-19 
09:53:53.444
GOT user19_BisqueBilby,BisqueBilby,6,144795563,2015-11-19 09:53:53.444
GOT user6_AmberNumbat,AmberNumbat,11,144795563,2015-11-19 09:53:53.444
GOT user0_MagentaKangaroo,MagentaKangaroo,3,144795563,2015-11-19 
09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,12,144795563,2015-11-19 
09:53:53.444
GOT user2_AmberCockatoo,AmberCockatoo,13,144796569,2015-11-19 
12:41:31.053

org.apache.beam.examples.complete.game.UserScoreTest > testTeamScoreSums 
STANDARD_OUT
GOT user13_ApricotQuokka,ApricotQuokka,15,144795563,2015-11-19 
09:53:53.444
GOT user7_AlmondWallaby,AlmondWallaby,15,144795563,2015-11-19 
09:53:53.444
GOT user6_AmberNumbat,AmberNumbat,11,144795563,2015-11-19 09:53:53.444
GOT user0_MagentaKangaroo,MagentaKangaroo,3,144795563,2015-11-19 
09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,11,144795563,2015-11-19 
09:53:53.444
GOT THIS IS A PARSE ERROR,2015-11-19 09:53:53.444

org.apache.beam.examples.complete.game.UserScoreTest > testTeamScoreSums 
STANDARD_ERROR
May 04, 2018 12:39:17 AM 
org.apache.beam.examples.complete.game.UserScore$ParseEventFn processElement
INFO: Parse error on THIS IS A 

[jira] [Work logged] (BEAM-4196) [SQL] Support Complex Types in DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4196?focusedWorklogId=98088=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98088
 ]

ASF GitHub Bot logged work on BEAM-4196:


Author: ASF GitHub Bot
Created on: 04/May/18 00:32
Start Date: 04/May/18 00:32
Worklog Time Spent: 10m 
  Work Description: akedin opened a new pull request #5276: 
[BEAM-4196][SQL] Support complex types in DDL
URL: https://github.com/apache/beam/pull/5276
 
 
   Support complex types in DDL
   
   Supported syntax:
   
   ```
   CREATE TABLE tableName (
 f_array1 INTEGER ARRAY,
 f_array2 ARRAY,
 f_array3 ARRAY,
 f_map MAP>,
 f_row ROW( f_int1 INTEGER, f_str1 VARCHAR))
   ```
   
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand:
  - [ ] What the pull request does
  - [ ] Why it does it
  - [ ] How it does it
  - [ ] Why this approach
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Run `./gradlew build` to make sure basic checks pass. A more thorough 
check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98088)
Time Spent: 10m
Remaining Estimate: 0h

> [SQL] Support Complex Types in DDL
> --
>
> Key: BEAM-4196
> URL: https://issues.apache.org/jira/browse/BEAM-4196
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Neither our DDL parser we copied from calcite-server or the calcite-server 
> don't support complex types in DDL. If we want to model something like JSON 
> objects we need to support at least Arrays and nested Rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4196) [SQL] Support Complex Types in DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4196?focusedWorklogId=98089=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98089
 ]

ASF GitHub Bot logged work on BEAM-4196:


Author: ASF GitHub Bot
Created on: 04/May/18 00:32
Start Date: 04/May/18 00:32
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5276: [BEAM-4196][SQL] 
Support complex types in DDL
URL: https://github.com/apache/beam/pull/5276#issuecomment-386476747
 
 
   R: @apilloud @kennknowles @XuMingmin @xumingming 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98089)
Time Spent: 20m  (was: 10m)

> [SQL] Support Complex Types in DDL
> --
>
> Key: BEAM-4196
> URL: https://issues.apache.org/jira/browse/BEAM-4196
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Neither our DDL parser we copied from calcite-server or the calcite-server 
> don't support complex types in DDL. If we want to model something like JSON 
> objects we need to support at least Arrays and nested Rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_TextIOIT_HDFS #132

[Apache Jenkins Server]

See 


Changes:

[apilloud] [SQL] Add BeamEnumerableConverter

[mairbek] Templatize host name in SpannerIO

[coheigea] A few small fixes to catch a few regressions - put String literals

[sidhom] Move SyntheticNodes to top-level construction package and rename

[sidhom] Pull Flink execution environment utilities into static methods

[sidhom] Refactor Runner/SDK wire coder creation into a shared utility

[sidhom] [BEAM-3972] Translate portable batch pipelines by proto

[sidhom] FIXUP: Use ModelCoders to create proto coders and use WireCoders to

[sidhom] FIXUP: Use RunnerApi in WireCoders to prepare for coder instantiation

[tgroh] Add additional methods to QueryablePipeline

[tgroh] Expand the ExecutableGraph interface

[tgroh] Add ProtoOverrides

[jasonkuster] Add labels to Go SDK Dataflow Runner.

[jasonkuster] Add verbose error message if unmarshaling fails for label flag.

--
[...truncated 404.28 KB...]
at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
at org.apache.hadoop.ipc.Client.call(Client.java:1451)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:923)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:138)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.AssignWindowsParDoFnFactory$AssignWindowsParDoFn.processElement(AssignWindowsParDoFnFactory.java:118)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT_HDFS #126

[Apache Jenkins Server]

See 


Changes:

[apilloud] [SQL] Add BeamEnumerableConverter

[mairbek] Templatize host name in SpannerIO

[coheigea] A few small fixes to catch a few regressions - put String literals

[sidhom] Move SyntheticNodes to top-level construction package and rename

[sidhom] Pull Flink execution environment utilities into static methods

[sidhom] Refactor Runner/SDK wire coder creation into a shared utility

[sidhom] [BEAM-3972] Translate portable batch pipelines by proto

[sidhom] FIXUP: Use ModelCoders to create proto coders and use WireCoders to

[sidhom] FIXUP: Use RunnerApi in WireCoders to prepare for coder instantiation

[tgroh] Add additional methods to QueryablePipeline

[tgroh] Expand the ExecutableGraph interface

[tgroh] Add ProtoOverrides

[jasonkuster] Add labels to Go SDK Dataflow Runner.

[jasonkuster] Add verbose error message if unmarshaling fails for label flag.

--
[...truncated 828.96 KB...]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.net.ConnectException: Call From 
textioit0writethenreadall-05031704-b4dr-harness-zdtr.c.apache-beam-testing.internal/10.128.0.14
 to 9.55.224.35.bc.googleusercontent.com:9000 failed on connection exception: 
java.net.ConnectException: Connection refused; For more details see:  
http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:923)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)

[jira] [Assigned] (BEAM-4201) Integration Tests for PubsubIO

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin reassigned BEAM-4201:
-

Assignee: Anton Kedin

> Integration Tests for PubsubIO
> --
>
> Key: BEAM-4201
> URL: https://issues.apache.org/jira/browse/BEAM-4201
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>
> Add integration tests for PubsubIO



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4196) [SQL] Support Complex Types in DDL

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin reassigned BEAM-4196:
-

Assignee: Anton Kedin

> [SQL] Support Complex Types in DDL
> --
>
> Key: BEAM-4196
> URL: https://issues.apache.org/jira/browse/BEAM-4196
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>
> Neither our DDL parser we copied from calcite-server or the calcite-server 
> don't support complex types in DDL. If we want to model something like JSON 
> objects we need to support at least Arrays and nested Rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #127

[Apache Jenkins Server]

See 


Changes:

[apilloud] [SQL] Add BeamEnumerableConverter

[mairbek] Templatize host name in SpannerIO

[coheigea] A few small fixes to catch a few regressions - put String literals

[sidhom] Move SyntheticNodes to top-level construction package and rename

[sidhom] Pull Flink execution environment utilities into static methods

[sidhom] Refactor Runner/SDK wire coder creation into a shared utility

[sidhom] [BEAM-3972] Translate portable batch pipelines by proto

[sidhom] FIXUP: Use ModelCoders to create proto coders and use WireCoders to

[sidhom] FIXUP: Use RunnerApi in WireCoders to prepare for coder instantiation

[tgroh] Add additional methods to QueryablePipeline

[tgroh] Expand the ExecutableGraph interface

[tgroh] Add ProtoOverrides

[jasonkuster] Add labels to Go SDK Dataflow Runner.

[jasonkuster] Add verbose error message if unmarshaling fails for label flag.

--
[...truncated 493.68 KB...]
at 
com.mongodb.connection.WriteCommandProtocol.receiveMessage(WriteCommandProtocol.java:234)
at 
com.mongodb.connection.WriteCommandProtocol.execute(WriteCommandProtocol.java:104)
at 
com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:67)
at 
com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:37)
at 
com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:159)
at 
com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:286)
at 
com.mongodb.connection.DefaultServerConnection.insertCommand(DefaultServerConnection.java:115)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run$2.executeWriteCommandProtocol(MixedBulkWriteOperation.java:455)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run$RunExecutor.execute(MixedBulkWriteOperation.java:646)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run.execute(MixedBulkWriteOperation.java:401)
at 
com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:179)
at 
com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:230)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:221)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
at 
org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:129)
at 
org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 

Build failed in Jenkins: beam_PerformanceTests_JDBC #529

[Apache Jenkins Server]

See 


Changes:

[apilloud] [SQL] Add BeamEnumerableConverter

[mairbek] Templatize host name in SpannerIO

[coheigea] A few small fixes to catch a few regressions - put String literals

[sidhom] Move SyntheticNodes to top-level construction package and rename

[sidhom] Pull Flink execution environment utilities into static methods

[sidhom] Refactor Runner/SDK wire coder creation into a shared utility

[sidhom] [BEAM-3972] Translate portable batch pipelines by proto

[sidhom] FIXUP: Use ModelCoders to create proto coders and use WireCoders to

[sidhom] FIXUP: Use RunnerApi in WireCoders to prepare for coder instantiation

[tgroh] Add additional methods to QueryablePipeline

[tgroh] Expand the ExecutableGraph interface

[tgroh] Add ProtoOverrides

[jasonkuster] Add labels to Go SDK Dataflow Runner.

[jasonkuster] Add verbose error message if unmarshaling fails for label flag.

--
[...truncated 90.78 KB...]
:beam-runners-google-cloud-dataflow-java:classes (Thread[Task worker for 
':',5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for 
':',5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:shadowJar UP-TO-DATE
Build cache key for task ':beam-runners-google-cloud-dataflow-java:shadowJar' 
is e172a38a138d7a2494e8a47a4f5dbf9b
Caching disabled for task ':beam-runners-google-cloud-dataflow-java:shadowJar': 
Caching has not been enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for 
':',5,main]) completed. Took 0.119 secs.

> Task :beam-sdks-java-io-google-cloud-platform:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:compileTestJava' is 
a46f3372face3378893e696fddb446c5
Skipping task ':beam-sdks-java-io-google-cloud-platform:compileTestJava' as it 
is up-to-date.
:beam-sdks-java-io-google-cloud-platform:compileTestJava (Thread[Task worker 
for ':' Thread 5,5,main]) completed. Took 0.158 secs.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 5,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:testClasses UP-TO-DATE
Skipping task ':beam-sdks-java-io-google-cloud-platform:testClasses' as it has 
no actions.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 5,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 5,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar' is 
2b531ae2f6d54d6d71d051bffe451dff
Caching disabled for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-sdks-java-io-google-cloud-platform:shadowTestJar' as it is 
up-to-date.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 5,5,main]) completed. Took 0.025 secs.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 5,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:compileTestJava' is 
47e8e2e4ae67b3f2224dfd0ac01741f0
Skipping task ':beam-runners-google-cloud-dataflow-java:compileTestJava' as it 
is up-to-date.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 5,5,main]) completed. Took 0.05 secs.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 5,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:testClasses UP-TO-DATE
Skipping task ':beam-runners-google-cloud-dataflow-java:testClasses' as it has 
no actions.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 5,5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 5,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar' is 
e74237c8c85a3212b491a083cd4f270a
Caching disabled for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowTestJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 5,5,main]) completed. Took 0.029 secs.
:beam-sdks-java-io-jdbc:compileTestJava (Thread[Task worker for ':' Thread 
5,5,main]) started.

> Task :beam-sdks-java-io-jdbc:compileTestJava 

Build failed in Jenkins: beam_PerformanceTests_Python #1225

[Apache Jenkins Server]

See 


Changes:

[apilloud] [SQL] Add BeamEnumerableConverter

[mairbek] Templatize host name in SpannerIO

[coheigea] A few small fixes to catch a few regressions - put String literals

[sidhom] Move SyntheticNodes to top-level construction package and rename

[sidhom] Pull Flink execution environment utilities into static methods

[sidhom] Refactor Runner/SDK wire coder creation into a shared utility

[sidhom] [BEAM-3972] Translate portable batch pipelines by proto

[sidhom] FIXUP: Use ModelCoders to create proto coders and use WireCoders to

[sidhom] FIXUP: Use RunnerApi in WireCoders to prepare for coder instantiation

[tgroh] Add additional methods to QueryablePipeline

[tgroh] Expand the ExecutableGraph interface

[tgroh] Add ProtoOverrides

[jasonkuster] Add labels to Go SDK Dataflow Runner.

[jasonkuster] Add verbose error message if unmarshaling fails for label flag.

--
[...truncated 4.55 KB...]
Collecting pywinrm (from -r PerfKitBenchmarker/requirements.txt (line 25))
  Using cached 
https://files.pythonhosted.org/packages/0d/12/13a3117bbd2230043aa32dcfa2198c33269665eaa1a8fa26174ce49b338f/pywinrm-0.3.0-py2.py3-none-any.whl
Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages 
(from absl-py->-r PerfKitBenchmarker/requirements.txt (line 14)) (1.11.0)
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15)) (1.0)
Collecting colorama; extra == "windows" (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/db/c8/7dcf9dbcb22429512708fe3a547f8b6101c0d02137acbd892505aee57adf/colorama-0.3.9-py2.py3-none-any.whl
Collecting requests-ntlm>=0.3.0 (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
  Using cached 
https://files.pythonhosted.org/packages/03/4b/8b9a1afde8072c4d5710d9fa91433d504325821b038e00237dc8d6d833dc/requests_ntlm-1.1.0-py2.py3-none-any.whl
Requirement already satisfied: requests>=2.9.1 in 
/usr/local/lib/python2.7/dist-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.18.4)
Collecting xmltodict (from pywinrm->-r PerfKitBenchmarker/requirements.txt 
(line 25))
  Using cached 
https://files.pythonhosted.org/packages/42/a9/7e99652c6bc619d19d58cdd8c47560730eb5825d43a7e25db2e1d776ceb7/xmltodict-0.11.0-py2.py3-none-any.whl
Requirement already satisfied: cryptography>=1.3 in 
/usr/local/lib/python2.7/dist-packages (from requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.2.2)
Collecting ntlm-auth>=1.0.2 (from requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
  Using cached 
https://files.pythonhosted.org/packages/69/bc/230987c0dc22c763529330b2e669dbdba374d6a10c1f61232274184731be/ntlm_auth-1.1.0-py2.py3-none-any.whl
Requirement already satisfied: certifi>=2017.4.17 in 
/usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2018.4.16)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in 
/usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (3.0.4)
Requirement already satisfied: idna<2.7,>=2.5 in 
/usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.6)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in 
/usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (1.22)
Requirement already satisfied: cffi>=1.7; platform_python_implementation != 
"PyPy" in /usr/local/lib/python2.7/dist-packages (from 
cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (1.11.5)
Requirement already satisfied: enum34; python_version < "3" in 
/usr/local/lib/python2.7/dist-packages (from 
cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (1.1.6)
Requirement already satisfied: asn1crypto>=0.21.0 in 
/usr/local/lib/python2.7/dist-packages (from 
cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (0.24.0)
Requirement already satisfied: ipaddress; python_version < "3" in 
/usr/local/lib/python2.7/dist-packages (from 
cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (1.0.22)
Requirement already satisfied: pycparser in 
/usr/local/lib/python2.7/dist-packages (from cffi>=1.7; 
platform_python_implementation != 
"PyPy"->cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.18)
Installing collected packages: absl-py, colorama, colorlog, blinker, futures, 
pint, 

[jira] [Work logged] (BEAM-4162) Wire up PubsubIO+JSON to Beam SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4162?focusedWorklogId=98079=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98079
 ]

ASF GitHub Bot logged work on BEAM-4162:


Author: ASF GitHub Bot
Created on: 03/May/18 23:42
Start Date: 03/May/18 23:42
Worklog Time Spent: 10m 
  Work Description: rangadi commented on a change in pull request #5253: 
[BEAM-4162][SQL] Wire up PubsubIO to SQL
URL: https://github.com/apache/beam/pull/5253#discussion_r185964760
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubIOJsonTable.java
 ##
 @@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.pubsub;
+
+import static com.google.api.client.util.DateTime.parseRfc3339;
+import static org.apache.beam.sdk.schemas.Schema.TypeName.DATETIME;
+import static org.apache.beam.sdk.util.JsonToRowUtils.newObjectMapperWith;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.impl.schema.BeamIOType;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubIO;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubMessage;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.util.RowJsonDeserializer;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.Row;
+import org.joda.time.DateTime;
+
+/**
+ * Experimental
+ *
+ * Wraps the {@link PubsubIO} with JSON messages into {@link BeamSqlTable}.
+ *
+ * This enables {@link PubsubIO} registration in Beam SQL environment as a 
table, including DDL
+ * support.
+ *
+ * For example:
+ * 
+ *
+ *  CREATE TABLE topic (name VARCHAR, age INTEGER)
+ * TYPE 'pubsub'
+ * LOCATION projects/GCP project id/topics/topic name
+ * TBLPROPERTIES '{ \"timestampAttributeKey\" : timestamp 
attribute }';
+ *
+ *   SELECT name, age FROM topic;
+ *
+ * 
+ */
+@AutoValue
+@Internal
+@Experimental
+abstract class PubsubIOJsonTable implements BeamSqlTable, Serializable {
+
+  /**
+   * Schema of the pubsubs message payload.
+   *
+   * Only UTF-8 flat JSON objects are supported at the moment.
+   */
+  abstract Schema getPayloadSchema();
+
+  /**
+   * Attribute key of the Pubsub message from which to extract the event 
timestamp.
+   *
+   * This attribute has to conform to the same requirements as in {@link
+   * PubsubIO.Read.Builder#withTimestampAttribute}.
+   *
+   * Short version: it has to be either millis since epoch or string in RFC 
3339 format.
+   */
+  abstract String getTimestampAttribute();
+
+  /**
+   * Pubsub topic name.
+   *
+   * Topic is the only way to specify the Pubsub source. Explicitly 
specifying the subscription
+   * is not supported at the moment. Subscriptions are automatically created 
an managed.
+   */
+  abstract String getTopic();
+
+  static Builder builder() {
+return new AutoValue_PubsubIOJsonTable.Builder();
+  }
+
+  /**
+   * Table schema.
+   *
+   * Inherited from {@link BeamSqlTable}. Different from {@link 
#getPayloadSchema()},
+   * includes timestamp attribute.
+   */
+   public abstract Schema getSchema();
+
+  @Override
+  public BeamIOType getSourceType() {
+return BeamIOType.UNBOUNDED;
+  }
+
+  @Override
+  public PCollection buildIOReader(Pipeline pipeline) {
+return
+PBegin
+.in(pipeline)
+.apply("readFromPubsub", readMessagesWithAttributes())
+

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #276

[Apache Jenkins Server]

See 


Changes:

[apilloud] [SQL] Add BeamEnumerableConverter

--
[...truncated 18.46 MB...]
May 03, 2018 11:36:44 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:36:40.812Z: Fusing consumer 
PAssert$3/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous) into 
PAssert$3/CreateActual/FilterActuals/Window.Assign
May 03, 2018 11:36:44 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:36:40.847Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 03, 2018 11:36:44 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:36:40.878Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 03, 2018 11:36:44 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:36:40.907Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
May 03, 2018 11:36:44 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:36:40.938Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
May 03, 2018 11:36:44 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:36:40.978Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
May 03, 2018 11:36:44 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:36:41.010Z: Fusing consumer 
PAssert$3/CreateActual/RewindowActuals/Window.Assign into 
PAssert$3/CreateActual/Flatten.Iterables/FlattenIterables/FlatMap
May 03, 2018 11:36:44 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:36:41.045Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
May 03, 2018 11:36:44 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:36:41.074Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
May 03, 2018 11:36:44 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:36:41.108Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
May 03, 2018 11:36:44 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:36:41.137Z: Fusing consumer 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #275

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add additional methods to QueryablePipeline

[tgroh] Expand the ExecutableGraph interface

--
[...truncated 19.11 MB...]
May 03, 2018 11:09:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:09:48.737Z: Fusing unzipped copy of 
PAssert$3/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous), through 
flatten s18-u63, into producer 
PAssert$3/CreateActual/FilterActuals/Window.Assign
May 03, 2018 11:09:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:09:48.791Z: Fusing consumer 
PAssert$3/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous) into 
PAssert$3/CreateActual/FilterActuals/Window.Assign
May 03, 2018 11:09:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:09:48.840Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 03, 2018 11:09:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:09:48.886Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 03, 2018 11:09:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:09:48.928Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
May 03, 2018 11:09:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:09:48.966Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
May 03, 2018 11:09:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:09:49.008Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
May 03, 2018 11:09:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:09:49.059Z: Fusing consumer 
PAssert$3/CreateActual/RewindowActuals/Window.Assign into 
PAssert$3/CreateActual/Flatten.Iterables/FlattenIterables/FlatMap
May 03, 2018 11:09:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:09:49.110Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
May 03, 2018 11:09:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:09:49.148Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
May 03, 2018 11:09:59 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T23:09:49.194Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
 into 

[jira] [Work logged] (BEAM-4231) Runner utility for Coder instantiation

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4231?focusedWorklogId=98074=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98074
 ]

ASF GitHub Bot logged work on BEAM-4231:


Author: ASF GitHub Bot
Created on: 03/May/18 23:13
Start Date: 03/May/18 23:13
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5274: 
[BEAM-4231] Runner utility for coder instantiation
URL: https://github.com/apache/beam/pull/5274#discussion_r185960682
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/wire/WireCoders.java
 ##
 @@ -17,29 +17,65 @@
  */
 package org.apache.beam.runners.fnexecution.wire;
 
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.io.IOException;
 import java.util.function.Predicate;
 import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.runners.core.construction.CoderTranslation;
 import org.apache.beam.runners.core.construction.ModelCoders;
+import org.apache.beam.runners.core.construction.RehydratedComponents;
 import org.apache.beam.runners.core.construction.SyntheticComponents;
 import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
 
 /** Helpers to construct coders for gRPC port reads and writes. */
 public class WireCoders {
-  /** Creates an SDK-side wire coder for a port read/write for the given 
PCollection. */
+  /**
+   * Creates an SDK-side wire coder for a port read/write for the given 
PCollection. Returns a
 
 Review comment:
   "The returned coder will be a windowed value coder containing..."


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98074)
Time Spent: 0.5h  (was: 20m)

> Runner utility for Coder instantiation
> --
>
> Key: BEAM-4231
> URL: https://issues.apache.org/jira/browse/BEAM-4231
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Portable runners need to instantiate coders for communicating with SDK 
> harnesses in a consistent way. They cannot simply instantiate coders as 
> defined by PCollections because some component coders may only be known to 
> specific SDKs. Unknown coders should be length-prefixed; the underlying 
> elements should only be exposed to runners as byte strings.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4233) SQL generated code don't match between Maven and Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4233?focusedWorklogId=98073=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98073
 ]

ASF GitHub Bot logged work on BEAM-4233:


Author: ASF GitHub Bot
Created on: 03/May/18 23:11
Start Date: 03/May/18 23:11
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5275: [BEAM-4233] [SQL] 
Use the same javacc and fmpp as calcite
URL: https://github.com/apache/beam/pull/5275#issuecomment-386463988
 
 
   Seems wise. I'll merge when green.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98073)
Time Spent: 0.5h  (was: 20m)

> SQL generated code don't match between Maven and Gradle 
> 
>
> Key: BEAM-4233
> URL: https://issues.apache.org/jira/browse/BEAM-4233
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The generated code between maven and gradle don't match. Particularly in the 
> SQL parser. This causes the behavior of our generated SQL parser to be 
> incorrect when built from gradle.
> The Maven plugin is using JavaCC version 4.0 while the Gradle plugin is using 
> version 6.1.2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4233) SQL generated code don't match between Maven and Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4233?focusedWorklogId=98071=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98071
 ]

ASF GitHub Bot logged work on BEAM-4233:


Author: ASF GitHub Bot
Created on: 03/May/18 23:07
Start Date: 03/May/18 23:07
Worklog Time Spent: 10m 
  Work Description: apilloud opened a new pull request #5275: [BEAM-4233] 
[SQL] Use the same javacc and fmpp as calcite
URL: https://github.com/apache/beam/pull/5275
 
 
   Calcite uses very specific versions of javacc and fmpp. We should use those 
so we don't hit bugs caused by incompatibilities.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98071)
Time Spent: 10m
Remaining Estimate: 0h

> SQL generated code don't match between Maven and Gradle 
> 
>
> Key: BEAM-4233
> URL: https://issues.apache.org/jira/browse/BEAM-4233
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The generated code between maven and gradle don't match. Particularly in the 
> SQL parser. This causes the behavior of our generated SQL parser to be 
> incorrect when built from gradle.
> The Maven plugin is using JavaCC version 4.0 while the Gradle plugin is using 
> version 6.1.2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4233) SQL generated code don't match between Maven and Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4233?focusedWorklogId=98072=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98072
 ]

ASF GitHub Bot logged work on BEAM-4233:


Author: ASF GitHub Bot
Created on: 03/May/18 23:07
Start Date: 03/May/18 23:07
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5275: [BEAM-4233] [SQL] 
Use the same javacc and fmpp as calcite
URL: https://github.com/apache/beam/pull/5275#issuecomment-386463145
 
 
   R: @kennknowles 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98072)
Time Spent: 20m  (was: 10m)

> SQL generated code don't match between Maven and Gradle 
> 
>
> Key: BEAM-4233
> URL: https://issues.apache.org/jira/browse/BEAM-4233
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The generated code between maven and gradle don't match. Particularly in the 
> SQL parser. This causes the behavior of our generated SQL parser to be 
> incorrect when built from gradle.
> The Maven plugin is using JavaCC version 4.0 while the Gradle plugin is using 
> version 6.1.2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4233) SQL generated code don't match between Maven and Gradle

[Andrew Pilloud (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463176#comment-16463176
 ] 

Andrew Pilloud commented on BEAM-4233:
--

This appears to be related to JavaCC version. The BeamSqlDslAggregationTest. 
testUnsupportedDistinct test found this. I get the following with JavaCC 4.0:
{code:java}
org.apache.calcite.sql.parser.SqlParseException: Encountered "*" at line 1, 
column 31.{code}
And with 6.1.2:
{code:java}
org.apache.calcite.sql.parser.SqlParseException: Encountered "" at line 1, 
column 31.{code}

> SQL generated code don't match between Maven and Gradle 
> 
>
> Key: BEAM-4233
> URL: https://issues.apache.org/jira/browse/BEAM-4233
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>
> The generated code between maven and gradle don't match. Particularly in the 
> SQL parser. This causes the behavior of our generated SQL parser to be 
> incorrect when built from gradle.
> The Maven plugin is using JavaCC version 4.0 while the Gradle plugin is using 
> version 6.1.2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #274

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add ProtoOverrides

--
[...truncated 19.44 MB...]
May 03, 2018 10:49:14 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:49:04.368Z: Fusing consumer 
PAssert$3/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous) into 
PAssert$3/CreateActual/FilterActuals/Window.Assign
May 03, 2018 10:49:14 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:49:04.414Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 03, 2018 10:49:14 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:49:04.460Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 03, 2018 10:49:14 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:49:04.497Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
May 03, 2018 10:49:14 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:49:04.546Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
May 03, 2018 10:49:14 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:49:04.600Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
May 03, 2018 10:49:14 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:49:04.643Z: Fusing consumer 
PAssert$3/CreateActual/RewindowActuals/Window.Assign into 
PAssert$3/CreateActual/Flatten.Iterables/FlattenIterables/FlatMap
May 03, 2018 10:49:14 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:49:04.712Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
May 03, 2018 10:49:14 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:49:04.749Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
May 03, 2018 10:49:14 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:49:04.792Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
May 03, 2018 10:49:14 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:49:04.829Z: Fusing consumer 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #273

[Apache Jenkins Server]

See 


Changes:

[jasonkuster] Add labels to Go SDK Dataflow Runner.

[jasonkuster] Add verbose error message if unmarshaling fails for label flag.

--
[...truncated 19.10 MB...]
INFO: 2018-05-03T22:46:14.188Z: Executing operation 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)

org.apache.beam.sdk.io.gcp.datastore.V1WriteIT > testE2EV1Write STANDARD_ERROR
May 03, 2018 10:46:19 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:46:18.884Z: Cleaning up.
May 03, 2018 10:46:19 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:46:19.001Z: Stopping worker pool...

org.apache.beam.sdk.io.gcp.datastore.V1ReadIT > 
testE2EV1ReadWithGQLQueryWithNoLimit STANDARD_ERROR
May 03, 2018 10:46:21 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:46:21.626Z: Executing operation 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/CreateDataflowView
May 03, 2018 10:46:24 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:46:21.802Z: Executing operation 
PAssert$3/Create.Values/Read(CreateSource)+PAssert$3/WindowToken/Window.Assign+PAssert$3/RunChecks+PAssert$3/VerifyAssertions/ParDo(DefaultConclude)
May 03, 2018 10:46:24 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:46:23.766Z: Cleaning up.
May 03, 2018 10:46:24 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:46:24.025Z: Stopping worker pool...
May 03, 2018 10:47:33 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:47:32.105Z: Autoscaling: Resized worker pool from 1 to 
0.
May 03, 2018 10:47:33 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:47:32.140Z: Autoscaling: Would further reduce the 
number of workers but reached the minimum number allowed for the job.
May 03, 2018 10:47:33 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:47:32.180Z: Worker pool stopped.

org.apache.beam.sdk.io.gcp.datastore.V1WriteIT > testE2EV1Write STANDARD_ERROR
May 03, 2018 10:47:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:47:34.030Z: Autoscaling: Resized worker pool from 1 to 
0.
May 03, 2018 10:47:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:47:34.056Z: Autoscaling: Would further reduce the 
number of workers but reached the minimum number allowed for the job.
May 03, 2018 10:47:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T22:47:34.094Z: Worker pool stopped.

org.apache.beam.sdk.io.gcp.datastore.V1ReadIT > 
testE2EV1ReadWithGQLQueryWithNoLimit STANDARD_ERROR
May 03, 2018 10:47:41 PM 
org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
INFO: Job 2018-05-03_15_44_29-5645445011856715246 finished with status DONE.
May 03, 2018 10:47:41 PM 
org.apache.beam.runners.dataflow.TestDataflowRunner checkForPAssertSuccess
INFO: Success result for Dataflow job 
2018-05-03_15_44_29-5645445011856715246. Found 1 success, 0 failures out of 1 
expected assertions.
May 03, 2018 10:47:41 PM 
org.apache.beam.sdk.io.gcp.datastore.V1TestUtil$V1TestWriter flushBatch
INFO: Writing batch of 500 entities
May 03, 2018 10:47:41 PM 
org.apache.beam.sdk.io.gcp.datastore.V1TestUtil$V1TestWriter flushBatch
INFO: Successfully wrote 500 entities
May 03, 2018 10:47:42 PM 
org.apache.beam.sdk.io.gcp.datastore.V1TestUtil$V1TestWriter flushBatch
INFO: Writing batch of 500 entities
May 03, 2018 10:47:42 PM 
org.apache.beam.sdk.io.gcp.datastore.V1TestUtil$V1TestWriter flushBatch
INFO: Successfully wrote 500 entities
May 03, 2018 10:47:42 PM org.apache.beam.sdk.io.gcp.datastore.V1TestUtil 
deleteAllEntities
INFO: Successfully deleted 1000 entities

org.apache.beam.sdk.io.gcp.datastore.SplitQueryFnIT > 
testSplitQueryFnWithLargeDataset STANDARD_ERROR
May 03, 2018 10:47:42 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.sdk.io.gcp.datastore.V1WriteIT > testE2EV1Write STANDARD_ERROR
May 03, 2018 10:47:42 PM 

[jira] [Updated] (BEAM-4233) SQL generated code don't match between Maven and Gradle

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-4233:
--
Description: 
The generated code between maven and gradle don't match. Particularly in the 
SQL parser. This causes the behavior of our generated SQL parser to be 
incorrect when built from gradle.

The Maven plugin is using JavaCC version 4.0 while the Gradle plugin is using 
version 6.1.2.

  was:The generated code between maven and gradle don't match. Particularly in 
the SQL parser. This causes the behavior of our generated SQL parser to be 
incorrect when built from gradle.


> SQL generated code don't match between Maven and Gradle 
> 
>
> Key: BEAM-4233
> URL: https://issues.apache.org/jira/browse/BEAM-4233
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>
> The generated code between maven and gradle don't match. Particularly in the 
> SQL parser. This causes the behavior of our generated SQL parser to be 
> incorrect when built from gradle.
> The Maven plugin is using JavaCC version 4.0 while the Gradle plugin is using 
> version 6.1.2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1523

[Apache Jenkins Server]

See 


Changes:

[apilloud] [SQL] Add BeamEnumerableConverter

--
Started by GitHub push by kennknowles
[EnvInject] - Loading node environment variables.
Building remotely on beam23 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision bf94e36f67a8bc5d24c795e40697ad2504c8594c (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f bf94e36f67a8bc5d24c795e40697ad2504c8594c
Commit message: "Merge pull request #5173: [BEAM-3773][SQL] Add 
EnumerableConverter for JDBC support"
 > git rev-list --no-walk 4971e7ec48ad85d002a1c7596cb436dc8c8e88ea # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins1870420410302368017.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user mair...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user jasonkus...@google.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user ankurgoe...@gmail.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com

[jira] [Updated] (BEAM-4233) SQL generated code don't match between Maven and Gradle

[Andrew Pilloud (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Pilloud updated BEAM-4233:
-
Issue Type: Bug  (was: New Feature)

> SQL generated code don't match between Maven and Gradle 
> 
>
> Key: BEAM-4233
> URL: https://issues.apache.org/jira/browse/BEAM-4233
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>
> The generated code between maven and gradle don't match. Particularly in the 
> SQL parser. This causes the behavior of our generated SQL parser to be 
> incorrect when built from gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3773) [SQL] Investigate JDBC interface for Beam SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3773?focusedWorklogId=98066=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98066
 ]

ASF GitHub Bot logged work on BEAM-3773:


Author: ASF GitHub Bot
Created on: 03/May/18 22:44
Start Date: 03/May/18 22:44
Worklog Time Spent: 10m 
  Work Description: kennknowles closed pull request #5173: [BEAM-3773][SQL] 
Add EnumerableConverter for JDBC support
URL: https://github.com/apache/beam/pull/5173
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java
index af74dae7510..8423ddaf153 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java
@@ -20,6 +20,7 @@
 import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
 import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
 import org.apache.beam.sdk.extensions.sql.impl.rule.BeamAggregationRule;
+import 
org.apache.beam.sdk.extensions.sql.impl.rule.BeamEnumerableConverterRule;
 import org.apache.beam.sdk.extensions.sql.impl.rule.BeamFilterRule;
 import org.apache.beam.sdk.extensions.sql.impl.rule.BeamIntersectRule;
 import org.apache.beam.sdk.extensions.sql.impl.rule.BeamJoinRule;
@@ -49,7 +50,8 @@
   BeamIntersectRule.INSTANCE,
   BeamMinusRule.INSTANCE,
   BeamUnionRule.INSTANCE,
-  BeamJoinRule.INSTANCE)
+  BeamJoinRule.INSTANCE,
+  BeamEnumerableConverterRule.INSTANCE)
 };
   }
 }
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverter.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverter.java
new file mode 100644
index 000..0ccc6c893a9
--- /dev/null
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverter.java
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rel;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.MetricNameFilter;
+import org.apache.beam.sdk.metrics.MetricQueryResults;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.metrics.MetricsFilter;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.Row;
+import org.apache.calcite.adapter.enumerable.EnumerableRel;
+import org.apache.calcite.adapter.enumerable.EnumerableRelImplementor;
+import org.apache.calcite.adapter.enumerable.PhysType;
+import org.apache.calcite.adapter.enumerable.PhysTypeImpl;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Linq4j;
+import org.apache.calcite.linq4j.tree.BlockBuilder;
+import org.apache.calcite.linq4j.tree.Expression;
+import org.apache.calcite.linq4j.tree.Expressions;
+import org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptCost;
+import 

[jira] [Created] (BEAM-4233) SQL generated code don't match between Maven and Gradle

[Andrew Pilloud (JIRA)]

Andrew Pilloud created BEAM-4233:


 Summary: SQL generated code don't match between Maven and Gradle 
 Key: BEAM-4233
 URL: https://issues.apache.org/jira/browse/BEAM-4233
 Project: Beam
  Issue Type: New Feature
  Components: dsl-sql
Reporter: Andrew Pilloud
Assignee: Andrew Pilloud


The generated code between maven and gradle don't match. Particularly in the 
SQL parser. This causes the behavior of our generated SQL parser to be 
incorrect when built from gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=98065=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98065
 ]

ASF GitHub Bot logged work on BEAM-4059:


Author: ASF GitHub Bot
Created on: 03/May/18 22:43
Start Date: 03/May/18 22:43
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5193: [BEAM-4059] 
Reduce number of ValidatesRunner tests and reorganize them for better 
parallelization
URL: https://github.com/apache/beam/pull/5193#issuecomment-386458894
 
 
   I think someone told me that the failure which you encountered is on 
`master` and fixed. Can you `git rebase -i` to resolve the fixups anyhow?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98065)
Time Spent: 5.5h  (was: 5h 20m)

> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Pablo Estrada
>Assignee: Scott Wegner
>Priority: Major
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>
> See 
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=98062=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98062
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 03/May/18 22:30
Start Date: 03/May/18 22:30
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5269: 
[BEAM-3326] Add an Unsupported StateRequestHandler
URL: https://github.com/apache/beam/pull/5269#discussion_r185953920
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandler.java
 ##
 @@ -34,6 +35,12 @@
* Throwing an error during handling will complete the handler result 
{@link CompletionStage}
* exceptionally.
*/
-  CompletionStage 
handle(BeamFnApi.StateRequest request)
-  throws Exception;
+  CompletionStage handle(StateRequest request) throws 
Exception;
+
+  static StateRequestHandler unsupported() {
 
 Review comment:
   For some reason I thought it was a class.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98062)
Time Spent: 7h 50m  (was: 7h 40m)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 7h 50m
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4073) The DirectRunner should interact with a Pipeline via an abstraction of the Graph rather than SDK types

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4073?focusedWorklogId=98061=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98061
 ]

ASF GitHub Bot logged work on BEAM-4073:


Author: ASF GitHub Bot
Created on: 03/May/18 22:28
Start Date: 03/May/18 22:28
Worklog Time Spent: 10m 
  Work Description: tgroh commented on issue #5270: [BEAM-4073] Migrate 
direct runner portability for nodes
URL: https://github.com/apache/beam/pull/5270#issuecomment-386456060
 
 
   R: @youngoli 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98061)
Time Spent: 1h 20m  (was: 1h 10m)

> The DirectRunner should interact with a Pipeline via an abstraction of the 
> Graph rather than SDK types
> --
>
> Key: BEAM-4073
> URL: https://issues.apache.org/jira/browse/BEAM-4073
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4073) The DirectRunner should interact with a Pipeline via an abstraction of the Graph rather than SDK types

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4073?focusedWorklogId=98060=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98060
 ]

ASF GitHub Bot logged work on BEAM-4073:


Author: ASF GitHub Bot
Created on: 03/May/18 22:20
Start Date: 03/May/18 22:20
Worklog Time Spent: 10m 
  Work Description: tgroh closed pull request #5228: [BEAM-4073] Expand 
ExecutableGraph, and add a Portable Representation
URL: https://github.com/apache/beam/pull/5228
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/QueryablePipeline.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/QueryablePipeline.java
index 11006014018..1e24a8ec540 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/QueryablePipeline.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/QueryablePipeline.java
@@ -165,6 +165,15 @@ private static boolean isPrimitiveTransform(PTransform 
transform) {
 return network;
   }
 
+  public Collection getTransforms() {
+return pipelineNetwork
+.nodes()
+.stream()
+.filter(PTransformNode.class::isInstance)
+.map(PTransformNode.class::cast)
+.collect(Collectors.toList());
+  }
+
   public Iterable getTopologicallyOrderedTransforms() {
 return StreamSupport.stream(
 Networks.topologicalOrder(pipelineNetwork, 
Comparator.comparing(PipelineNode::getId))
@@ -217,6 +226,19 @@ public PTransformNode getProducer(PCollectionNode 
pcollection) {
 .collect(Collectors.toSet());
   }
 
+  /**
+   * Gets each {@link PCollectionNode} that the provided {@link 
PTransformNode} consumes on a
+   * per-element basis.
+   */
+  public Set getPerElementInputPCollections(PTransformNode 
ptransform) {
+return pipelineNetwork
+.inEdges(ptransform)
+.stream()
+.filter(PipelineEdge::isPerElement)
+.map(edge -> (PCollectionNode) 
pipelineNetwork.incidentNodes(edge).source())
+.collect(Collectors.toSet());
+  }
+
   public Set getOutputPCollections(PTransformNode ptransform) 
{
 return pipelineNetwork
 .successors(ptransform)
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGraph.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGraph.java
index f52c667abe0..e5a2fa8e641 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGraph.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGraph.java
@@ -81,10 +81,10 @@ private DirectGraph(
   }
 
   @Override
-  public Collection getProduced(AppliedPTransform toRefresh) {
+  public Collection getProduced(AppliedPTransform producer) {
 // TODO: This must only be called on primitive transforms; composites 
should return empty
 // values.
-return toRefresh.getOutputs().values();
+return producer.getOutputs().values();
   }
 
   @Override
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutableGraph.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutableGraph.java
index bfdf692c85a..48f119b7463 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutableGraph.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutableGraph.java
@@ -35,7 +35,7 @@
 
   ExecutableT getProducer(CollectionT collection);
 
-  Collection getProduced(ExecutableT toRefresh);
+  Collection getProduced(ExecutableT producer);
 
   Collection getPerElementInputs(ExecutableT transform);
 
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/PortableGraph.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/PortableGraph.java
new file mode 100644
index 000..0e349a2203b
--- /dev/null
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/PortableGraph.java
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1522

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add additional methods to QueryablePipeline

[tgroh] Expand the ExecutableGraph interface

--
Started by GitHub push by tgroh
[EnvInject] - Loading node environment variables.
Building remotely on beam17 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4971e7ec48ad85d002a1c7596cb436dc8c8e88ea (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4971e7ec48ad85d002a1c7596cb436dc8c8e88ea
Commit message: "Merge pull request #5228: Expand ExecutableGraph, and add a 
Portable Representation"
 > git rev-list --no-walk 184b4a600e49873c015f1c72cd04e65196210fb1 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins3121169963595458885.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user mair...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user jasonkus...@google.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user ankurgoe...@gmail.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com

[beam] branch master updated (184b4a6 -> 4971e7e)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 184b4a6  Merge pull request #5172: Add ProtoOverrides
 add b0cafc9  Add additional methods to QueryablePipeline
 add 7a948d1  Expand the ExecutableGraph interface
 new 4971e7e  Merge pull request #5228: Expand ExecutableGraph, and add a 
Portable Representation

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/construction/graph/QueryablePipeline.java |  22 +++
 .../apache/beam/runners/direct/DirectGraph.java|   4 +-
 .../beam/runners/direct/ExecutableGraph.java   |   2 +-
 .../apache/beam/runners/direct/PortableGraph.java  |  69 
 .../beam/runners/direct/PortableGraphTest.java | 178 +
 5 files changed, 272 insertions(+), 3 deletions(-)
 create mode 100644 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/PortableGraph.java
 create mode 100644 
runners/direct-java/src/test/java/org/apache/beam/runners/direct/PortableGraphTest.java

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] 01/01: Merge pull request #5228: Expand ExecutableGraph, and add a Portable Representation

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 4971e7ec48ad85d002a1c7596cb436dc8c8e88ea
Merge: 184b4a6 7a948d1
Author: Thomas Groh 
AuthorDate: Thu May 3 15:20:07 2018 -0700

Merge pull request #5228: Expand ExecutableGraph, and add a Portable 
Representation

[BEAM-4073]

 .../core/construction/graph/QueryablePipeline.java |  22 +++
 .../apache/beam/runners/direct/DirectGraph.java|   4 +-
 .../beam/runners/direct/ExecutableGraph.java   |   2 +-
 .../apache/beam/runners/direct/PortableGraph.java  |  69 
 .../beam/runners/direct/PortableGraphTest.java | 178 +
 5 files changed, 272 insertions(+), 3 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[jira] [Work logged] (BEAM-4073) The DirectRunner should interact with a Pipeline via an abstraction of the Graph rather than SDK types

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4073?focusedWorklogId=98059=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98059
 ]

ASF GitHub Bot logged work on BEAM-4073:


Author: ASF GitHub Bot
Created on: 03/May/18 22:10
Start Date: 03/May/18 22:10
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5228: 
[BEAM-4073] Expand ExecutableGraph, and add a Portable Representation
URL: https://github.com/apache/beam/pull/5228#discussion_r185914949
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/QueryablePipeline.java
 ##
 @@ -217,6 +226,19 @@ public PTransformNode getProducer(PCollectionNode 
pcollection) {
 .collect(Collectors.toSet());
   }
 
+  /**
+   * Gets each {@link PCollectionNode} that the provided {@link 
PTransformNode} consumes on a
+   * per-element basis.
+   */
+  public Set getPerElementInputs(PTransformNode ptransform) {
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98059)
Time Spent: 1h  (was: 50m)

> The DirectRunner should interact with a Pipeline via an abstraction of the 
> Graph rather than SDK types
> --
>
> Key: BEAM-4073
> URL: https://issues.apache.org/jira/browse/BEAM-4073
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4231) Runner utility for Coder instantiation

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4231?focusedWorklogId=98056=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98056
 ]

ASF GitHub Bot logged work on BEAM-4231:


Author: ASF GitHub Bot
Created on: 03/May/18 22:06
Start Date: 03/May/18 22:06
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on issue #5274: [BEAM-4231] Runner 
utility for coder instantiation
URL: https://github.com/apache/beam/pull/5274#issuecomment-386451775
 
 
   R: @tgroh 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98056)
Time Spent: 20m  (was: 10m)

> Runner utility for Coder instantiation
> --
>
> Key: BEAM-4231
> URL: https://issues.apache.org/jira/browse/BEAM-4231
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Portable runners need to instantiate coders for communicating with SDK 
> harnesses in a consistent way. They cannot simply instantiate coders as 
> defined by PCollections because some component coders may only be known to 
> specific SDKs. Unknown coders should be length-prefixed; the underlying 
> elements should only be exposed to runners as byte strings.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4231) Runner utility for Coder instantiation

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4231?focusedWorklogId=98055=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98055
 ]

ASF GitHub Bot logged work on BEAM-4231:


Author: ASF GitHub Bot
Created on: 03/May/18 22:06
Start Date: 03/May/18 22:06
Worklog Time Spent: 10m 
  Work Description: bsidhom opened a new pull request #5274: [BEAM-4231] 
Runner utility for coder instantiation
URL: https://github.com/apache/beam/pull/5274
 
 
   I also slipped in some doc comment fixes.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand:
  - [ ] What the pull request does
  - [ ] Why it does it
  - [ ] How it does it
  - [ ] Why this approach
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Run `./gradlew build` to make sure basic checks pass. A more thorough 
check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98055)
Time Spent: 10m
Remaining Estimate: 0h

> Runner utility for Coder instantiation
> --
>
> Key: BEAM-4231
> URL: https://issues.apache.org/jira/browse/BEAM-4231
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Portable runners need to instantiate coders for communicating with SDK 
> harnesses in a consistent way. They cannot simply instantiate coders as 
> defined by PCollections because some component coders may only be known to 
> specific SDKs. Unknown coders should be length-prefixed; the underlying 
> elements should only be exposed to runners as byte strings.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4125) Add a library to manipulate the proto representation of a pipeline

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-4125.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Add a library to manipulate the proto representation of a pipeline
> --
>
> Key: BEAM-4125
> URL: https://issues.apache.org/jira/browse/BEAM-4125
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
> Fix For: Not applicable
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> This is important for a transform which includes in-environment transforms 
> (such as a lifted Combine), or for runners which use the beam representation 
> as their internal representation (such as the directrunner)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4125) Add a library to manipulate the proto representation of a pipeline

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4125?focusedWorklogId=98053=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98053
 ]

ASF GitHub Bot logged work on BEAM-4125:


Author: ASF GitHub Bot
Created on: 03/May/18 21:59
Start Date: 03/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: tgroh closed pull request #5172: [BEAM-4125] Add 
ProtoOverrides
URL: https://github.com/apache/beam/pull/5172
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ProtoOverrides.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ProtoOverrides.java
new file mode 100644
index 000..cd6bc32650a
--- /dev/null
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ProtoOverrides.java
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.core.construction.graph;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
+import org.apache.beam.model.pipeline.v1.RunnerApi.ComponentsOrBuilder;
+import org.apache.beam.model.pipeline.v1.RunnerApi.MessageWithComponents;
+import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.runners.PTransformOverride;
+
+/**
+ * A way to apply a Proto-based {@link PTransformOverride}.
+ *
+ * This should generally be used to replace runner-executed transforms with 
runner-executed
+ * composites and simpler runner-executed primitives. It is generically less 
powerful than the
+ * native {@link org.apache.beam.sdk.Pipeline#replaceAll(List)} and more 
error-prone, so should only
+ * be used for relatively simple replacements.
+ */
+@Experimental
+public class ProtoOverrides {
+  /**
+   * Update all composites present in the {@code originalPipeline} with an URN 
equal to the provided
+   * {@code urn} using the provided {@link TransformReplacement}.
+   */
+  public static Pipeline updateTransform(
+  String urn, Pipeline originalPipeline, TransformReplacement 
compositeBuilder) {
+Components.Builder resultComponents = 
originalPipeline.getComponents().toBuilder();
+for (Map.Entry pt :
+originalPipeline.getComponents().getTransformsMap().entrySet()) {
+  if (pt.getValue().getSpec() != null && 
urn.equals(pt.getValue().getSpec().getUrn())) {
+MessageWithComponents updated =
+compositeBuilder.getReplacement(pt.getKey(), 
originalPipeline.getComponents());
+checkArgument(
+
updated.getPtransform().getOutputsMap().equals(pt.getValue().getOutputsMap()),
+"A %s must produce all of the outputs of the original %s",
+TransformReplacement.class.getSimpleName(),
+PTransform.class.getSimpleName());
+removeSubtransforms(pt.getValue(), resultComponents);
+resultComponents
+.mergeFrom(updated.getComponents())
+.putTransforms(pt.getKey(), updated.getPtransform());
+  }
+}
+return 
originalPipeline.toBuilder().setComponents(resultComponents).build();
+  }
+
+  /**
+   * Remove all subtransforms of the provided transform recursively.A {@link 
PTransform} can be the
+   * subtransform of only one enclosing transform.
+   */
+  private static void removeSubtransforms(PTransform pt, Components.Builder 
target) {
+for (String subtransformId : pt.getSubtransformsList()) {
+  PTransform subtransform = target.getTransformsOrThrow(subtransformId);
+  removeSubtransforms(subtransform, target);
+  

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1521

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add ProtoOverrides

--
Started by GitHub push by tgroh
[EnvInject] - Loading node environment variables.
Building remotely on beam17 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 184b4a600e49873c015f1c72cd04e65196210fb1 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 184b4a600e49873c015f1c72cd04e65196210fb1
Commit message: "Merge pull request #5172: Add ProtoOverrides"
 > git rev-list --no-walk b6dadffb3c35648839909790de3a070fb6c34127 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins2575969427361887179.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user mair...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user jasonkus...@google.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user ankurgoe...@gmail.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com

[beam] 01/01: Merge pull request #5172: Add ProtoOverrides

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 184b4a600e49873c015f1c72cd04e65196210fb1
Merge: b6dadff 02fbd1f
Author: Thomas Groh 
AuthorDate: Thu May 3 14:59:11 2018 -0700

Merge pull request #5172: Add ProtoOverrides

[BEAM-4125]

 .../core/construction/graph/ProtoOverrides.java| 102 +
 .../construction/graph/ProtoOverridesTest.java | 255 +
 2 files changed, 357 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] branch master updated (b6dadff -> 184b4a6)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b6dadff  Merge pull request #5272: Add labels flag to Go SDK Dataflow 
Runner.
 add 02fbd1f  Add ProtoOverrides
 new 184b4a6  Merge pull request #5172: Add ProtoOverrides

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/construction/graph/ProtoOverrides.java| 102 +
 .../construction/graph/ProtoOverridesTest.java | 255 +
 2 files changed, 357 insertions(+)
 create mode 100644 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ProtoOverrides.java
 create mode 100644 
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/ProtoOverridesTest.java

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[jira] [Work logged] (BEAM-4031) Add missing dataflow customization options for Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4031?focusedWorklogId=98052=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98052
 ]

ASF GitHub Bot logged work on BEAM-4031:


Author: ASF GitHub Bot
Created on: 03/May/18 21:57
Start Date: 03/May/18 21:57
Worklog Time Spent: 10m 
  Work Description: tgroh closed pull request #5272: [BEAM-4031] Add labels 
flag to Go SDK Dataflow Runner.
URL: https://github.com/apache/beam/pull/5272
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/go/pkg/beam/runners/dataflow/dataflow.go 
b/sdks/go/pkg/beam/runners/dataflow/dataflow.go
index 093628769f3..38d04fee143 100644
--- a/sdks/go/pkg/beam/runners/dataflow/dataflow.go
+++ b/sdks/go/pkg/beam/runners/dataflow/dataflow.go
@@ -52,6 +52,7 @@ var (
endpoint= flag.String("dataflow_endpoint", "", "Dataflow 
endpoint (optional).")
stagingLocation = flag.String("staging_location", "", "GCS staging 
location (required).")
image   = flag.String("worker_harness_container_image", "", 
"Worker harness container image (required).")
+   labels  = flag.String("labels", "", "JSON-formatted 
map[string]string of job labels (optional).")
numWorkers  = flag.Int64("num_workers", 0, "Number of workers 
(optional).")
zone= flag.String("zone", "", "GCP zone (optional)")
region  = flag.String("region", "us-central1", "GCP Region 
(optional)")
@@ -93,6 +94,12 @@ func Execute(ctx context.Context, p *beam.Pipeline) error {
if *image == "" {
*image = jobopts.GetContainerImage(ctx)
}
+   var jobLabels map[string]string
+   if *labels != "" {
+   if err := json.Unmarshal([]byte(*labels), ); err != 
nil {
+   return fmt.Errorf("Error reading --label flag as JSON: 
%v", err)
+   }
+   }
jobName := jobopts.GetJobName()
 
edges, _, err := p.Build()
@@ -201,6 +208,7 @@ func Execute(ctx context.Context, p *beam.Pipeline) error {
TempStoragePrefix: *stagingLocation + "/tmp",
Experiments:   append(jobopts.GetExperiments(), 
"beam_fn_api"),
},
+   Labels: jobLabels,
Steps: steps,
}
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98052)
Time Spent: 2h  (was: 1h 50m)

> Add missing dataflow customization options for Go SDK
> -
>
> Key: BEAM-4031
> URL: https://issues.apache.org/jira/browse/BEAM-4031
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Jason Kuster
>Priority: Minor
> Fix For: 2.5.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> We're missing at least:
> zone
> temp_location
> worker_machine_type



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #272

[Apache Jenkins Server]

See 


Changes:

[sidhom] Move SyntheticNodes to top-level construction package and rename

[sidhom] Pull Flink execution environment utilities into static methods

[sidhom] Refactor Runner/SDK wire coder creation into a shared utility

[sidhom] [BEAM-3972] Translate portable batch pipelines by proto

[sidhom] FIXUP: Use ModelCoders to create proto coders and use WireCoders to

[sidhom] FIXUP: Use RunnerApi in WireCoders to prepare for coder instantiation

--
[...truncated 19.00 MB...]
INFO: 2018-05-03T21:53:58.855Z: Fusing unzipped copy of 
PAssert$3/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous), through 
flatten s18-u63, into producer 
PAssert$3/CreateActual/FilterActuals/Window.Assign
May 03, 2018 9:54:04 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T21:53:58.894Z: Fusing consumer 
PAssert$3/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous) into 
PAssert$3/CreateActual/FilterActuals/Window.Assign
May 03, 2018 9:54:04 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T21:53:58.937Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 03, 2018 9:54:04 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T21:53:58.969Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 03, 2018 9:54:04 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T21:53:59.011Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
May 03, 2018 9:54:04 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T21:53:59.051Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
May 03, 2018 9:54:04 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T21:53:59.086Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
May 03, 2018 9:54:04 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T21:53:59.126Z: Fusing consumer 
PAssert$3/CreateActual/RewindowActuals/Window.Assign into 
PAssert$3/CreateActual/Flatten.Iterables/FlattenIterables/FlatMap
May 03, 2018 9:54:04 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T21:53:59.169Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
May 03, 2018 9:54:04 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T21:53:59.206Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
May 03, 2018 9:54:04 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T21:53:59.239Z: Fusing consumer 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1520

[Apache Jenkins Server]

See 


Changes:

[jasonkuster] Add labels to Go SDK Dataflow Runner.

[jasonkuster] Add verbose error message if unmarshaling fails for label flag.

--
Started by GitHub push by tgroh
[EnvInject] - Loading node environment variables.
Building remotely on beam23 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision b6dadffb3c35648839909790de3a070fb6c34127 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b6dadffb3c35648839909790de3a070fb6c34127
Commit message: "Merge pull request #5272: Add labels flag to Go SDK Dataflow 
Runner."
 > git rev-list --no-walk 5cb3969b53ce7fecb33b347c7113602957f5ee9f # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins5918516035352385300.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user mair...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user jasonkus...@google.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user ankurgoe...@gmail.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com

[beam] 01/01: Merge pull request #5272: Add labels flag to Go SDK Dataflow Runner.

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b6dadffb3c35648839909790de3a070fb6c34127
Merge: 5cb3969 b390ba3
Author: Thomas Groh 
AuthorDate: Thu May 3 14:57:00 2018 -0700

Merge pull request #5272: Add labels flag to Go SDK Dataflow Runner.

[BEAM-4031]

 sdks/go/pkg/beam/runners/dataflow/dataflow.go | 8 
 1 file changed, 8 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] branch master updated (5cb3969 -> b6dadff)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 5cb3969  Merge pull request #5226: [BEAM-3972] Translate portable 
batch pipelines by proto
 add ea9ade2  Add labels to Go SDK Dataflow Runner.
 add b390ba3  Add verbose error message if unmarshaling fails for label 
flag.
 new b6dadff  Merge pull request #5272: Add labels flag to Go SDK Dataflow 
Runner.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/go/pkg/beam/runners/dataflow/dataflow.go | 8 
 1 file changed, 8 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[jira] [Created] (BEAM-4232) ValueProvider with default value fails using DirectRunner

[Frank Yellin (JIRA)]

Frank Yellin created BEAM-4232:
--

 Summary: ValueProvider with default value fails using DirectRunner
 Key: BEAM-4232
 URL: https://issues.apache.org/jira/browse/BEAM-4232
 Project: Beam
  Issue Type: Bug
  Components: runner-dataflow
Reporter: Frank Yellin
Assignee: Thomas Groh


 The bug is shown by the following short piece of code. 
{code:java}
public class ShowProblem implements Serializable {
  public interface MyOptions 
 extends PipelineOptions, DataflowPipelineOptions {
      @Description("OutputFile")
      @Default.String("gs://xyzzy/helloWorld")
      ValueProvider getOutput();
      void setOutput(ValueProvider value);
    }
  public static void main(String args[]) {
    MyOptions options =  PipelineOptionsFactory.fromArgs(args)              
.as(MyOptions.class);
    Pipeline pipeline = Pipeline.create(options);
    pipeline
        .apply(Create.of("hello", "world", "!"))
        .apply(TextIO.write().to(options.getOutput()));
    pipeline.run();
  }
}

{code}
 If this pipeline is run with the "–output=gs://xyzzy/myfile" option, it runs 
fine both locally and on Google.  If this pipeline is run without the --option, 
it runs fine on Google, but fails when run locally.

When given an output argument, the ValueProvider returned by getOutput() is a 
StaticValueProvider; when using the default value returns a 
RuntimeValueProvider,   The latter seems to have a bug hinted at in the comment 
just above RuntimeValueProvider.get().  The code expects that when running the 
pipeline (and only when running the pipeline), someone has already called
{code:java}
RuntimeValueProvider.setRuntimeOptions(options){code}
There is apparently no call to this method when using a DirectRunner. 

 

 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4031) Add missing dataflow customization options for Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4031?focusedWorklogId=98047=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98047
 ]

ASF GitHub Bot logged work on BEAM-4031:


Author: ASF GitHub Bot
Created on: 03/May/18 21:43
Start Date: 03/May/18 21:43
Worklog Time Spent: 10m 
  Work Description: jasonkuster commented on issue #5272: [BEAM-4031] Add 
labels flag to Go SDK Dataflow Runner.
URL: https://github.com/apache/beam/pull/5272#issuecomment-386446626
 
 
   @tgroh can you give this a quick look?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98047)
Time Spent: 1h 50m  (was: 1h 40m)

> Add missing dataflow customization options for Go SDK
> -
>
> Key: BEAM-4031
> URL: https://issues.apache.org/jira/browse/BEAM-4031
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Jason Kuster
>Priority: Minor
> Fix For: 2.5.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> We're missing at least:
> zone
> temp_location
> worker_machine_type



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4031) Add missing dataflow customization options for Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4031?focusedWorklogId=98046=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98046
 ]

ASF GitHub Bot logged work on BEAM-4031:


Author: ASF GitHub Bot
Created on: 03/May/18 21:42
Start Date: 03/May/18 21:42
Worklog Time Spent: 10m 
  Work Description: jasonkuster commented on a change in pull request 
#5272: [BEAM-4031] Add labels flag to Go SDK Dataflow Runner.
URL: https://github.com/apache/beam/pull/5272#discussion_r185944748
 
 

 ##
 File path: sdks/go/pkg/beam/runners/dataflow/dataflow.go
 ##
 @@ -93,6 +94,12 @@ func Execute(ctx context.Context, p *beam.Pipeline) error {
if *image == "" {
*image = jobopts.GetContainerImage(ctx)
}
+   var jobLabels map[string]string
+   if *labels != "" {
+   if err := json.Unmarshal([]byte(*labels), ); err != 
nil {
+   return err
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98046)
Time Spent: 1h 40m  (was: 1.5h)

> Add missing dataflow customization options for Go SDK
> -
>
> Key: BEAM-4031
> URL: https://issues.apache.org/jira/browse/BEAM-4031
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Jason Kuster
>Priority: Minor
> Fix For: 2.5.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> We're missing at least:
> zone
> temp_location
> worker_machine_type



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=98044=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98044
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 03/May/18 21:31
Start Date: 03/May/18 21:31
Worklog Time Spent: 10m 
  Work Description: angoenka opened a new pull request #5273: [BEAM-3883] 
Adding Client to push artifacts to artifact staging service
URL: https://github.com/apache/beam/pull/5273
 
 
   Portability framework requires artifacts to be staged at artifact staging 
service. This PR contains the changes to:
   1. Provide a client to push changes to artifact staging service.
   2. Integrating it with ULR.
   3. Provide a file based artifact staging service in ULR job service.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand:
  - [ ] What the pull request does
  - [ ] Why it does it
  - [ ] How it does it
  - [ ] Why this approach
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Run `./gradlew build` to make sure basic checks pass. A more thorough 
check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98044)
Time Spent: 1h 20m  (was: 1h 10m)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4031) Add missing dataflow customization options for Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4031?focusedWorklogId=98042=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98042
 ]

ASF GitHub Bot logged work on BEAM-4031:


Author: ASF GitHub Bot
Created on: 03/May/18 21:26
Start Date: 03/May/18 21:26
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #5272: 
[BEAM-4031] Add labels flag to Go SDK Dataflow Runner.
URL: https://github.com/apache/beam/pull/5272#discussion_r185941021
 
 

 ##
 File path: sdks/go/pkg/beam/runners/dataflow/dataflow.go
 ##
 @@ -93,6 +94,12 @@ func Execute(ctx context.Context, p *beam.Pipeline) error {
if *image == "" {
*image = jobopts.GetContainerImage(ctx)
}
+   var jobLabels map[string]string
+   if *labels != "" {
+   if err := json.Unmarshal([]byte(*labels), ); err != 
nil {
+   return err
 
 Review comment:
   return  a descriptive error


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98042)
Time Spent: 1.5h  (was: 1h 20m)

> Add missing dataflow customization options for Go SDK
> -
>
> Key: BEAM-4031
> URL: https://issues.apache.org/jira/browse/BEAM-4031
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Jason Kuster
>Priority: Minor
> Fix For: 2.5.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> We're missing at least:
> zone
> temp_location
> worker_machine_type



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4031) Add missing dataflow customization options for Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4031?focusedWorklogId=98039=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98039
 ]

ASF GitHub Bot logged work on BEAM-4031:


Author: ASF GitHub Bot
Created on: 03/May/18 21:23
Start Date: 03/May/18 21:23
Worklog Time Spent: 10m 
  Work Description: jasonkuster opened a new pull request #5272: 
[BEAM-4031] Add labels flag to Go SDK Dataflow Runner.
URL: https://github.com/apache/beam/pull/5272
 
 
   Signed-off-by: Jason Kuster 
   
   Add labels flag to Go SDK Dataflow Runner.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand:
  - [ ] What the pull request does
  - [ ] Why it does it
  - [ ] How it does it
  - [ ] Why this approach
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Run `./gradlew build` to make sure basic checks pass. A more thorough 
check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98039)
Time Spent: 1h 10m  (was: 1h)

> Add missing dataflow customization options for Go SDK
> -
>
> Key: BEAM-4031
> URL: https://issues.apache.org/jira/browse/BEAM-4031
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Jason Kuster
>Priority: Minor
> Fix For: 2.5.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> We're missing at least:
> zone
> temp_location
> worker_machine_type



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4031) Add missing dataflow customization options for Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4031?focusedWorklogId=98040=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98040
 ]

ASF GitHub Bot logged work on BEAM-4031:


Author: ASF GitHub Bot
Created on: 03/May/18 21:23
Start Date: 03/May/18 21:23
Worklog Time Spent: 10m 
  Work Description: jasonkuster commented on issue #5272: [BEAM-4031] Add 
labels flag to Go SDK Dataflow Runner.
URL: https://github.com/apache/beam/pull/5272#issuecomment-386440902
 
 
   R: @herohde 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98040)
Time Spent: 1h 20m  (was: 1h 10m)

> Add missing dataflow customization options for Go SDK
> -
>
> Key: BEAM-4031
> URL: https://issues.apache.org/jira/browse/BEAM-4031
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Jason Kuster
>Priority: Minor
> Fix For: 2.5.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> We're missing at least:
> zone
> temp_location
> worker_machine_type



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (BEAM-4031) Add missing dataflow customization options for Go SDK

[Jason Kuster (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Kuster reopened BEAM-4031:

  Assignee: Jason Kuster  (was: Henning Rohde)

Reopening this issue to track adding "labels" flag.

> Add missing dataflow customization options for Go SDK
> -
>
> Key: BEAM-4031
> URL: https://issues.apache.org/jira/browse/BEAM-4031
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Jason Kuster
>Priority: Minor
> Fix For: 2.5.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> We're missing at least:
> zone
> temp_location
> worker_machine_type



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2133) Python SDK should accept a WindowMappingFn in AsIter, AsSingleton, ...

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463082#comment-16463082
 ] 

Luke Cwik commented on BEAM-2133:
-

Is this an oversight in the Java SDK as this isn't allowed there supported 
there as well or is the strategy to have users write new window fns?

> Python SDK should accept a WindowMappingFn in AsIter, AsSingleton, ...
> --
>
> Key: BEAM-2133
> URL: https://issues.apache.org/jira/browse/BEAM-2133
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Ahmet Altay
>Priority: Major
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4231) Runner utility for Coder instantiation

[Ben Sidhom (JIRA)]

Ben Sidhom created BEAM-4231:


 Summary: Runner utility for Coder instantiation
 Key: BEAM-4231
 URL: https://issues.apache.org/jira/browse/BEAM-4231
 Project: Beam
  Issue Type: Bug
  Components: runner-core
Reporter: Ben Sidhom
Assignee: Ben Sidhom


Portable runners need to instantiate coders for communicating with SDK 
harnesses in a consistent way. They cannot simply instantiate coders as defined 
by PCollections because some component coders may only be known to specific 
SDKs. Unknown coders should be length-prefixed; the underlying elements should 
only be exposed to runners as byte strings.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=98038=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98038
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 03/May/18 21:03
Start Date: 03/May/18 21:03
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #5251: [BEAM-3883] Refactor 
and clean dependency.py to make it reusable with artifact service
URL: https://github.com/apache/beam/pull/5251#issuecomment-386435923
 
 
   Ping...
   Emphasizing again 
   "The changes seems to be a lot because of refactoring. I suggest checking 
individual commits to reduce area to focus while reviewing."


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98038)
Time Spent: 1h 10m  (was: 1h)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3972) Flink runner translates batch pipelines directly by proto

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3972?focusedWorklogId=98037=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98037
 ]

ASF GitHub Bot logged work on BEAM-3972:


Author: ASF GitHub Bot
Created on: 03/May/18 21:01
Start Date: 03/May/18 21:01
Worklog Time Spent: 10m 
  Work Description: jkff closed pull request #5226: [BEAM-3972] Translate 
portable batch pipelines by proto
URL: https://github.com/apache/beam/pull/5226
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/SyntheticNodes.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SyntheticComponents.java
similarity index 87%
rename from 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/SyntheticNodes.java
rename to 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SyntheticComponents.java
index fc2cb3dc562..f7adf6de5e1 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/SyntheticNodes.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SyntheticComponents.java
@@ -16,15 +16,15 @@
  * limitations under the License.
  */
 
-package org.apache.beam.runners.core.construction.graph;
+package org.apache.beam.runners.core.construction;
 
 import java.util.function.Predicate;
 
 /**
- * A utility class to interact with synthetic {@link PipelineNode Pipeline 
Nodes}.
+ * A utility class to interact with synthetic pipeline components.
  */
-class SyntheticNodes {
-  private SyntheticNodes() {}
+public class SyntheticComponents {
+  private SyntheticComponents() {}
 
   /**
* Generate an ID which does not collide with any existing ID, as determined 
by the input
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/FusedPipeline.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/FusedPipeline.java
index ddc03355a90..df3aa5fdfb7 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/FusedPipeline.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/FusedPipeline.java
@@ -30,6 +30,7 @@
 import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
 import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
 import org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline;
+import org.apache.beam.runners.core.construction.SyntheticComponents;
 import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
 
 /** A {@link Pipeline} which has been separated into collections of executable 
components. */
@@ -106,7 +107,7 @@ static FusedPipeline of(
   Set usedNames =
   Sets.union(topLevelTransforms.keySet(), 
getComponents().getTransformsMap().keySet());
   topLevelTransforms.put(
-  SyntheticNodes.uniqueId(baseName, usedNames::contains), 
stage.toPTransform());
+  SyntheticComponents.uniqueId(baseName, usedNames::contains), 
stage.toPTransform());
 }
 return topLevelTransforms;
   }
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/OutputDeduplicator.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/OutputDeduplicator.java
index 4419787ede1..45c4a27cdcf 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/OutputDeduplicator.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/OutputDeduplicator.java
@@ -39,6 +39,7 @@
 import org.apache.beam.model.pipeline.v1.RunnerApi.PCollection;
 import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
 import org.apache.beam.runners.core.construction.PTransformTranslation;
+import org.apache.beam.runners.core.construction.SyntheticComponents;
 import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
 import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
 
@@ -131,7 +132,7 @@ static DeduplicationResult ensureSingleProducer(
   PTransform flattenPartialPCollections =
   createFlattenOfPartials(partialFlattenTargets.getKey(), 
partialFlattenTargets.getValue());
   String flattenId =
-  SyntheticNodes.uniqueId("unzipped_flatten", 
unzippedComponents::containsTransforms);
+  

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1519

[Apache Jenkins Server]

See 


Changes:

[sidhom] Move SyntheticNodes to top-level construction package and rename

[sidhom] Pull Flink execution environment utilities into static methods

[sidhom] Refactor Runner/SDK wire coder creation into a shared utility

[sidhom] [BEAM-3972] Translate portable batch pipelines by proto

[sidhom] FIXUP: Use ModelCoders to create proto coders and use WireCoders to

[sidhom] FIXUP: Use RunnerApi in WireCoders to prepare for coder instantiation

--
Started by GitHub push by jkff
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 5cb3969b53ce7fecb33b347c7113602957f5ee9f (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5cb3969b53ce7fecb33b347c7113602957f5ee9f
Commit message: "Merge pull request #5226: [BEAM-3972] Translate portable batch 
pipelines by proto"
 > git rev-list --no-walk ca4e28484eac3911f03159fd5602f3290fad1e71 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins458823943787584.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user mair...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user ankurgoe...@gmail.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com

[beam] branch master updated (ca4e284 -> 5cb3969)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from ca4e284  A few small fixes to catch a few regressions - put String 
literals fi…
 add 06ed16b  Move SyntheticNodes to top-level construction package and 
rename
 add 8346c83  Pull Flink execution environment utilities into static methods
 add 7e6bef9  Refactor Runner/SDK wire coder creation into a shared utility
 add 1e47617  [BEAM-3972] Translate portable batch pipelines by proto
 add c80f90a  FIXUP: Use ModelCoders to create proto coders and use 
WireCoders to create sdk coders
 add 4686be1  FIXUP: Use RunnerApi in WireCoders to prepare for coder 
instantiation
 new 5cb3969  Merge pull request #5226: [BEAM-3972] Translate portable 
batch pipelines by proto

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 ...yntheticNodes.java => SyntheticComponents.java} |   8 +-
 .../core/construction/graph/FusedPipeline.java |   3 +-
 .../construction/graph/OutputDeduplicator.java |   5 +-
 runners/flink/build.gradle |   1 +
 runners/flink/pom.xml  |  11 +
 .../FlinkBatchPortablePipelineTranslator.java  | 610 +
 ...onment.java => FlinkExecutionEnvironments.java} | 153 ++
 .../flink/FlinkPipelineExecutionEnvironment.java   | 145 +
 .../flink/FlinkPortablePipelineTranslator.java}|  25 +-
 .../functions/FlinkExecutableStageFunction.java|  73 +++
 ...va => FlinkExecutableStagePruningFunction.java} |  31 +-
 .../control/ProcessBundleDescriptors.java  |  43 +-
 .../{graph => wire}/LengthPrefixUnknownCoders.java |   2 +-
 .../beam/runners/fnexecution/wire/WireCoders.java  |  65 +++
 .../runners/fnexecution/wire}/package-info.java|   6 +-
 .../LengthPrefixUnknownCodersTest.java |   2 +-
 16 files changed, 846 insertions(+), 337 deletions(-)
 rename 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/{graph/SyntheticNodes.java
 => SyntheticComponents.java} (87%)
 create mode 100644 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
 copy 
runners/flink/src/main/java/org/apache/beam/runners/flink/{FlinkPipelineExecutionEnvironment.java
 => FlinkExecutionEnvironments.java} (51%)
 copy 
runners/{core-construction-java/src/main/java/org/apache/beam/runners/core/construction/BeamUrns.java
 => 
flink/src/main/java/org/apache/beam/runners/flink/FlinkPortablePipelineTranslator.java}
 (51%)
 create mode 100644 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java
 copy 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/{FlinkMultiOutputPruningFunction.java
 => FlinkExecutableStagePruningFunction.java} (56%)
 rename 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/{graph
 => wire}/LengthPrefixUnknownCoders.java (99%)
 create mode 100644 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/wire/WireCoders.java
 copy runners/{apex/src/test/java/org/apache/beam/runners/apex/examples => 
java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/wire}/package-info.java
 (87%)
 rename 
runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/{graph
 => wire}/LengthPrefixUnknownCodersTest.java (99%)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] 01/01: Merge pull request #5226: [BEAM-3972] Translate portable batch pipelines by proto

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 5cb3969b53ce7fecb33b347c7113602957f5ee9f
Merge: ca4e284 4686be1
Author: Eugene Kirpichov 
AuthorDate: Thu May 3 14:01:15 2018 -0700

Merge pull request #5226: [BEAM-3972] Translate portable batch pipelines by 
proto

[BEAM-3972] Translate portable batch pipelines by proto

 ...yntheticNodes.java => SyntheticComponents.java} |   8 +-
 .../core/construction/graph/FusedPipeline.java |   3 +-
 .../construction/graph/OutputDeduplicator.java |   5 +-
 runners/flink/build.gradle |   1 +
 runners/flink/pom.xml  |  11 +
 .../FlinkBatchPortablePipelineTranslator.java  | 610 +
 ...onment.java => FlinkExecutionEnvironments.java} | 153 ++
 .../flink/FlinkPipelineExecutionEnvironment.java   | 145 +
 .../flink/FlinkPortablePipelineTranslator.java |  40 ++
 .../functions/FlinkExecutableStageFunction.java|  73 +++
 .../FlinkExecutableStagePruningFunction.java   |  45 ++
 .../control/ProcessBundleDescriptors.java  |  43 +-
 .../{graph => wire}/LengthPrefixUnknownCoders.java |   2 +-
 .../beam/runners/fnexecution/wire/WireCoders.java  |  65 +++
 .../runners/fnexecution/wire/package-info.java}|  28 +-
 .../LengthPrefixUnknownCodersTest.java |   2 +-
 16 files changed, 900 insertions(+), 334 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #271

[Apache Jenkins Server]

See 


Changes:

[coheigea] A few small fixes to catch a few regressions - put String literals

--
[...truncated 19.59 MB...]
INFO: 2018-05-03T20:56:33.045Z: Unzipping flatten s18-u63 for input 
s19.output-c61
May 03, 2018 8:56:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T20:56:33.071Z: Fusing unzipped copy of 
PAssert$3/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous), through 
flatten s18-u63, into producer 
PAssert$3/CreateActual/FilterActuals/Window.Assign
May 03, 2018 8:56:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T20:56:33.102Z: Fusing consumer 
PAssert$3/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous) into 
PAssert$3/CreateActual/FilterActuals/Window.Assign
May 03, 2018 8:56:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T20:56:33.138Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 03, 2018 8:56:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T20:56:33.175Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 03, 2018 8:56:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T20:56:33.209Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
May 03, 2018 8:56:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T20:56:33.239Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
May 03, 2018 8:56:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T20:56:33.278Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
May 03, 2018 8:56:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T20:56:33.307Z: Fusing consumer 
PAssert$3/CreateActual/RewindowActuals/Window.Assign into 
PAssert$3/CreateActual/Flatten.Iterables/FlattenIterables/FlatMap
May 03, 2018 8:56:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T20:56:33.348Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
May 03, 2018 8:56:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T20:56:33.374Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
May 03, 2018 8:56:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T20:56:33.410Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
 into 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1518

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam23 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision ca4e28484eac3911f03159fd5602f3290fad1e71 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ca4e28484eac3911f03159fd5602f3290fad1e71
Commit message: "A few small fixes to catch a few regressions - put String 
literals fi…"
 > git rev-list --no-walk ca4e28484eac3911f03159fd5602f3290fad1e71 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins7543009452845685142.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user mair...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user ankurgoe...@gmail.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com

[jira] [Updated] (BEAM-2937) Fn API combiner support w/ lifting to PGBK

[Daniel Oliveira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Oliveira updated BEAM-2937:
--
Description: 
The FnAPI should support this optimization. Detailed design: 
https://s.apache.org/beam-runner-api-combine-model

Once design is ready, expand subtasks similarly to BEAM-2822.

  was:
The FnAPI should support this optimization. Detailed design TBD.

Once design is ready, expand subtasks similarly to BEAM-2822.


> Fn API combiner support w/ lifting to PGBK
> --
>
> Key: BEAM-2937
> URL: https://issues.apache.org/jira/browse/BEAM-2937
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Henning Rohde
>Assignee: Daniel Oliveira
>Priority: Major
>  Labels: portability
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> The FnAPI should support this optimization. Detailed design: 
> https://s.apache.org/beam-runner-api-combine-model
> Once design is ready, expand subtasks similarly to BEAM-2822.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=98030=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98030
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 03/May/18 20:32
Start Date: 03/May/18 20:32
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5269: 
[BEAM-3326] Add an Unsupported StateRequestHandler
URL: https://github.com/apache/beam/pull/5269#discussion_r185927830
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandler.java
 ##
 @@ -34,6 +35,12 @@
* Throwing an error during handling will complete the handler result 
{@link CompletionStage}
* exceptionally.
*/
-  CompletionStage 
handle(BeamFnApi.StateRequest request)
-  throws Exception;
+  CompletionStage handle(StateRequest request) throws 
Exception;
+
+  static StateRequestHandler unsupported() {
 
 Review comment:
   It is definitely not redundant - you must choose between `static`, 
`default`, or no-implementation for interface methods


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98030)
Time Spent: 7h 40m  (was: 7.5h)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 7h 40m
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1517

[Apache Jenkins Server]

See 


Changes:

[coheigea] A few small fixes to catch a few regressions - put String literals

--
Started by GitHub push by lukecwik
[EnvInject] - Loading node environment variables.
Building remotely on beam23 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision ca4e28484eac3911f03159fd5602f3290fad1e71 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ca4e28484eac3911f03159fd5602f3290fad1e71
Commit message: "A few small fixes to catch a few regressions - put String 
literals fi…"
 > git rev-list --no-walk fe7717da7bbf04b2f58a4c09a1b79225f85b33a7 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins1388054051810542423.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user mair...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user ankurgoe...@gmail.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com

[beam] branch master updated (fe7717d -> ca4e284)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from fe7717d  Merge pull request #5265: Templatize host name in SpannerIO
 add c9d65b2  A few small fixes to catch a few regressions - put String 
literals first, and avoid unnecessary autoboxing etc.
 new ca4e284  A few small fixes to catch a few regressions - put String 
literals fi…

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../java/org/apache/beam/sdk/options/PipelineOptionsFactory.java| 2 +-
 .../main/java/org/apache/beam/sdk/util/RowJsonValueExtractors.java  | 2 +-
 .../org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystem.java   | 2 +-
 .../java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java | 2 +-
 .../src/main/java/org/apache/beam/sdk/io/hdfs/HadoopResourceId.java | 2 +-
 .../src/main/java/org/apache/beam/sdk/io/kafka/ConsumerSpEL.java| 6 +++---
 6 files changed, 8 insertions(+), 8 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] 01/01: A few small fixes to catch a few regressions - put String literals fi…

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit ca4e28484eac3911f03159fd5602f3290fad1e71
Merge: fe7717d c9d65b2
Author: Lukasz Cwik 
AuthorDate: Thu May 3 13:02:31 2018 -0700

A few small fixes to catch a few regressions - put String literals fi…

 .../java/org/apache/beam/sdk/options/PipelineOptionsFactory.java| 2 +-
 .../main/java/org/apache/beam/sdk/util/RowJsonValueExtractors.java  | 2 +-
 .../org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystem.java   | 2 +-
 .../java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java | 2 +-
 .../src/main/java/org/apache/beam/sdk/io/hdfs/HadoopResourceId.java | 2 +-
 .../src/main/java/org/apache/beam/sdk/io/kafka/ConsumerSpEL.java| 6 +++---
 6 files changed, 8 insertions(+), 8 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=98027=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98027
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 03/May/18 20:01
Start Date: 03/May/18 20:01
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5269: 
[BEAM-3326] Add an Unsupported StateRequestHandler
URL: https://github.com/apache/beam/pull/5269#discussion_r185919685
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandler.java
 ##
 @@ -34,6 +35,12 @@
* Throwing an error during handling will complete the handler result 
{@link CompletionStage}
* exceptionally.
*/
-  CompletionStage 
handle(BeamFnApi.StateRequest request)
-  throws Exception;
+  CompletionStage handle(StateRequest request) throws 
Exception;
+
+  static StateRequestHandler unsupported() {
 
 Review comment:
   `static` is also redundant.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98027)
Time Spent: 7.5h  (was: 7h 20m)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #270

[Apache Jenkins Server]

See 


Changes:

[mairbek] Templatize host name in SpannerIO

--
[...truncated 18.41 MB...]
May 03, 2018 7:35:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T19:35:34.400Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
May 03, 2018 7:35:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T19:35:34.428Z: Fusing consumer 
PAssert$3/CreateActual/ParDo(Anonymous) into 
PAssert$3/CreateActual/RewindowActuals/Window.Assign
May 03, 2018 7:35:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T19:35:34.466Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into PAssert$3/CreateActual/ParDo(Anonymous)
May 03, 2018 7:35:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T19:35:34.510Z: Fusing consumer 
Combine.globally(Count)/ProduceDefault into 
Combine.globally(Count)/CreateVoid/Read(CreateSource)
May 03, 2018 7:35:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T19:35:34.556Z: Fusing consumer 
Combine.globally(Count)/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow) into 
Combine.globally(Count)/Values/Values/Map
May 03, 2018 7:35:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T19:35:34.602Z: Fusing consumer 
Combine.globally(Count)/Combine.perKey(Count)/Combine.GroupedValues into 
Combine.globally(Count)/Combine.perKey(Count)/GroupByKey/Read
May 03, 2018 7:35:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T19:35:34.649Z: Fusing consumer 
Combine.globally(Count)/Combine.perKey(Count)/GroupByKey+Combine.globally(Count)/Combine.perKey(Count)/Combine.GroupedValues/Partial
 into Combine.globally(Count)/WithKeys/AddKeys/Map
May 03, 2018 7:35:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T19:35:34.695Z: Fusing consumer 
DatastoreV1.Read/Reshuffle/Reshuffle/GroupByKey/GroupByWindow into 
DatastoreV1.Read/Reshuffle/Reshuffle/GroupByKey/Read
May 03, 2018 7:35:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T19:35:34.740Z: Fusing consumer 
Combine.globally(Count)/Combine.perKey(Count)/GroupByKey/Reify into 
Combine.globally(Count)/Combine.perKey(Count)/GroupByKey+Combine.globally(Count)/Combine.perKey(Count)/Combine.GroupedValues/Partial
May 03, 2018 7:35:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T19:35:34.788Z: Fusing consumer 
Combine.globally(Count)/WithKeys/AddKeys/Map into DatastoreV1.Read/Read
May 03, 2018 7:35:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T19:35:34.835Z: Fusing consumer 
DatastoreV1.Read/Reshuffle/Pair with random key into DatastoreV1.Read/Split
May 03, 2018 7:35:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T19:35:34.867Z: Fusing consumer 
DatastoreV1.Read/Reshuffle/Reshuffle/GroupByKey/Write into 
DatastoreV1.Read/Reshuffle/Reshuffle/GroupByKey/Reify
May 03, 2018 7:35:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T19:35:34.912Z: Fusing consumer 
DatastoreV1.Read/Reshuffle/Reshuffle/GroupByKey/Reify into 
DatastoreV1.Read/Reshuffle/Reshuffle/Window.Into()/Window.Assign
May 03, 2018 7:35:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T19:35:34.960Z: Fusing consumer 
Combine.globally(Count)/Values/Values/Map into 
Combine.globally(Count)/Combine.perKey(Count)/Combine.GroupedValues/Extract
May 03, 2018 7:35:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T19:35:35.004Z: Fusing consumer DatastoreV1.Read/Split into 
DatastoreV1.Read/ParDo(GqlQueryTranslate)
May 03, 2018 7:35:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T19:35:35.041Z: Fusing consumer 
DatastoreV1.Read/Reshuffle/Reshuffle/Window.Into()/Window.Assign into 
DatastoreV1.Read/Reshuffle/Pair with 

[jira] [Work logged] (BEAM-4073) The DirectRunner should interact with a Pipeline via an abstraction of the Graph rather than SDK types

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4073?focusedWorklogId=98013=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98013
 ]

ASF GitHub Bot logged work on BEAM-4073:


Author: ASF GitHub Bot
Created on: 03/May/18 19:14
Start Date: 03/May/18 19:14
Worklog Time Spent: 10m 
  Work Description: huygaa11 commented on a change in pull request #5228: 
[BEAM-4073] Expand ExecutableGraph, and add a Portable Representation
URL: https://github.com/apache/beam/pull/5228#discussion_r185877595
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/QueryablePipeline.java
 ##
 @@ -217,6 +226,19 @@ public PTransformNode getProducer(PCollectionNode 
pcollection) {
 .collect(Collectors.toSet());
   }
 
+  /**
+   * Gets each {@link PCollectionNode} that the provided {@link 
PTransformNode} consumes on a
+   * per-element basis.
+   */
+  public Set getPerElementInputs(PTransformNode ptransform) {
 
 Review comment:
   Unless there is a method name length concern, naming this method 
getPerElementInputPCollections would make it clear and consistent with the 
getOutputPCollections method underneath.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98013)
Time Spent: 50m  (was: 40m)

> The DirectRunner should interact with a Pipeline via an abstraction of the 
> Graph rather than SDK types
> --
>
> Key: BEAM-4073
> URL: https://issues.apache.org/jira/browse/BEAM-4073
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4073) The DirectRunner should interact with a Pipeline via an abstraction of the Graph rather than SDK types

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4073?focusedWorklogId=98014=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98014
 ]

ASF GitHub Bot logged work on BEAM-4073:


Author: ASF GitHub Bot
Created on: 03/May/18 19:14
Start Date: 03/May/18 19:14
Worklog Time Spent: 10m 
  Work Description: huygaa11 commented on a change in pull request #5228: 
[BEAM-4073] Expand ExecutableGraph, and add a Portable Representation
URL: https://github.com/apache/beam/pull/5228#discussion_r185907309
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/PortableGraph.java
 ##
 @@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct;
+
+import java.util.Collection;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
+import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
+import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
+
+/** A {@link ExecutableGraph} for a Portable {@link RunnerApi.Pipeline}. */
+class PortableGraph implements ExecutableGraph {
+  private final QueryablePipeline queryable;
 
 Review comment:
   The name "queryable" can be changed with "queryablePipeline" to add 
understandability to the code.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98014)
Time Spent: 50m  (was: 40m)

> The DirectRunner should interact with a Pipeline via an abstraction of the 
> Graph rather than SDK types
> --
>
> Key: BEAM-4073
> URL: https://issues.apache.org/jira/browse/BEAM-4073
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=98011=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98011
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 03/May/18 19:06
Start Date: 03/May/18 19:06
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5269: 
[BEAM-3326] Add an Unsupported StateRequestHandler
URL: https://github.com/apache/beam/pull/5269#discussion_r185906501
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandler.java
 ##
 @@ -34,6 +35,12 @@
* Throwing an error during handling will complete the handler result 
{@link CompletionStage}
* exceptionally.
*/
-  CompletionStage 
handle(BeamFnApi.StateRequest request)
-  throws Exception;
+  CompletionStage handle(StateRequest request) throws 
Exception;
+
+  static StateRequestHandler unsupported() {
 
 Review comment:
   Ah, I hadn't noticed this was an interface.
   
   I don't think `static` is redundant though.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98011)
Time Spent: 7h 20m  (was: 7h 10m)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=98006=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98006
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 03/May/18 18:55
Start Date: 03/May/18 18:55
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5269: 
[BEAM-3326] Add an Unsupported StateRequestHandler
URL: https://github.com/apache/beam/pull/5269#discussion_r185903479
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandler.java
 ##
 @@ -34,6 +35,12 @@
* Throwing an error during handling will complete the handler result 
{@link CompletionStage}
* exceptionally.
*/
-  CompletionStage 
handle(BeamFnApi.StateRequest request)
-  throws Exception;
+  CompletionStage handle(StateRequest request) throws 
Exception;
+
+  static StateRequestHandler unsupported() {
 
 Review comment:
   classes inside interfaces are public and not package private.
   
   Drop `static` as it is also a redundant modifier.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98006)
Time Spent: 7h 10m  (was: 7h)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 7h 10m
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #190

[Apache Jenkins Server]

See 


--
[...truncated 23.49 MB...]
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 03, 2018 6:41:52 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 03, 2018 6:41:52 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 03, 2018 6:41:52 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 03, 2018 6:41:52 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0503184134-7c1de3e7/output/results/staging/
May 03, 2018 6:41:52 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71843 bytes, hash CAaIYQ3kSHkQK3F-R20-CA> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0503184134-7c1de3e7/output/results/staging/pipeline-CAaIYQ3kSHkQK3F-R20-CA.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 03, 2018 6:41:53 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-03_11_41_53-13289306695269608344?project=apache-beam-testing

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Submitted job: 2018-05-03_11_41_53-13289306695269608344

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 03, 2018 6:41:53 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-03_11_41_53-13289306695269608344
May 03, 2018 6:41:53 PM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-03_11_41_53-13289306695269608344 with 1 
expected assertions.
May 03, 2018 6:42:10 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T18:41:53.146Z: Autoscaling is enabled for job 
2018-05-03_11_41_53-13289306695269608344. The number of workers will be between 
1 and 1000.
May 03, 2018 6:42:10 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T18:41:53.188Z: Autoscaling was automatically enabled for 
job 2018-05-03_11_41_53-13289306695269608344.
May 03, 2018 6:42:10 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T18:41:55.920Z: Checking required Cloud APIs are enabled.
May 03, 2018 6:42:10 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T18:41:56.024Z: Checking permissions granted to controller 
Service Account.
May 03, 2018 6:42:10 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T18:42:01.217Z: Worker configuration: n1-standard-1 in 
us-central1-c.
May 03, 2018 6:42:10 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T18:42:01.672Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 03, 2018 6:42:10 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T18:42:01.875Z: Expanding GroupByKey operations into 
optimizable parts.
May 03, 2018 6:42:10 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T18:42:01.923Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 03, 2018 6:42:10 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T18:42:02.161Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 03, 2018 6:42:10 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T18:42:02.198Z: Elided trivial flatten 
May 03, 2018 6:42:10 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T18:42:02.237Z: Elided trivial flatten 
May 03, 2018 6:42:10 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-03T18:42:02.284Z: Elided trivial flatten 
May 03, 2018 6:42:10 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1516

[Apache Jenkins Server]

See 


Changes:

[mairbek] Templatize host name in SpannerIO

--
Started by GitHub push by jkff
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision fe7717da7bbf04b2f58a4c09a1b79225f85b33a7 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f fe7717da7bbf04b2f58a4c09a1b79225f85b33a7
Commit message: "Merge pull request #5265: Templatize host name in SpannerIO"
 > git rev-list --no-walk 797908421efa94bb1cedf2a96a435f877c27e98b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins9201512649221847527.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user mair...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user ankurgoe...@gmail.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=98004=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98004
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 03/May/18 18:35
Start Date: 03/May/18 18:35
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5269: 
[BEAM-3326] Add an Unsupported StateRequestHandler
URL: https://github.com/apache/beam/pull/5269#discussion_r185898147
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandler.java
 ##
 @@ -34,6 +35,12 @@
* Throwing an error during handling will complete the handler result 
{@link CompletionStage}
* exceptionally.
*/
-  CompletionStage 
handle(BeamFnApi.StateRequest request)
-  throws Exception;
+  CompletionStage handle(StateRequest request) throws 
Exception;
+
+  static StateRequestHandler unsupported() {
 
 Review comment:
   Why package-private? This seems useful anywhere we don't/can't support state.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98004)
Time Spent: 7h  (was: 6h 50m)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 7h
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #269

[Apache Jenkins Server]

See 


--
[...truncated 18.02 MB...]
May 03, 2018 6:35:33 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.JoinExamplesTest > testExtractCountryInfoFn 
STANDARD_ERROR
May 03, 2018 6:35:33 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.JoinExamplesTest > testExtractEventDataFn 
STANDARD_ERROR
May 03, 2018 6:35:33 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.DebuggingWordCountTest > testDebuggingWordCount 
STANDARD_ERROR
May 03, 2018 6:35:33 PM org.apache.beam.sdk.io.FileBasedSource 
getEstimatedSizeBytes
INFO: Filepattern 
/tmp/junit6409348636958124581/junit2342581293048061808.tmp matched 1 files with 
total size 54
May 03, 2018 6:35:33 PM org.apache.beam.sdk.io.FileBasedSource split
INFO: Splitting filepattern 
/tmp/junit6409348636958124581/junit2342581293048061808.tmp into bundles of size 
3 took 1 ms and produced 1 files and 18 bundles

org.apache.beam.examples.WordCountTest > testExtractWordsFn STANDARD_ERROR
May 03, 2018 6:35:34 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.subprocess.ExampleEchoPipelineTest > 
testExampleEchoPipeline STANDARD_ERROR
May 03, 2018 6:35:35 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-Echo4341223240733571356.sh 
May 03, 2018 6:35:35 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
May 03, 2018 6:35:35 PM org.apache.beam.examples.subprocess.utils.FileUtils 
copyFileFromGCSToWorker
INFO: Moving File /tmp/test-Echo4341223240733571356.sh to 
/tmp/test-Echoo5831730217039580461/test-Echo4341223240733571356.sh 
May 03, 2018 6:35:35 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-EchoAgain2220651213357172878.sh 
May 03, 2018 6:35:35 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
May 03, 2018 6:35:35 PM org.apache.beam.examples.subprocess.utils.FileUtils 
copyFileFromGCSToWorker
INFO: Moving File /tmp/test-EchoAgain2220651213357172878.sh to 
/tmp/test-Echoo5831730217039580461/test-EchoAgain2220651213357172878.sh 

org.apache.beam.examples.complete.game.HourlyTeamScoreTest > 
testUserScoresFilter STANDARD_OUT
GOT user6_AmberNumbat,AmberNumbat,11,144795563,2015-11-19 09:53:53.444
GOT user18_BananaEmu,BananaEmu,1,144796569,2015-11-19 12:41:31.053
GOT user18_BananaEmu,BananaEmu,7,144796569,2015-11-19 12:41:31.053
GOT user18_ApricotCaneToad,ApricotCaneToad,14,144796569,2015-11-19 
12:41:31.053
GOT user2_AmberCockatoo,AmberCockatoo,13,144796569,2015-11-19 
12:41:31.053
GOT user3_BananaEmu,BananaEmu,17,144796569,2015-11-19 12:41:31.053
GOT 
user0_AndroidGreenEchidna,AndroidGreenEchidna,0,144796569,2015-11-19 
12:41:31.053
GOT user7_AlmondWallaby,AlmondWallaby,15,144795563,2015-11-19 
09:53:53.444
GOT user13_ApricotQuokka,ApricotQuokka,15,144795563,2015-11-19 
09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,11,144795563,2015-11-19 
09:53:53.444
GOT user19_BisqueBilby,BisqueBilby,8,144795563,2015-11-19 09:53:53.444
GOT user0_MagentaKangaroo,MagentaKangaroo,4,144796569,2015-11-19 
12:41:31.053
GOT user0_MagentaKangaroo,MagentaKangaroo,3,144795563,2015-11-19 
09:53:53.444
GOT user19_BisqueBilby,BisqueBilby,6,144795563,2015-11-19 09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,12,144795563,2015-11-19 
09:53:53.444

org.apache.beam.examples.complete.game.UserScoreTest > testTeamScoreSums 
STANDARD_OUT
GOT user0_MagentaKangaroo,MagentaKangaroo,3,144795563,2015-11-19 
09:53:53.444
GOT user6_AmberNumbat,AmberNumbat,11,144795563,2015-11-19 09:53:53.444
GOT user13_ApricotQuokka,ApricotQuokka,15,144795563,2015-11-19 
09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,12,144795563,2015-11-19 
09:53:53.444
GOT user6_AliceBlueDingo,AliceBlueDingo,4,xxx,2015-11-19 09:53:53.444
GOT user7_AlmondWallaby,AlmondWallaby,15,144795563,2015-11-19 
09:53:53.444
GOT 

[jira] [Work logged] (BEAM-4044) Take advantage of Calcite DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4044?focusedWorklogId=97994=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97994
 ]

ASF GitHub Bot logged work on BEAM-4044:


Author: ASF GitHub Bot
Created on: 03/May/18 18:16
Start Date: 03/May/18 18:16
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #5224: 
[BEAM-4044] [SQL] Add tables via TableStore in Schema, execute DDL in Calcite 
model
URL: https://github.com/apache/beam/pull/5224#discussion_r185884947
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/BeamSqlTableProvider.java
 ##
 @@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.extensions.sql.meta.provider;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+
+/**
+ * A {@code BeamSqlTableProvider} provides read only set of {@code 
BeamSqlTable}.
+ */
+public class BeamSqlTableProvider implements TableProvider {
+  private final String typeName;
+  private final Map tables;
+
+  public BeamSqlTableProvider(String typeName, Map 
tables) {
+this.typeName = typeName;
+this.tables = tables;
+  }
+
+  @Override public String getTableType() {
+return typeName;
+  }
+
+  @Override
+  public void createTable(Table table) {
+throw new UnsupportedOperationException();
+  }
+
+  @Override
+  public void dropTable(String tableName) {
+throw new UnsupportedOperationException();
+  }
+
+  @Override
+  public List listTables() {
+List list = new ArrayList(tables.size());
+for (Map.Entry table : tables.entrySet()) {
+  list.add(Table.builder()
+  .type(getTableType())
+  .name(table.getKey())
+  .columns(Collections.emptyList())
+  .build());
 
 Review comment:
   nit: I would rewrite it this way:
   
   ```java
   tables
 .values()
 .stream()
 .map(sqlTable ->
   Table
   .builder()
   .type(getTableType())
   .name(sqlTable.getKey())
   .columns(Collections.emptyList())
   .build())
 .collect(toList());
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97994)
Time Spent: 10h 40m  (was: 10.5h)

> Take advantage of Calcite DDL
> -
>
> Key: BEAM-4044
> URL: https://issues.apache.org/jira/browse/BEAM-4044
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 10h 40m
>  Remaining Estimate: 0h
>
> In Calcite 1.15 support for abstract DDL moved into calcite core. We should 
> take advantage of that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4044) Take advantage of Calcite DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4044?focusedWorklogId=98000=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98000
 ]

ASF GitHub Bot logged work on BEAM-4044:


Author: ASF GitHub Bot
Created on: 03/May/18 18:16
Start Date: 03/May/18 18:16
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #5224: 
[BEAM-4044] [SQL] Add tables via TableStore in Schema, execute DDL in Calcite 
model
URL: https://github.com/apache/beam/pull/5224#discussion_r185883258
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/BeamSqlTableProvider.java
 ##
 @@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.extensions.sql.meta.provider;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+
+/**
+ * A {@code BeamSqlTableProvider} provides read only set of {@code 
BeamSqlTable}.
+ */
+public class BeamSqlTableProvider implements TableProvider {
 
 Review comment:
   make it `@AutoValue`+`Builder`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 98000)
Time Spent: 11.5h  (was: 11h 20m)

> Take advantage of Calcite DDL
> -
>
> Key: BEAM-4044
> URL: https://issues.apache.org/jira/browse/BEAM-4044
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 11.5h
>  Remaining Estimate: 0h
>
> In Calcite 1.15 support for abstract DDL moved into calcite core. We should 
> take advantage of that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4044) Take advantage of Calcite DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4044?focusedWorklogId=97995=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97995
 ]

ASF GitHub Bot logged work on BEAM-4044:


Author: ASF GitHub Bot
Created on: 03/May/18 18:16
Start Date: 03/May/18 18:16
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #5224: 
[BEAM-4044] [SQL] Add tables via TableStore in Schema, execute DDL in Calcite 
model
URL: https://github.com/apache/beam/pull/5224#discussion_r185889574
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlCli.java
 ##
 @@ -78,45 +67,15 @@ public void execute(String sqlString) throws Exception {
 BeamSqlParser parser = new BeamSqlParser(sqlString);
 SqlNode sqlNode = parser.impl().parseSqlStmtEof();
 
-if (sqlNode instanceof SqlCreateTable) {
-  handleCreateTable((SqlCreateTable) sqlNode, metaStore);
-} else if (sqlNode instanceof SqlDropTable) {
-  handleDropTable((SqlDropTable) sqlNode);
+if (sqlNode instanceof SqlExecutableStatement) {
+  ((SqlExecutableStatement) sqlNode).execute(env.getContext());
 } else {
 
 Review comment:
   nit: add a comment what is executable statement, what is not?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97995)
Time Spent: 10h 50m  (was: 10h 40m)

> Take advantage of Calcite DDL
> -
>
> Key: BEAM-4044
> URL: https://issues.apache.org/jira/browse/BEAM-4044
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 10h 50m
>  Remaining Estimate: 0h
>
> In Calcite 1.15 support for abstract DDL moved into calcite core. We should 
> take advantage of that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4044) Take advantage of Calcite DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4044?focusedWorklogId=97991=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97991
 ]

ASF GitHub Bot logged work on BEAM-4044:


Author: ASF GitHub Bot
Created on: 03/May/18 18:16
Start Date: 03/May/18 18:16
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #5224: 
[BEAM-4044] [SQL] Add tables via TableStore in Schema, execute DDL in Calcite 
model
URL: https://github.com/apache/beam/pull/5224#discussion_r185892321
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStore.java
 ##
 @@ -37,7 +37,8 @@
   private Map tables = new HashMap<>();
   private Map providers = new HashMap<>();
 
-  public InMemoryMetaStore() {
+  @Override public String getTableType() {
+return "";
 
 Review comment:
   I think it should have its own table type still


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97991)
Time Spent: 10h 20m  (was: 10h 10m)

> Take advantage of Calcite DDL
> -
>
> Key: BEAM-4044
> URL: https://issues.apache.org/jira/browse/BEAM-4044
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 10h 20m
>  Remaining Estimate: 0h
>
> In Calcite 1.15 support for abstract DDL moved into calcite core. We should 
> take advantage of that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4044) Take advantage of Calcite DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4044?focusedWorklogId=97998=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97998
 ]

ASF GitHub Bot logged work on BEAM-4044:


Author: ASF GitHub Bot
Created on: 03/May/18 18:16
Start Date: 03/May/18 18:16
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #5224: 
[BEAM-4044] [SQL] Add tables via TableStore in Schema, execute DDL in Calcite 
model
URL: https://github.com/apache/beam/pull/5224#discussion_r185889031
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteTable.java
 ##
 @@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl;
+
+import java.util.Collection;
+import java.util.List;
+import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSinkRel;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSourceRel;
+import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
+import org.apache.calcite.adapter.java.AbstractQueryableTable;
+import org.apache.calcite.linq4j.QueryProvider;
+import org.apache.calcite.linq4j.Queryable;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.prepare.Prepare;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.TableModify;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.schema.ModifiableTable;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.TranslatableTable;
+
+/**
+ * Adapter from {@link BeamSqlTable} to a calcite Table.
+ */
+class BeamCalciteTable extends AbstractQueryableTable
+implements ModifiableTable, TranslatableTable {
+  private BeamSqlTable beamTable;
+
+  public BeamCalciteTable(BeamSqlTable beamTable) {
+super(Object[].class);
+this.beamTable = beamTable;
+  }
+
+  @Override
+  public RelDataType getRowType(RelDataTypeFactory typeFactory) {
+return CalciteUtils.toCalciteRowType(this.beamTable.getSchema(), 
BeamQueryPlanner.TYPE_FACTORY);
 
 Review comment:
   Create this in constructor?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97998)
Time Spent: 11h 10m  (was: 11h)

> Take advantage of Calcite DDL
> -
>
> Key: BEAM-4044
> URL: https://issues.apache.org/jira/browse/BEAM-4044
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 11h 10m
>  Remaining Estimate: 0h
>
> In Calcite 1.15 support for abstract DDL moved into calcite core. We should 
> take advantage of that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4044) Take advantage of Calcite DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4044?focusedWorklogId=97992=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97992
 ]

ASF GitHub Bot logged work on BEAM-4044:


Author: ASF GitHub Bot
Created on: 03/May/18 18:16
Start Date: 03/May/18 18:16
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #5224: 
[BEAM-4044] [SQL] Add tables via TableStore in Schema, execute DDL in Calcite 
model
URL: https://github.com/apache/beam/pull/5224#discussion_r185880448
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDropObject.java
 ##
 @@ -51,8 +59,39 @@
 name.unparse(writer, leftPrec, rightPrec);
   }
 
-  public String getNameSimple() {
-return name.getSimple().toLowerCase();
+  public void execute(CalcitePrepare.Context context) {
+final List path = context.getDefaultSchemaPath();
+CalciteSchema schema = context.getRootSchema();
+for (String p : path) {
+  schema = schema.getSubSchema(p, true);
+}
+final boolean existed;
+switch (getKind()) {
+case DROP_TABLE:
+case DROP_MATERIALIZED_VIEW:
 
 Review comment:
   Do we support views? If we don't have concrete plans to support them i'd 
rather remove all related code


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97992)

> Take advantage of Calcite DDL
> -
>
> Key: BEAM-4044
> URL: https://issues.apache.org/jira/browse/BEAM-4044
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 10h 20m
>  Remaining Estimate: 0h
>
> In Calcite 1.15 support for abstract DDL moved into calcite core. We should 
> take advantage of that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4044) Take advantage of Calcite DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4044?focusedWorklogId=97997=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97997
 ]

ASF GitHub Bot logged work on BEAM-4044:


Author: ASF GitHub Bot
Created on: 03/May/18 18:16
Start Date: 03/May/18 18:16
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #5224: 
[BEAM-4044] [SQL] Add tables via TableStore in Schema, execute DDL in Calcite 
model
URL: https://github.com/apache/beam/pull/5224#discussion_r185890896
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteSchema.java
 ##
 @@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
+import org.apache.calcite.linq4j.tree.Expression;
+import org.apache.calcite.schema.Function;
+import org.apache.calcite.schema.Schema;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.SchemaVersion;
+import org.apache.calcite.schema.Schemas;
+
+/**
+ * Adapter from {@link TableProvider} to {@link Schema}.
+ */
+public class BeamCalciteSchema implements Schema {
+  private TableProvider tableProvider;
+
+  public BeamCalciteSchema(TableProvider tableProvider) {
+this.tableProvider = tableProvider;
+  }
+
+  public TableProvider getTableProvider() {
+return tableProvider;
+  }
+
+  @Override
+  public boolean isMutable() {
+return true;
+  }
+
+  @Override
+  public Schema snapshot(SchemaVersion version) {
+return this;
+  }
+
+  @Override
+  public Expression getExpression(SchemaPlus parentSchema, String name) {
+return Schemas.subSchemaExpression(parentSchema, name, getClass());
+  }
+
+  @Override
+  public Set getTableNames() {
+Set tables = new HashSet();
+for (Table table : tableProvider.listTables()) {
+  tables.add(table.getName());
+}
+return tables;
+  }
+
+  @Override
+  public org.apache.calcite.schema.Table getTable(String name) {
+for (Table table : tableProvider.listTables()) {
+  if (table.getName().equals(name)) {
+return new BeamCalciteTable(tableProvider.buildBeamSqlTable(table));
+  }
+}
 
 Review comment:
   nit: looks like it would be better to convert this to a `Map` once in constructor, this way you wouldn't need to implement 
`map.keySet()` or `map.get()`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97997)
Time Spent: 11h  (was: 10h 50m)

> Take advantage of Calcite DDL
> -
>
> Key: BEAM-4044
> URL: https://issues.apache.org/jira/browse/BEAM-4044
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 11h
>  Remaining Estimate: 0h
>
> In Calcite 1.15 support for abstract DDL moved into calcite core. We should 
> take advantage of that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4044) Take advantage of Calcite DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4044?focusedWorklogId=97993=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97993
 ]

ASF GitHub Bot logged work on BEAM-4044:


Author: ASF GitHub Bot
Created on: 03/May/18 18:16
Start Date: 03/May/18 18:16
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #5224: 
[BEAM-4044] [SQL] Add tables via TableStore in Schema, execute DDL in Calcite 
model
URL: https://github.com/apache/beam/pull/5224#discussion_r185888542
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/QueryTransform.java
 ##
 @@ -78,25 +76,18 @@
 }
   }
 
-  private PCollectionTuple toPCollectionTuple(PInput inputs) {
-return (inputs instanceof PCollection)
-? PCollectionTuple.of(new TupleTag<>(PCOLLECTION_NAME), toRows(inputs))
-: tupleOfAllInputs(inputs.getPipeline(), inputs.expand());
-  }
-
-  private PCollectionTuple tupleOfAllInputs(
-  Pipeline pipeline,
-  Map taggedInputs) {
-
-PCollectionTuple tuple = PCollectionTuple.empty(pipeline);
-
-for (Map.Entry input : taggedInputs.entrySet()) {
-  tuple = tuple.and(
-  new TupleTag<>(input.getKey().getId()),
-  toRows(input.getValue()));
+  private BeamSqlTableProvider toTableProvider(PInput inputs) {
+ImmutableMap.Builder tables = ImmutableMap.builder();
+if (inputs instanceof PCollection) {
+  tables.put(PCOLLECTION_NAME,
+  new BeamPCollectionTable(toRows(inputs)));
+} else {
+  for (Map.Entry input : inputs.expand().entrySet()) {
+tables.put(input.getKey().getId(),
+new BeamPCollectionTable(toRows(input.getValue(;
+  }
 
 Review comment:
   nit: I would avoid stateful if/else with loops with generics, hurts 
readability. Might consider extracting something like this:
   
   ```java
   if (input instanceof PCollection) {
 return 
ImmuableMap.of(
   PCOLLECTION_NAME, 
   new BeamPCollectionTable(toRows(inputs)))
   }
   
   return
   inputs
   .expand()
   .entrySet()
   .stream()
   .collect(
   toMap(
   keyedPCollection -> keyedPCollection.getKey().getId(),
   keyedPCollection -> keyedPCollection.getValue()))
   ```
   
   and then create BeamSqlTableProvider outside


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 97993)
Time Spent: 10.5h  (was: 10h 20m)

> Take advantage of Calcite DDL
> -
>
> Key: BEAM-4044
> URL: https://issues.apache.org/jira/browse/BEAM-4044
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 10.5h
>  Remaining Estimate: 0h
>
> In Calcite 1.15 support for abstract DDL moved into calcite core. We should 
> take advantage of that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)