[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[ https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=228945&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228945 ] ASF GitHub Bot logged work on BEAM-7039: Author: ASF GitHub Bot Created on: 17/Apr/19 08:58 Start Date: 17/Apr/19 08:58 Worklog Time Spent: 10m Work Description: mxm commented on pull request #8285: [BEAM-7039] set up validatesPortableRunner tests for Spark URL: https://github.com/apache/beam/pull/8285 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228945) Time Spent: 3.5h (was: 3h 20m) > Spark portable runner: run validatesRunner tests > > > Key: BEAM-7039 > URL: https://issues.apache.org/jira/browse/BEAM-7039 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Fix For: 2.13.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[ https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=228943&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228943 ] ASF GitHub Bot logged work on BEAM-7039: Author: ASF GitHub Bot Created on: 17/Apr/19 08:58 Start Date: 17/Apr/19 08:58 Worklog Time Spent: 10m Work Description: mxm commented on issue #8285: [BEAM-7039] set up validatesPortableRunner tests for Spark URL: https://github.com/apache/beam/pull/8285#issuecomment-483999047 Thanks! Website test failures unrelated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228943) Time Spent: 3h 20m (was: 3h 10m) > Spark portable runner: run validatesRunner tests > > > Key: BEAM-7039 > URL: https://issues.apache.org/jira/browse/BEAM-7039 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Fix For: 2.13.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[ https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=228722&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228722 ] ASF GitHub Bot logged work on BEAM-7039: Author: ASF GitHub Bot Created on: 16/Apr/19 22:22 Start Date: 16/Apr/19 22:22 Worklog Time Spent: 10m Work Description: ibzib commented on issue #8285: [BEAM-7039] set up validatesPortableRunner tests for Spark URL: https://github.com/apache/beam/pull/8285#issuecomment-483866092 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228722) Time Spent: 3h 10m (was: 3h) > Spark portable runner: run validatesRunner tests > > > Key: BEAM-7039 > URL: https://issues.apache.org/jira/browse/BEAM-7039 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[ https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=228601&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228601 ] ASF GitHub Bot logged work on BEAM-7039: Author: ASF GitHub Bot Created on: 16/Apr/19 18:26 Start Date: 16/Apr/19 18:26 Worklog Time Spent: 10m Work Description: ibzib commented on issue #8285: [BEAM-7039] set up validatesPortableRunner tests for Spark URL: https://github.com/apache/beam/pull/8285#issuecomment-483790177 > Needs to be rebased now due to a merge conflict. resolved This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228601) Time Spent: 3h (was: 2h 50m) > Spark portable runner: run validatesRunner tests > > > Key: BEAM-7039 > URL: https://issues.apache.org/jira/browse/BEAM-7039 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[ https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=228265&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228265 ] ASF GitHub Bot logged work on BEAM-7039: Author: ASF GitHub Bot Created on: 16/Apr/19 09:48 Start Date: 16/Apr/19 09:48 Worklog Time Spent: 10m Work Description: mxm commented on issue #8285: [BEAM-7039] set up validatesPortableRunner tests for Spark URL: https://github.com/apache/beam/pull/8285#issuecomment-483590658 Needs to be rebased now due to a merge conflict. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228265) Time Spent: 2h 50m (was: 2h 40m) > Spark portable runner: run validatesRunner tests > > > Key: BEAM-7039 > URL: https://issues.apache.org/jira/browse/BEAM-7039 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[ https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=228035&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228035 ] ASF GitHub Bot logged work on BEAM-7039: Author: ASF GitHub Bot Created on: 15/Apr/19 23:04 Start Date: 15/Apr/19 23:04 Worklog Time Spent: 10m Work Description: ibzib commented on issue #8285: [BEAM-7039] set up validatesPortableRunner tests for Spark URL: https://github.com/apache/beam/pull/8285#issuecomment-483450682 Run Website PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228035) Time Spent: 2h 40m (was: 2.5h) > Spark portable runner: run validatesRunner tests > > > Key: BEAM-7039 > URL: https://issues.apache.org/jira/browse/BEAM-7039 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[ https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=228008&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228008 ] ASF GitHub Bot logged work on BEAM-7039: Author: ASF GitHub Bot Created on: 15/Apr/19 22:31 Start Date: 15/Apr/19 22:31 Worklog Time Spent: 10m Work Description: ibzib commented on issue #8285: [BEAM-7039] set up validatesPortableRunner tests for Spark URL: https://github.com/apache/beam/pull/8285#issuecomment-483442862 Run Website PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228008) Time Spent: 2.5h (was: 2h 20m) > Spark portable runner: run validatesRunner tests > > > Key: BEAM-7039 > URL: https://issues.apache.org/jira/browse/BEAM-7039 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[ https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=228007&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228007 ] ASF GitHub Bot logged work on BEAM-7039: Author: ASF GitHub Bot Created on: 15/Apr/19 22:31 Start Date: 15/Apr/19 22:31 Worklog Time Spent: 10m Work Description: ibzib commented on issue #8285: [BEAM-7039] set up validatesPortableRunner tests for Spark URL: https://github.com/apache/beam/pull/8285#issuecomment-483442812 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228007) Time Spent: 2h 20m (was: 2h 10m) > Spark portable runner: run validatesRunner tests > > > Key: BEAM-7039 > URL: https://issues.apache.org/jira/browse/BEAM-7039 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[
https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=227833&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227833
]
ASF GitHub Bot logged work on BEAM-7039:
Author: ASF GitHub Bot
Created on: 15/Apr/19 16:58
Start Date: 15/Apr/19 16:58
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #8285: [BEAM-7039] set
up validatesPortableRunner tests for Spark
URL: https://github.com/apache/beam/pull/8285#discussion_r275455079
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1559,6 +1561,7 @@ class BeamModulePlugin implements Plugin {
project.tasks.create(name: name, type: Test) {
group = "Verification"
description = "Validates the PortableRunner with JobServer
${config.jobServerDriver}"
+systemProperties config.systemProperties
systemProperty "beamTestPipelineOptions",
JsonOutput.toJson(beamTestPipelineOptions)
Review comment:
updated
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 227833)
Time Spent: 2h 10m (was: 2h)
> Spark portable runner: run validatesRunner tests
>
>
> Key: BEAM-7039
> URL: https://issues.apache.org/jira/browse/BEAM-7039
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[
https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=227537&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227537
]
ASF GitHub Bot logged work on BEAM-7039:
Author: ASF GitHub Bot
Created on: 15/Apr/19 09:07
Start Date: 15/Apr/19 09:07
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #8285: [BEAM-7039] set up
validatesPortableRunner tests for Spark
URL: https://github.com/apache/beam/pull/8285#discussion_r275267941
##
File path: runners/spark/job-server/build.gradle
##
@@ -0,0 +1,122 @@
+import org.apache.beam.gradle.BeamModulePlugin
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * Spark Runner JobServer build file
+ */
+
+apply plugin: 'org.apache.beam.module'
+apply plugin: 'application'
+// we need to set mainClassName before applying shadow plugin
+mainClassName = "org.apache.beam.runners.spark.SparkJobServerDriver"
+
+applyJavaNature(
+ validateShadowJar: false,
+ exportJavadoc: false,
+ shadowClosure: {
+append "reference.conf"
+ },
+)
+
+def sparkRunnerProject = ":${project.name.replace("-job-server", "")}"
+
+description = project(sparkRunnerProject).description + " :: Job Server"
+
+configurations {
+ validatesPortableRunner
+}
+
+configurations.all {
+ exclude group: "org.slf4j", module: "slf4j-jdk14"
+}
+
+dependencies {
+ compile project(path: sparkRunnerProject, configuration: "shadow")
+ compile project(path: sparkRunnerProject, configuration: "provided")
+ validatesPortableRunner project(path: sparkRunnerProject, configuration:
"shadowTest")
+ validatesPortableRunner project(path: sparkRunnerProject, configuration:
"provided")
+ validatesPortableRunner project(path: ":beam-sdks-java-core", configuration:
"shadowTest")
+ validatesPortableRunner project(path: ":beam-runners-core-java",
configuration: "shadowTest")
+ validatesPortableRunner project(path: ":beam-runners-reference-java",
configuration: "shadowTest")
+ compile project(path:
":beam-sdks-java-extensions-google-cloud-platform-core", configuration:
"shadow")
+// TODO: Enable AWS and HDFS file system.
+}
+
+// NOTE: runShadow must be used in order to run the job server. The standard
run
+// task will not work because the Spark runner classes only exist in the shadow
+// jar.
+runShadow {
+ args = []
+ if (project.hasProperty('jobHost'))
+args += ["--job-host=${project.property('jobHost')}"]
+ if (project.hasProperty('artifactsDir'))
+args += ["--artifacts-dir=${project.property('artifactsDir')}"]
+ if (project.hasProperty('cleanArtifactsPerJob'))
+args += ["--clean-artifacts-per-job"]
Review comment:
Thanks for following up with this!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 227537)
Time Spent: 2h (was: 1h 50m)
> Spark portable runner: run validatesRunner tests
>
>
> Key: BEAM-7039
> URL: https://issues.apache.org/jira/browse/BEAM-7039
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[
https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=227093&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227093
]
ASF GitHub Bot logged work on BEAM-7039:
Author: ASF GitHub Bot
Created on: 13/Apr/19 01:57
Start Date: 13/Apr/19 01:57
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #8285: [BEAM-7039]
set up validatesPortableRunner tests for Spark
URL: https://github.com/apache/beam/pull/8285#discussion_r275100838
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1559,6 +1561,7 @@ class BeamModulePlugin implements Plugin {
project.tasks.create(name: name, type: Test) {
group = "Verification"
description = "Validates the PortableRunner with JobServer
${config.jobServerDriver}"
+systemProperties config.systemProperties
systemProperty "beamTestPipelineOptions",
JsonOutput.toJson(beamTestPipelineOptions)
Review comment:
Shall we also merge this system property to systemProperties set above,
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 227093)
Time Spent: 1h 50m (was: 1h 40m)
> Spark portable runner: run validatesRunner tests
>
>
> Key: BEAM-7039
> URL: https://issues.apache.org/jira/browse/BEAM-7039
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[ https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=226858&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226858 ] ASF GitHub Bot logged work on BEAM-7039: Author: ASF GitHub Bot Created on: 12/Apr/19 19:06 Start Date: 12/Apr/19 19:06 Worklog Time Spent: 10m Work Description: ibzib commented on issue #8285: [BEAM-7039] set up validatesPortableRunner tests for Spark URL: https://github.com/apache/beam/pull/8285#issuecomment-482689060 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 226858) Time Spent: 1h 40m (was: 1.5h) > Spark portable runner: run validatesRunner tests > > > Key: BEAM-7039 > URL: https://issues.apache.org/jira/browse/BEAM-7039 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[ https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=226840&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226840 ] ASF GitHub Bot logged work on BEAM-7039: Author: ASF GitHub Bot Created on: 12/Apr/19 18:35 Start Date: 12/Apr/19 18:35 Worklog Time Spent: 10m Work Description: ibzib commented on issue #8285: [BEAM-7039] set up validatesPortableRunner tests for Spark URL: https://github.com/apache/beam/pull/8285#issuecomment-482679460 Run Website PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 226840) Time Spent: 1.5h (was: 1h 20m) > Spark portable runner: run validatesRunner tests > > > Key: BEAM-7039 > URL: https://issues.apache.org/jira/browse/BEAM-7039 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[
https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=226831&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226831
]
ASF GitHub Bot logged work on BEAM-7039:
Author: ASF GitHub Bot
Created on: 12/Apr/19 18:04
Start Date: 12/Apr/19 18:04
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #8285: [BEAM-7039] set
up validatesPortableRunner tests for Spark
URL: https://github.com/apache/beam/pull/8285#discussion_r275010043
##
File path: runners/spark/job-server/build.gradle
##
@@ -0,0 +1,122 @@
+import org.apache.beam.gradle.BeamModulePlugin
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * Spark Runner JobServer build file
+ */
+
+apply plugin: 'org.apache.beam.module'
+apply plugin: 'application'
+// we need to set mainClassName before applying shadow plugin
+mainClassName = "org.apache.beam.runners.spark.SparkJobServerDriver"
+
+applyJavaNature(
+ validateShadowJar: false,
+ exportJavadoc: false,
+ shadowClosure: {
+append "reference.conf"
+ },
+)
+
+def sparkRunnerProject = ":${project.name.replace("-job-server", "")}"
+
+description = project(sparkRunnerProject).description + " :: Job Server"
+
+configurations {
+ validatesPortableRunner
+}
+
+configurations.all {
+ exclude group: "org.slf4j", module: "slf4j-jdk14"
+}
+
+dependencies {
+ compile project(path: sparkRunnerProject, configuration: "shadow")
+ compile project(path: sparkRunnerProject, configuration: "provided")
+ validatesPortableRunner project(path: sparkRunnerProject, configuration:
"shadowTest")
+ validatesPortableRunner project(path: sparkRunnerProject, configuration:
"provided")
+ validatesPortableRunner project(path: ":beam-sdks-java-core", configuration:
"shadowTest")
+ validatesPortableRunner project(path: ":beam-runners-core-java",
configuration: "shadowTest")
+ validatesPortableRunner project(path: ":beam-runners-reference-java",
configuration: "shadowTest")
+ compile project(path:
":beam-sdks-java-extensions-google-cloud-platform-core", configuration:
"shadow")
+// TODO: Enable AWS and HDFS file system.
+}
+
+// NOTE: runShadow must be used in order to run the job server. The standard
run
+// task will not work because the Spark runner classes only exist in the shadow
+// jar.
+runShadow {
+ args = []
+ if (project.hasProperty('jobHost'))
+args += ["--job-host=${project.property('jobHost')}"]
+ if (project.hasProperty('artifactsDir'))
+args += ["--artifacts-dir=${project.property('artifactsDir')}"]
+ if (project.hasProperty('cleanArtifactsPerJob'))
+args += ["--clean-artifacts-per-job"]
Review comment:
I made a PR to fix this for Flink as well. #8293
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 226831)
Time Spent: 1h 20m (was: 1h 10m)
> Spark portable runner: run validatesRunner tests
>
>
> Key: BEAM-7039
> URL: https://issues.apache.org/jira/browse/BEAM-7039
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[
https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=226825&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226825
]
ASF GitHub Bot logged work on BEAM-7039:
Author: ASF GitHub Bot
Created on: 12/Apr/19 17:53
Start Date: 12/Apr/19 17:53
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #8285: [BEAM-7039] set
up validatesPortableRunner tests for Spark
URL: https://github.com/apache/beam/pull/8285#discussion_r275006433
##
File path: runners/spark/job-server/build.gradle
##
@@ -0,0 +1,122 @@
+import org.apache.beam.gradle.BeamModulePlugin
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * Spark Runner JobServer build file
+ */
+
+apply plugin: 'org.apache.beam.module'
+apply plugin: 'application'
+// we need to set mainClassName before applying shadow plugin
+mainClassName = "org.apache.beam.runners.spark.SparkJobServerDriver"
+
+applyJavaNature(
+ validateShadowJar: false,
+ exportJavadoc: false,
+ shadowClosure: {
+append "reference.conf"
+ },
+)
+
+def sparkRunnerProject = ":${project.name.replace("-job-server", "")}"
+
+description = project(sparkRunnerProject).description + " :: Job Server"
+
+configurations {
+ validatesPortableRunner
+}
+
+configurations.all {
+ exclude group: "org.slf4j", module: "slf4j-jdk14"
+}
+
+dependencies {
+ compile project(path: sparkRunnerProject, configuration: "shadow")
+ compile project(path: sparkRunnerProject, configuration: "provided")
+ validatesPortableRunner project(path: sparkRunnerProject, configuration:
"shadowTest")
+ validatesPortableRunner project(path: sparkRunnerProject, configuration:
"provided")
+ validatesPortableRunner project(path: ":beam-sdks-java-core", configuration:
"shadowTest")
+ validatesPortableRunner project(path: ":beam-runners-core-java",
configuration: "shadowTest")
+ validatesPortableRunner project(path: ":beam-runners-reference-java",
configuration: "shadowTest")
+ compile project(path:
":beam-sdks-java-extensions-google-cloud-platform-core", configuration:
"shadow")
+// TODO: Enable AWS and HDFS file system.
+}
+
+// NOTE: runShadow must be used in order to run the job server. The standard
run
+// task will not work because the Spark runner classes only exist in the shadow
+// jar.
+runShadow {
+ args = []
+ if (project.hasProperty('jobHost'))
+args += ["--job-host=${project.property('jobHost')}"]
+ if (project.hasProperty('artifactsDir'))
+args += ["--artifacts-dir=${project.property('artifactsDir')}"]
+ if (project.hasProperty('cleanArtifactsPerJob'))
+args += ["--clean-artifacts-per-job"]
Review comment:
fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 226825)
Time Spent: 1h 10m (was: 1h)
> Spark portable runner: run validatesRunner tests
>
>
> Key: BEAM-7039
> URL: https://issues.apache.org/jira/browse/BEAM-7039
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[
https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=226592&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226592
]
ASF GitHub Bot logged work on BEAM-7039:
Author: ASF GitHub Bot
Created on: 12/Apr/19 11:45
Start Date: 12/Apr/19 11:45
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #8285: [BEAM-7039] set up
validatesPortableRunner tests for Spark
URL: https://github.com/apache/beam/pull/8285#discussion_r274869862
##
File path: runners/spark/job-server/build.gradle
##
@@ -0,0 +1,122 @@
+import org.apache.beam.gradle.BeamModulePlugin
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * Spark Runner JobServer build file
+ */
+
+apply plugin: 'org.apache.beam.module'
+apply plugin: 'application'
+// we need to set mainClassName before applying shadow plugin
+mainClassName = "org.apache.beam.runners.spark.SparkJobServerDriver"
+
+applyJavaNature(
+ validateShadowJar: false,
+ exportJavadoc: false,
+ shadowClosure: {
+append "reference.conf"
+ },
+)
+
+def sparkRunnerProject = ":${project.name.replace("-job-server", "")}"
+
+description = project(sparkRunnerProject).description + " :: Job Server"
+
+configurations {
+ validatesPortableRunner
+}
+
+configurations.all {
+ exclude group: "org.slf4j", module: "slf4j-jdk14"
+}
+
+dependencies {
+ compile project(path: sparkRunnerProject, configuration: "shadow")
+ compile project(path: sparkRunnerProject, configuration: "provided")
+ validatesPortableRunner project(path: sparkRunnerProject, configuration:
"shadowTest")
+ validatesPortableRunner project(path: sparkRunnerProject, configuration:
"provided")
+ validatesPortableRunner project(path: ":beam-sdks-java-core", configuration:
"shadowTest")
+ validatesPortableRunner project(path: ":beam-runners-core-java",
configuration: "shadowTest")
+ validatesPortableRunner project(path: ":beam-runners-reference-java",
configuration: "shadowTest")
+ compile project(path:
":beam-sdks-java-extensions-google-cloud-platform-core", configuration:
"shadow")
+// TODO: Enable AWS and HDFS file system.
+}
+
+// NOTE: runShadow must be used in order to run the job server. The standard
run
+// task will not work because the Spark runner classes only exist in the shadow
+// jar.
+runShadow {
+ args = []
+ if (project.hasProperty('jobHost'))
+args += ["--job-host=${project.property('jobHost')}"]
+ if (project.hasProperty('artifactsDir'))
+args += ["--artifacts-dir=${project.property('artifactsDir')}"]
+ if (project.hasProperty('cleanArtifactsPerJob'))
+args += ["--clean-artifacts-per-job"]
Review comment:
This is actually outdated. As of #8210, this is always true. We should
parameterize the option to allow it to be set to false/true. I think this also
needs to be fixed for Flink.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 226592)
Time Spent: 50m (was: 40m)
> Spark portable runner: run validatesRunner tests
>
>
> Key: BEAM-7039
> URL: https://issues.apache.org/jira/browse/BEAM-7039
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[
https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=226593&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226593
]
ASF GitHub Bot logged work on BEAM-7039:
Author: ASF GitHub Bot
Created on: 12/Apr/19 11:45
Start Date: 12/Apr/19 11:45
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #8285: [BEAM-7039] set up
validatesPortableRunner tests for Spark
URL: https://github.com/apache/beam/pull/8285#discussion_r274870335
##
File path: runners/spark/job-server/build.gradle
##
@@ -0,0 +1,122 @@
+import org.apache.beam.gradle.BeamModulePlugin
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * Spark Runner JobServer build file
+ */
+
+apply plugin: 'org.apache.beam.module'
+apply plugin: 'application'
+// we need to set mainClassName before applying shadow plugin
+mainClassName = "org.apache.beam.runners.spark.SparkJobServerDriver"
+
+applyJavaNature(
+ validateShadowJar: false,
+ exportJavadoc: false,
+ shadowClosure: {
+append "reference.conf"
+ },
+)
+
+def sparkRunnerProject = ":${project.name.replace("-job-server", "")}"
+
+description = project(sparkRunnerProject).description + " :: Job Server"
+
+configurations {
+ validatesPortableRunner
+}
+
+configurations.all {
+ exclude group: "org.slf4j", module: "slf4j-jdk14"
+}
+
+dependencies {
+ compile project(path: sparkRunnerProject, configuration: "shadow")
+ compile project(path: sparkRunnerProject, configuration: "provided")
+ validatesPortableRunner project(path: sparkRunnerProject, configuration:
"shadowTest")
+ validatesPortableRunner project(path: sparkRunnerProject, configuration:
"provided")
+ validatesPortableRunner project(path: ":beam-sdks-java-core", configuration:
"shadowTest")
+ validatesPortableRunner project(path: ":beam-runners-core-java",
configuration: "shadowTest")
+ validatesPortableRunner project(path: ":beam-runners-reference-java",
configuration: "shadowTest")
+ compile project(path:
":beam-sdks-java-extensions-google-cloud-platform-core", configuration:
"shadow")
+// TODO: Enable AWS and HDFS file system.
+}
+
+// NOTE: runShadow must be used in order to run the job server. The standard
run
+// task will not work because the Spark runner classes only exist in the shadow
+// jar.
+runShadow {
+ args = []
+ if (project.hasProperty('jobHost'))
+args += ["--job-host=${project.property('jobHost')}"]
+ if (project.hasProperty('artifactsDir'))
+args += ["--artifacts-dir=${project.property('artifactsDir')}"]
+ if (project.hasProperty('cleanArtifactsPerJob'))
+args += ["--clean-artifacts-per-job"]
+
+ // Enable remote debugging.
+ jvmArgs = ["-Xdebug",
"-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"]
+ if (project.hasProperty("logLevel"))
+jvmArgs +=
["-Dorg.slf4j.simpleLogger.defaultLogLevel=${project.property('logLevel')}"]
+}
+
+def portableValidatesRunnerTask(String name) {
+ createPortableValidatesRunnerTask(
+name: "validatesPortableRunner${name}",
+jobServerDriver: "org.apache.beam.runners.spark.SparkJobServerDriver",
+jobServerConfig:
"--clean-artifacts-per-job,--job-host=localhost,--job-port=0,--artifact-port=0",
Review comment:
We can remove the clean artifacts option here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 226593)
Time Spent: 1h (was: 50m)
> Spark portable runner: run validatesRunner tests
>
>
> Key: BEAM-7039
> URL: https://issues.apache.org/jira/browse/BEAM-7039
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Majo
[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[ https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=226437&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226437 ] ASF GitHub Bot logged work on BEAM-7039: Author: ASF GitHub Bot Created on: 12/Apr/19 00:39 Start Date: 12/Apr/19 00:39 Worklog Time Spent: 10m Work Description: ibzib commented on issue #8285: [BEAM-7039] set up validatesPortableRunner tests for Spark URL: https://github.com/apache/beam/pull/8285#issuecomment-482391620 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 226437) Time Spent: 40m (was: 0.5h) > Spark portable runner: run validatesRunner tests > > > Key: BEAM-7039 > URL: https://issues.apache.org/jira/browse/BEAM-7039 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[ https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=226414&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226414 ] ASF GitHub Bot logged work on BEAM-7039: Author: ASF GitHub Bot Created on: 11/Apr/19 23:34 Start Date: 11/Apr/19 23:34 Worklog Time Spent: 10m Work Description: ibzib commented on issue #8285: [BEAM-7039] set up validatesPortableRunner tests for Spark URL: https://github.com/apache/beam/pull/8285#issuecomment-482372719 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 226414) Time Spent: 20m (was: 10m) > Spark portable runner: run validatesRunner tests > > > Key: BEAM-7039 > URL: https://issues.apache.org/jira/browse/BEAM-7039 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[ https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=226416&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226416 ] ASF GitHub Bot logged work on BEAM-7039: Author: ASF GitHub Bot Created on: 11/Apr/19 23:34 Start Date: 11/Apr/19 23:34 Worklog Time Spent: 10m Work Description: ibzib commented on issue #8285: [BEAM-7039] set up validatesPortableRunner tests for Spark URL: https://github.com/apache/beam/pull/8285#issuecomment-482372845 Run Java_Examples_Dataflow PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 226416) Time Spent: 0.5h (was: 20m) > Spark portable runner: run validatesRunner tests > > > Key: BEAM-7039 > URL: https://issues.apache.org/jira/browse/BEAM-7039 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7039) Spark portable runner: run validatesRunner tests
[ https://issues.apache.org/jira/browse/BEAM-7039?focusedWorklogId=226391&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226391 ] ASF GitHub Bot logged work on BEAM-7039: Author: ASF GitHub Bot Created on: 11/Apr/19 22:48 Start Date: 11/Apr/19 22:48 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #8285: [BEAM-7039] set up validatesPortableRunner tests for Spark URL: https://github.com/apache/beam/pull/8285 This adds a couple useful Gradle commands for the Spark portable runner: 1. `:beam-runners-spark-job-server:runShadow` 2. `:beam-runners-spark-job-server:validatesPortableRunner` At this commit, most tests fail for the latter because PAssert is not yet supported (pending Metrics & Flatten). Disclaimer: I am not too familiar with Gradle/Groovy, so please make extra sure I didn't make any rookie mistakes :) R: @mxm Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | --- Pre-Commit Tests Status (on master branch) --- |Java | Python | Go | Website --- | --- | --- | --- | --- Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://build
