[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=174381=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-174381
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 12/Dec/18 01:49
Start Date: 12/Dec/18 01:49
Worklog Time Spent: 10m
Work Description: tweise closed pull request #7246: [BEAM-5058] remove
unused envdir parameter
URL: https://github.com/apache/beam/pull/7246
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index 4b0036824f37..a17b032a0a48 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -112,8 +112,8 @@ task preCommit() {
task portablePreCommit() {
dependsOn ':beam-runners-flink_2.11-job-server-container:docker'
dependsOn ':beam-sdks-python-container:docker'
- dependsOn portableWordCountTask('portableWordCountBatch', false, envdir)
- dependsOn portableWordCountTask('portableWordCountStreaming', true, envdir)
+ dependsOn portableWordCountTask('portableWordCountBatch', false)
+ dependsOn portableWordCountTask('portableWordCountStreaming', true)
}
@@ -207,10 +207,10 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
//./gradlew :beam-sdks-python:portableWordCount
-PjobEndpoint=localhost:8099
//
task portableWordCount {
- dependsOn portableWordCountTask('portableWordCountExample',
project.hasProperty("streaming"), envdir)
+ dependsOn portableWordCountTask('portableWordCountExample',
project.hasProperty("streaming"))
}
-def portableWordCountTask(name, streaming, envdir) {
+def portableWordCountTask(name, streaming) {
tasks.create(name) {
dependsOn = ['installGcpTest']
mustRunAfter = [':beam-runners-flink_2.11-job-server-container:docker',
':beam-sdks-python-container:docker']
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 174381)
Time Spent: 6h 50m (was: 6h 40m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=174369=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-174369
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 12/Dec/18 00:59
Start Date: 12/Dec/18 00:59
Worklog Time Spent: 10m
Work Description: tweise commented on a change in pull request #7246:
[BEAM-5058] remove unused envdir parameter
URL: https://github.com/apache/beam/pull/7246#discussion_r240849570
##
File path: sdks/python/build.gradle
##
@@ -207,10 +207,10 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
//./gradlew :beam-sdks-python:portableWordCount
-PjobEndpoint=localhost:8099
//
task portableWordCount {
- dependsOn portableWordCountTask('portableWordCountExample',
project.hasProperty("streaming"), envdir)
Review comment:
It's no longer used and should have been removed in the referenced PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 174369)
Time Spent: 6h 40m (was: 6.5h)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=174368=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-174368
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 12/Dec/18 00:58
Start Date: 12/Dec/18 00:58
Worklog Time Spent: 10m
Work Description: tweise commented on issue #7246: [BEAM-5058] remove
unused envdir parameter
URL: https://github.com/apache/beam/pull/7246#issuecomment-446421171
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 174368)
Time Spent: 6.5h (was: 6h 20m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=174347=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-174347
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 11/Dec/18 23:32
Start Date: 11/Dec/18 23:32
Worklog Time Spent: 10m
Work Description: udim commented on a change in pull request #7246:
[BEAM-5058] remove unused envdir parameter
URL: https://github.com/apache/beam/pull/7246#discussion_r240832311
##
File path: sdks/python/build.gradle
##
@@ -207,10 +207,10 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
//./gradlew :beam-sdks-python:portableWordCount
-PjobEndpoint=localhost:8099
//
task portableWordCount {
- dependsOn portableWordCountTask('portableWordCountExample',
project.hasProperty("streaming"), envdir)
Review comment:
I approved it, but I just don't understand if Gradle is ignoring and
undefined `envdir` or if it's being defined elsewhere, since #7163 removed its
definition.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 174347)
Time Spent: 6h 20m (was: 6h 10m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=174305=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-174305
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 11/Dec/18 21:51
Start Date: 11/Dec/18 21:51
Worklog Time Spent: 10m
Work Description: tweise commented on a change in pull request #7246:
[BEAM-5058] remove unused envdir parameter
URL: https://github.com/apache/beam/pull/7246#discussion_r240804705
##
File path: sdks/python/build.gradle
##
@@ -207,10 +207,10 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
//./gradlew :beam-sdks-python:portableWordCount
-PjobEndpoint=localhost:8099
//
task portableWordCount {
- dependsOn portableWordCountTask('portableWordCountExample',
project.hasProperty("streaming"), envdir)
Review comment:
This is actually cleanup related to #7163 which you had reviewed. Maybe
@markflyhigh can approve the change?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 174305)
Time Spent: 6h 10m (was: 6h)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=174288=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-174288
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 11/Dec/18 21:06
Start Date: 11/Dec/18 21:06
Worklog Time Spent: 10m
Work Description: udim commented on a change in pull request #7246:
[BEAM-5058] remove unused envdir parameter
URL: https://github.com/apache/beam/pull/7246#discussion_r240790791
##
File path: sdks/python/build.gradle
##
@@ -207,10 +207,10 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
//./gradlew :beam-sdks-python:portableWordCount
-PjobEndpoint=localhost:8099
//
task portableWordCount {
- dependsOn portableWordCountTask('portableWordCountExample',
project.hasProperty("streaming"), envdir)
Review comment:
Where is envdir defined? Or was this broken?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 174288)
Time Spent: 6h (was: 5h 50m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 6h
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[ https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=173879=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-173879 ] ASF GitHub Bot logged work on BEAM-5058: Author: ASF GitHub Bot Created on: 11/Dec/18 03:05 Start Date: 11/Dec/18 03:05 Worklog Time Spent: 10m Work Description: tweise opened a new pull request #7246: [BEAM-5058] remove unused envdir parameter URL: https://github.com/apache/beam/pull/7246 **Please** add a meaningful description for your change here Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 173879) Time Spent: 5h 50m (was: 5h 40m) > Python precommits should run E2E tests > -- > > Key: BEAM-5058 > URL:
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=171777=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171777
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 04/Dec/18 01:02
Start Date: 04/Dec/18 01:02
Worklog Time Spent: 10m
Work Description: aaltay closed pull request #7163: [BEAM-5058] Run basic
ITs in Python Precommit in parallel
URL: https://github.com/apache/beam/pull/7163
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/build.gradle b/build.gradle
index a74817f70e8d..fc11ecc390a2 100644
--- a/build.gradle
+++ b/build.gradle
@@ -240,6 +240,7 @@ task goIntegrationTests() {
task pythonPreCommit() {
dependsOn ":beam-sdks-python:preCommit"
+ dependsOn ":beam-sdks-python-precommit-dataflow:precommitIT"
}
task pythonPostCommit() {
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 63c6729c2d61..329a6edbb3ec 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -1488,5 +1488,81 @@ artifactId=${project.name}
dependsOn ':beam-sdks-java-container:docker'
}
}
+
+/**
***/
+
+project.ext.applyPythonNature = {
+
+ // Define common lifecycle tasks and artifact types
+ project.apply plugin: "base"
+
+ // For some reason base doesn't define a test task so we define it
below and make
+ // check depend on it. This makes the Python project similar to the task
layout like
+ // Java projects, see
https://docs.gradle.org/4.2.1/userguide/img/javaPluginTasks.png
+ project.task('test', type: Test) {}
+ project.check.dependsOn project.test
+
+
project.evaluationDependsOn(":beam-runners-google-cloud-dataflow-java-fn-api-worker")
+
+ // Due to Beam-4256, we need to limit the length of virtualenv path to
make the
+ // virtualenv activated properly. So instead of include project name in
the path,
+ // we use the hash value.
+ project.ext.envdir =
"${project.rootProject.buildDir}/gradleenv/${project.name.hashCode()}"
+ project.ext.pythonRootDir = "${project.rootDir}/sdks/python"
+
+ project.task('setupVirtualenv') {
+doLast {
+ project.exec { commandLine 'virtualenv', "${project.ext.envdir}" }
+ project.exec {
+executable 'sh'
+args '-c', ". ${project.ext.envdir}/bin/activate && pip install
--upgrade tox==3.0.0 grpcio-tools==1.3.5"
+ }
+}
+// Gradle will delete outputs whenever it thinks they are stale.
Putting a
+// specific binary here could make gradle delete it while pip will
believe
+// the package is fully installed.
+outputs.dirs(project.ext.envdir)
+ }
+
+ project.configurations { distConfig }
+
+ project.task('sdist', dependsOn: 'setupVirtualenv') {
+doLast {
+ project.exec {
+executable 'sh'
+args '-c', ". ${project.ext.envdir}/bin/activate && python
${project.ext.pythonRootDir}/setup.py sdist --formats zip,gztar --dist-dir
${project.buildDir}"
+ }
+ def collection = project.fileTree("${project.buildDir}"){ include
'**/*.tar.gz' exclude '**/apache-beam.tar.gz'}
+ println "sdist archive name: ${collection.singleFile}"
+ // we need a fixed name for the artifact
+ project.copy { from collection.singleFile; into
"${project.buildDir}"; rename { 'apache-beam.tar.gz' } }
+}
+ }
+
+ project.artifacts {
+distConfig file:
project.file("${project.buildDir}/apache-beam.tar.gz"), builtBy: project.sdist
+ }
+
+ project.task('installGcpTest', dependsOn: 'setupVirtualenv') {
+doLast {
+ project.exec {
+executable 'sh'
+args '-c', ". ${project.ext.envdir}/bin/activate && pip install -e
${project.ext.pythonRootDir}/[gcp,test]"
+ }
+}
+ }
+ project.installGcpTest.mustRunAfter project.sdist
+
+ project.task('cleanPython', dependsOn: 'setupVirtualenv') {
+doLast {
+ project.exec {
+executable 'sh'
+args '-c', ". ${project.ext.envdir}/bin/activate && python
${project.ext.pythonRootDir}/setup.py clean"
+ }
+ project.delete project.buildDir
+}
+ }
+ project.clean.dependsOn project.cleanPython
+}
}
}
diff
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=171671=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171671
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 03/Dec/18 18:35
Start Date: 03/Dec/18 18:35
Worklog Time Spent: 10m
Work Description: markflyhigh commented on issue #7163: [BEAM-5058] Run
basic ITs in Python Precommit in parallel
URL: https://github.com/apache/beam/pull/7163#issuecomment-443818622
PTAL @udim
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 171671)
Time Spent: 5.5h (was: 5h 20m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=171253=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171253
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 01/Dec/18 00:30
Start Date: 01/Dec/18 00:30
Worklog Time Spent: 10m
Work Description: markflyhigh commented on a change in pull request
#7163: [BEAM-5058] Run basic ITs in Python Precommit in parallel
URL: https://github.com/apache/beam/pull/7163#discussion_r238039772
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1502,5 +1502,89 @@ artifactId=${project.name}
dependsOn ':beam-sdks-java-container:docker'
}
}
+
Review comment:
Actually we have
[javaNature](https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L552)
and
[goNature](https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1250)
in this file already.
However, I agree that this file is huge and hard to read, and totally agree
to refactor it. To me, it sounds like a separate work and would love to involve
people who expert on gradle/groovy.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 171253)
Time Spent: 5h 20m (was: 5h 10m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=171254=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171254
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 01/Dec/18 00:30
Start Date: 01/Dec/18 00:30
Worklog Time Spent: 10m
Work Description: markflyhigh commented on a change in pull request
#7163: [BEAM-5058] Run basic ITs in Python Precommit in parallel
URL: https://github.com/apache/beam/pull/7163#discussion_r238040508
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1502,5 +1502,89 @@ artifactId=${project.name}
dependsOn ':beam-sdks-java-container:docker'
}
}
+
+/**
***/
+
+def randomGenerator = { int length ->
+ String charSet = (('A'..'Z')+('0'..'9')).join()
+ new Random().with{
+(1..length).collect { charSet[ nextInt(charSet.length()) ] }.join()
+ }
+}
+
+project.ext.applyPythonNature = {
+
+ // Define common lifecycle tasks and artifact types
+ project.apply plugin: "base"
+
+ // For some reason base doesn't define a test task so we define it
below and make
+ // check depend on it. This makes the Python project similar to the task
layout like
+ // Java projects, see
https://docs.gradle.org/4.2.1/userguide/img/javaPluginTasks.png
+ project.task('test', type: Test) {}
+ project.check.dependsOn project.test
+
+
project.evaluationDependsOn(":beam-runners-google-cloud-dataflow-java-fn-api-worker")
+
+ // Due to Beam-4256, we need to limit the length of virtualenv path to
make the
+ // virtualenv activated properly. So instead of using build directory of
the
+ // subproject, we put all in build directory of root with fixed length
of random
+ // string as path suffix.
+ project.ext.envdir =
"${project.rootProject.buildDir}/gradleenv/${randomGenerator(15)}"
Review comment:
Good point. A consistent env path make build directory more cleaner. Will
change to hash value.
As for clean up, we can consider call `cleanupPython` (line 1578) in Jenkins
before and after each build. However, I feel that Jenkins starts a clean
directory before each build but I don't have evidence yet.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 171254)
Time Spent: 5h 20m (was: 5h 10m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=171230=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171230
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 30/Nov/18 23:25
Start Date: 30/Nov/18 23:25
Worklog Time Spent: 10m
Work Description: udim commented on a change in pull request #7163:
[BEAM-5058] Run basic ITs in Python Precommit in parallel
URL: https://github.com/apache/beam/pull/7163#discussion_r238031870
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1502,5 +1502,89 @@ artifactId=${project.name}
dependsOn ':beam-sdks-java-container:docker'
}
}
+
+/**
***/
+
+def randomGenerator = { int length ->
+ String charSet = (('A'..'Z')+('0'..'9')).join()
+ new Random().with{
+(1..length).collect { charSet[ nextInt(charSet.length()) ] }.join()
+ }
+}
+
+project.ext.applyPythonNature = {
+
+ // Define common lifecycle tasks and artifact types
+ project.apply plugin: "base"
+
+ // For some reason base doesn't define a test task so we define it
below and make
+ // check depend on it. This makes the Python project similar to the task
layout like
+ // Java projects, see
https://docs.gradle.org/4.2.1/userguide/img/javaPluginTasks.png
+ project.task('test', type: Test) {}
+ project.check.dependsOn project.test
+
+
project.evaluationDependsOn(":beam-runners-google-cloud-dataflow-java-fn-api-worker")
+
+ // Due to Beam-4256, we need to limit the length of virtualenv path to
make the
+ // virtualenv activated properly. So instead of using build directory of
the
+ // subproject, we put all in build directory of root with fixed length
of random
+ // string as path suffix.
+ project.ext.envdir =
"${project.rootProject.buildDir}/gradleenv/${randomGenerator(15)}"
Review comment:
Doesn't this generate a new virtualenv every time you invoke gradle with
setupVirtualenv?
It probably also doesn't clean it up.
How about using a hash value of the task name instead?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 171230)
Time Spent: 5h 10m (was: 5h)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=171186=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171186
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 30/Nov/18 19:42
Start Date: 30/Nov/18 19:42
Worklog Time Spent: 10m
Work Description: markflyhigh edited a comment on issue #7163:
[BEAM-5058] Run basic ITs in Python Precommit in parallel
URL: https://github.com/apache/beam/pull/7163#issuecomment-443037544
+R: PTAL @udim @aaltay
Addressed issues in https://github.com/apache/beam/pull/6707 and start a new
thread for review.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 171186)
Time Spent: 4h 50m (was: 4h 40m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=170879=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-170879
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 29/Nov/18 23:51
Start Date: 29/Nov/18 23:51
Worklog Time Spent: 10m
Work Description: markflyhigh commented on issue #7163: [BEAM-5058] Run
basic ITs in Python Precommit in parallel
URL: https://github.com/apache/beam/pull/7163#issuecomment-443037544
+R: PTAL @udim
Addressed issues in https://github.com/apache/beam/pull/6707 and start a new
thread for review.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 170879)
Time Spent: 4h 40m (was: 4.5h)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=170877=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-170877
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 29/Nov/18 23:47
Start Date: 29/Nov/18 23:47
Worklog Time Spent: 10m
Work Description: markflyhigh commented on issue #7163: [BEAM-5058] Run
basic ITs in Python Precommit in parallel
URL: https://github.com/apache/beam/pull/7163#issuecomment-443036759
Run Java_Examples_Dataflow PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 170877)
Time Spent: 4.5h (was: 4h 20m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=170875=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-170875
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 29/Nov/18 23:44
Start Date: 29/Nov/18 23:44
Worklog Time Spent: 10m
Work Description: markflyhigh commented on issue #7163: [BEAM-5058] Run
basic ITs in Python Precommit in parallel
URL: https://github.com/apache/beam/pull/7163#issuecomment-443036188
Run Website PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 170875)
Time Spent: 4h 20m (was: 4h 10m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[ https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=170773=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-170773 ] ASF GitHub Bot logged work on BEAM-5058: Author: ASF GitHub Bot Created on: 29/Nov/18 19:15 Start Date: 29/Nov/18 19:15 Worklog Time Spent: 10m Work Description: markflyhigh opened a new pull request #7163: [BEAM-5058] Run basic ITs in Python Precommit in parallel URL: https://github.com/apache/beam/pull/7163 This change contains two things: - Move basic python Gradle tasks to BeamModulePlugin so that any Python sub-projects can reuse them for environment setup, build tar file, dependency install and cleanup. - Isolate virtualenv of different projects for parallel testing. - Create a sub-project beam-sdks-python-precommit-dataflow which defines basic Dataflow integration tests running in PreCommit. Those tests can run in parallel along with :beam-sdks-python:preCommit. Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message,
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=170754=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-170754
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 29/Nov/18 18:24
Start Date: 29/Nov/18 18:24
Worklog Time Spent: 10m
Work Description: markflyhigh commented on issue #7146: [BEAM-5058] Run
basic ITs in Python Precommit in parallel
URL: https://github.com/apache/beam/pull/7146#issuecomment-442940168
Portable_Python test is introduced by a pending PR. I'll close this PR and
create a new one to to get this test removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 170754)
Time Spent: 3h 50m (was: 3h 40m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=170691=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-170691
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 29/Nov/18 16:04
Start Date: 29/Nov/18 16:04
Worklog Time Spent: 10m
Work Description: markflyhigh commented on issue #7146: [BEAM-5058] Run
basic ITs in Python Precommit in parallel
URL: https://github.com/apache/beam/pull/7146#issuecomment-442889324
Run Website PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 170691)
Time Spent: 3h 40m (was: 3.5h)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=170063=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-170063
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 28/Nov/18 00:20
Start Date: 28/Nov/18 00:20
Worklog Time Spent: 10m
Work Description: markflyhigh closed pull request #6707: [BEAM-5058]
Parallelize Python Precommit and add basic ITs
URL: https://github.com/apache/beam/pull/6707
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.test-infra/jenkins/CommonJobProperties.groovy
b/.test-infra/jenkins/CommonJobProperties.groovy
index fcac3eca26ec..c85c77560c51 100644
--- a/.test-infra/jenkins/CommonJobProperties.groovy
+++ b/.test-infra/jenkins/CommonJobProperties.groovy
@@ -331,4 +331,24 @@ class CommonJobProperties {
static String makePathAbsolute(String path) {
return '"$WORKSPACE/' + path + '"'
}
+
+ /**
+ * Create a temporary directory.
+ * @return Absolute path of the directory.
+ */
+ static String createTempDirectory() {
+def tempDir = File.createTempDir()
+assert tempDir.exists()
+return tempDir.absolutePath
+ }
+
+ /**
+ * Delete a directory recursively.
+ * @param Abusolute path of a directory to be deleted.
+ */
+ static void deleteDirectory(String path) {
+def dir = new File(path)
+dir.deleteDir()
+assert !dir.exists()
+ }
}
diff --git a/.test-infra/jenkins/job_PreCommit_Python.groovy
b/.test-infra/jenkins/job_PreCommit_Python.groovy
index 35b34f2c6789..43d23004cc40 100644
--- a/.test-infra/jenkins/job_PreCommit_Python.groovy
+++ b/.test-infra/jenkins/job_PreCommit_Python.groovy
@@ -16,12 +16,16 @@
* limitations under the License.
*/
+import CommonJobProperties as commonJobProperties
import PrecommitJobBuilder
+// Generate a temporary directory to hold virtualenvs
+def tempDir = commonJobProperties.createTempDirectory()
+
PrecommitJobBuilder builder = new PrecommitJobBuilder(
scope: this,
nameBase: 'Python',
-gradleTask: ':pythonPreCommit',
+gradleTask: ":pythonPreCommit -PenvBaseDir=$tempDir",
triggerPathPatterns: [
'^model/.*$',
'^runners/.*$',
@@ -36,3 +40,6 @@ builder.build {
archiveJunit('**/nosetests.xml')
}
}
+
+// Clean up
+commonJobProperties.deleteDirectory(tempDir)
\ No newline at end of file
diff --git a/build.gradle b/build.gradle
index 269da86f8656..e9a2cadc5b8f 100644
--- a/build.gradle
+++ b/build.gradle
@@ -240,6 +240,7 @@ task goIntegrationTests() {
task pythonPreCommit() {
dependsOn ":beam-sdks-python:preCommit"
+ dependsOn ":beam-sdks-python-precommit-dataflow:precommitIT"
}
task pythonPostCommit() {
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index ba17983856ce..aaf717dc2e8d 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -1492,5 +1492,79 @@ artifactId=${project.name}
dependsOn ':beam-sdks-java-container:docker'
}
}
+
+/**
***/
+
+project.ext.applyPythonNature = {
+
+ // Define common lifecycle tasks and artifact types
+ project.apply plugin: "base"
+
+ // For some reason base doesn't define a test task so we define it
below and make
+ // check depend on it. This makes the Python project similar to the task
layout like
+ // Java projects, see
https://docs.gradle.org/4.2.1/userguide/img/javaPluginTasks.png
+ project.task('test', type: Test) {}
+ project.check.dependsOn project.test
+
+
project.evaluationDependsOn(":beam-runners-google-cloud-dataflow-java-fn-api-worker")
+
+ project.ext.envdir = project.findProperty('envBaseDir') ?:
"${project.rootProject.buildDir}"
+ project.ext.envdir = project.ext.envdir + "/${project.name}/gradleenv"
+ project.ext.pythonRootDir = "${project.rootDir}/sdks/python"
+
+ project.task('setupVirtualenv') {
+doLast {
+ project.exec { commandLine 'virtualenv', "${project.ext.envdir}" }
+ project.exec {
+executable 'sh'
+args '-c', ". ${project.ext.envdir}/bin/activate && pip install
--upgrade tox==3.0.0 grpcio-tools==1.3.5"
+ }
+}
+// Gradle will delete outputs whenever it thinks they are stale.
Putting a
+// specific binary here could make gradle delete it while pip will
believe
+// the package is fully
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=170062=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-170062
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 28/Nov/18 00:20
Start Date: 28/Nov/18 00:20
Worklog Time Spent: 10m
Work Description: markflyhigh commented on issue #6707: [BEAM-5058]
Parallelize Python Precommit and add basic ITs
URL: https://github.com/apache/beam/pull/6707#issuecomment-442269765
I don't know why this PR have some many Pre/Post commit tests enabled. Will
close it and create a cleaner one for future review.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 170062)
Time Spent: 3h 20m (was: 3h 10m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=161353=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161353
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 31/Oct/18 21:59
Start Date: 31/Oct/18 21:59
Worklog Time Spent: 10m
Work Description: udim commented on issue #6707: [BEAM-5058] Parallelize
Python Precommit and add basic ITs
URL: https://github.com/apache/beam/pull/6707#issuecomment-434862676
setupVirtualenv is probably failing due to the path being too long:
https://issues.apache.org/jira/browse/BEAM-4256
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161353)
Time Spent: 3h (was: 2h 50m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=161342=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161342
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 31/Oct/18 21:50
Start Date: 31/Oct/18 21:50
Worklog Time Spent: 10m
Work Description: udim commented on a change in pull request #6707:
[BEAM-5058] Parallelize Python Precommit and add basic ITs
URL: https://github.com/apache/beam/pull/6707#discussion_r229444669
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1580,5 +1580,76 @@ artifactId=${project.name}
dependsOn ':beam-sdks-java-container:docker'
}
}
+
+/**
***/
+
+project.ext.applyPythonNature = {
+
+ // Define common lifecycle tasks and artifact types
+ project.apply plugin: "base"
+
+ // For some reason base doesn't define a test task so we define it
below and make
+ // check depend on it. This makes the Python project similar to the task
layout like
+ // Java projects, see
https://docs.gradle.org/4.2.1/userguide/img/javaPluginTasks.png
+ project.task('test', type: Test) {}
+ project.check.dependsOn project.test
Review comment:
I don't believe that projects under subdirs define tasks for `test`.
It doesn't look right to define a new task named `test` more than once. How
does Gradle handle this? Will we have 2 or more `test` tasks?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161342)
Time Spent: 2h 50m (was: 2h 40m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=161240=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161240
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 31/Oct/18 17:26
Start Date: 31/Oct/18 17:26
Worklog Time Spent: 10m
Work Description: markflyhigh commented on issue #6707: [BEAM-5058]
Parallelize Python Precommit and add basic ITs
URL: https://github.com/apache/beam/pull/6707#issuecomment-434776546
One thing to notice is that with above change, calling `setupVirtualenv` in
different sub-projects will install virtualenv in their own build directory.
For example, tasks in `:beam-sdks-python-precommit-dataflow` will setup
virtualenv in `sdks/python/precommit/dataflow/build/`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161240)
Time Spent: 2h 40m (was: 2.5h)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=160931=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-160931
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 31/Oct/18 00:01
Start Date: 31/Oct/18 00:01
Worklog Time Spent: 10m
Work Description: udim commented on issue #6707: [BEAM-5058] Parallelize
Python Precommit and add basic ITs
URL: https://github.com/apache/beam/pull/6707#issuecomment-434514170
`pip install --user` isn't safe to use concurrently.
I'm not sure why you're getting the error you're currently seeing, but
running `virtualenv` concurrently on the same path is not a good idea.
I don't agree with redefining the tasks setupVirtualenv, sdist,
installGcpTest, and cleanPython for every sub-project. This allows for running
them more than once. All 4 tasks above should only be run once during a gradle
invocation, and some of them give undefined results if run concurrently.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 160931)
Time Spent: 2.5h (was: 2h 20m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=160411=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-160411
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 30/Oct/18 06:00
Start Date: 30/Oct/18 06:00
Worklog Time Spent: 10m
Work Description: markflyhigh commented on issue #6707: [BEAM-5058]
Parallelize Python Precommit and add basic ITs
URL: https://github.com/apache/beam/pull/6707#issuecomment-434181746
Interesting. PreCommit failed due to virtualenv of
`sdks-python-precommit-dataflow:precommitIT` is not activated successfully.
From [Gradle failure
log](https://scans.gradle.com/s/hdawv5ou4girm/console-log?task=:beam-sdks-python-precommit-dataflow:setupVirtualenv),
new `precommitIT` use system package
(`/home/jenkins/.local/lib/python2.7/site-packages`) instead of virtualenv in
build directory.
Guess the reason may come from parallelization. Before each python command,
we call `. ${envdir}/bin/activate` but this may not work if two virtualenv
activate simultaneously. However, I cannot reproduce same error from local
machine.
Two possible solutions:
1. Add `--user` to pip install as [failure
log](https://scans.gradle.com/s/hdawv5ou4girm/console-log?task=:beam-sdks-python-precommit-dataflow:setupVirtualenv)
suggested. Unfortunately, the gradle task will not work on my laptop.
1. Instead of calling `. ${envdir}/bin/activate`, we can directly use the
Python interpreter or pip in virtualenv.
Any thoughts? @udim @aaltay
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 160411)
Time Spent: 2h 20m (was: 2h 10m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=160301=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-160301
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 29/Oct/18 23:36
Start Date: 29/Oct/18 23:36
Worklog Time Spent: 10m
Work Description: markflyhigh commented on a change in pull request
#6707: [BEAM-5058] Parallelize Python Precommit and add basic ITs
URL: https://github.com/apache/beam/pull/6707#discussion_r229134653
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1580,5 +1580,76 @@ artifactId=${project.name}
dependsOn ':beam-sdks-java-container:docker'
}
}
+
+/**
***/
+
+project.ext.applyPythonNature = {
+
+ // Define common lifecycle tasks and artifact types
+ project.apply plugin: "base"
+
+ // For some reason base doesn't define a test task so we define it
below and make
+ // check depend on it. This makes the Python project similar to the task
layout like
+ // Java projects, see
https://docs.gradle.org/4.2.1/userguide/img/javaPluginTasks.png
+ project.task('test', type: Test) {}
+ project.check.dependsOn project.test
+
+ project.ext.envdir = "${project.buildDir}/gradleenv"
+ def pythonRootDir = "${project.rootDir}/sdks/python"
+
+ project.task('setupVirtualenv') {
+doLast {
+ project.exec { commandLine 'virtualenv', "${project.ext.envdir}" }
+ project.exec {
+executable 'sh'
+args '-c', ". ${project.ext.envdir}/bin/activate && pip install
--upgrade tox==3.0.0 grpcio-tools==1.3.5"
+ }
+}
+// Gradle will delete outputs whenever it thinks they are stale.
Putting a
+// specific binary here could make gradle delete it while pip will
believe
+// the package is fully installed.
+outputs.dirs(project.ext.envdir)
+ }
+
+ project.configurations { distConfig }
+
+ project.task('sdist', dependsOn: 'setupVirtualenv') {
+doLast {
+ project.exec {
+executable 'sh'
+args '-c', ". ${project.ext.envdir}/bin/activate && python
${pythonRootDir}/setup.py sdist --formats zip,gztar --dist-dir
${project.buildDir}"
+ }
+ def collection = project.fileTree("${project.buildDir}"){ include
'**/*.tar.gz' exclude '**/apache-beam.tar.gz'}
+ println "sdist archive name: ${collection.singleFile}"
Review comment:
It's the only place shows full path of the tar ball in Gradle log which I
think is very helpful for debugging/developing. Prefer to keep it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 160301)
Time Spent: 2h 10m (was: 2h)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=160297=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-160297
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 29/Oct/18 23:17
Start Date: 29/Oct/18 23:17
Worklog Time Spent: 10m
Work Description: markflyhigh commented on a change in pull request
#6707: [BEAM-5058] Parallelize Python Precommit and add basic ITs
URL: https://github.com/apache/beam/pull/6707#discussion_r229131355
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1580,5 +1580,76 @@ artifactId=${project.name}
dependsOn ':beam-sdks-java-container:docker'
}
}
+
+/**
***/
+
+project.ext.applyPythonNature = {
+
+ // Define common lifecycle tasks and artifact types
+ project.apply plugin: "base"
+
+ // For some reason base doesn't define a test task so we define it
below and make
+ // check depend on it. This makes the Python project similar to the task
layout like
+ // Java projects, see
https://docs.gradle.org/4.2.1/userguide/img/javaPluginTasks.png
+ project.task('test', type: Test) {}
+ project.check.dependsOn project.test
Review comment:
Projects under sub-directories may not necessarily depend on
`:beam-sdks-python`, since we explicitly set all subprojects in
`setting.gradle`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 160297)
Time Spent: 2h (was: 1h 50m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=160152=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-160152
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 29/Oct/18 19:48
Start Date: 29/Oct/18 19:48
Worklog Time Spent: 10m
Work Description: udim commented on a change in pull request #6707:
[BEAM-5058] Parallelize Python Precommit and add basic ITs
URL: https://github.com/apache/beam/pull/6707#discussion_r229053009
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1580,5 +1580,76 @@ artifactId=${project.name}
dependsOn ':beam-sdks-java-container:docker'
}
}
+
+/**
***/
+
+project.ext.applyPythonNature = {
+
+ // Define common lifecycle tasks and artifact types
+ project.apply plugin: "base"
+
+ // For some reason base doesn't define a test task so we define it
below and make
+ // check depend on it. This makes the Python project similar to the task
layout like
+ // Java projects, see
https://docs.gradle.org/4.2.1/userguide/img/javaPluginTasks.png
+ project.task('test', type: Test) {}
+ project.check.dependsOn project.test
+
+ project.ext.envdir = "${project.buildDir}/gradleenv"
+ def pythonRootDir = "${project.rootDir}/sdks/python"
Review comment:
This could also be a `project.ext.` variable, since it's used in more than
one file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 160152)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=159857=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159857
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 29/Oct/18 05:10
Start Date: 29/Oct/18 05:10
Worklog Time Spent: 10m
Work Description: markflyhigh commented on a change in pull request
#6707: [BEAM-5058] Run basic ITs in Python Precommit
URL: https://github.com/apache/beam/pull/6707#discussion_r228802561
##
File path: sdks/python/build.gradle
##
@@ -226,6 +228,26 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
}
}
+task precommitIT(dependsOn: ['installGcpTest', 'sdist']) {
Review comment:
Leverage Gradle parallel mechanism is a great idea. I move some basic setup
tasks into `BeamModulePlugins` and move `precommitIT` under
`.../python/precommit/dataflow`. Let's see how this effects PreCommit build
time.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 159857)
Time Spent: 1h 20m (was: 1h 10m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=159854=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159854
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 29/Oct/18 05:05
Start Date: 29/Oct/18 05:05
Worklog Time Spent: 10m
Work Description: markflyhigh commented on a change in pull request
#6707: [BEAM-5058] Run basic ITs in Python Precommit
URL: https://github.com/apache/beam/pull/6707#discussion_r228802133
##
File path: sdks/python/build.gradle
##
@@ -226,6 +228,26 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
}
}
+task precommitIT(dependsOn: ['installGcpTest', 'sdist']) {
+ doLast {
+// List of integration tests running in Python PreCommit.
+def precommitTests = [
+"apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it",
+
"apache_beam.examples.streaming_wordcount_it_test:StreamingWordCountIT.test_streaming_wordcount_it",
+]
+def testOpts = [
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 159854)
Time Spent: 1h 10m (was: 1h)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=159852=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159852
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 29/Oct/18 05:04
Start Date: 29/Oct/18 05:04
Worklog Time Spent: 10m
Work Description: markflyhigh commented on a change in pull request
#6707: [BEAM-5058] Run basic ITs in Python Precommit
URL: https://github.com/apache/beam/pull/6707#discussion_r228802106
##
File path: sdks/python/build.gradle
##
@@ -226,6 +228,26 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
}
}
+task precommitIT(dependsOn: ['installGcpTest', 'sdist']) {
+ doLast {
+// List of integration tests running in Python PreCommit.
+def precommitTests = [
+"apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it",
+
"apache_beam.examples.streaming_wordcount_it_test:StreamingWordCountIT.test_streaming_wordcount_it",
+]
+def testOpts = [
+"--tests=${precommitTests.join(',')}",
+"--processes=4",
+"--process-timeout=1800", // Total timeout includes all tests run.
+]
+
+exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate &&
./scripts/run_integration_test.sh --test_opts \"${testOpts.join(' ')}\""
Review comment:
sg. I'd love to put it in next PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 159852)
Time Spent: 50m (was: 40m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=159853=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159853
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 29/Oct/18 05:04
Start Date: 29/Oct/18 05:04
Worklog Time Spent: 10m
Work Description: markflyhigh commented on a change in pull request
#6707: [BEAM-5058] Run basic ITs in Python Precommit
URL: https://github.com/apache/beam/pull/6707#discussion_r228802125
##
File path: sdks/python/build.gradle
##
@@ -226,6 +228,26 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
}
}
+task precommitIT(dependsOn: ['installGcpTest', 'sdist']) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 159853)
Time Spent: 1h (was: 50m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=155230=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-155230
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 17/Oct/18 00:48
Start Date: 17/Oct/18 00:48
Worklog Time Spent: 10m
Work Description: udim commented on a change in pull request #6707:
[BEAM-5058] Run basic ITs in Python Precommit
URL: https://github.com/apache/beam/pull/6707#discussion_r225749774
##
File path: sdks/python/build.gradle
##
@@ -226,6 +228,26 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
}
}
+task precommitIT(dependsOn: ['installGcpTest', 'sdist']) {
Review comment:
This task should run in parallel to the rest of the precommit tasks. This
can be done if it is placed in a separate sub-project. Sub-projects are created
by creating a new build.gradle file in a subdirectory, such as
`sdks/python/precommit/dataflow/build.gradle`. (example of making tests
parallel: https://github.com/apache/beam/pull/5731/files)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 155230)
Time Spent: 0.5h (was: 20m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=155229=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-155229
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 17/Oct/18 00:48
Start Date: 17/Oct/18 00:48
Worklog Time Spent: 10m
Work Description: udim commented on a change in pull request #6707:
[BEAM-5058] Run basic ITs in Python Precommit
URL: https://github.com/apache/beam/pull/6707#discussion_r225751687
##
File path: sdks/python/build.gradle
##
@@ -226,6 +228,26 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
}
}
+task precommitIT(dependsOn: ['installGcpTest', 'sdist']) {
+ doLast {
+// List of integration tests running in Python PreCommit.
+def precommitTests = [
+"apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it",
+
"apache_beam.examples.streaming_wordcount_it_test:StreamingWordCountIT.test_streaming_wordcount_it",
+]
+def testOpts = [
+"--tests=${precommitTests.join(',')}",
+"--processes=4",
+"--process-timeout=1800", // Total timeout includes all tests run.
+]
+
+exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate &&
./scripts/run_integration_test.sh --test_opts \"${testOpts.join(' ')}\""
Review comment:
After parallelizing this task, please add a copy of it in
`sdks/python/precommit/directrunner/build.gradle` with the option
`--runner=TestDirectRunner`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 155229)
Time Spent: 0.5h (was: 20m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=155232=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-155232
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 17/Oct/18 00:48
Start Date: 17/Oct/18 00:48
Worklog Time Spent: 10m
Work Description: udim commented on a change in pull request #6707:
[BEAM-5058] Run basic ITs in Python Precommit
URL: https://github.com/apache/beam/pull/6707#discussion_r225750634
##
File path: sdks/python/build.gradle
##
@@ -226,6 +228,26 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
}
}
+task precommitIT(dependsOn: ['installGcpTest', 'sdist']) {
Review comment:
sdist should run before installGcpTest. I believe gradle has a rule for that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 155232)
Time Spent: 40m (was: 0.5h)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=155113=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-155113
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 16/Oct/18 20:48
Start Date: 16/Oct/18 20:48
Worklog Time Spent: 10m
Work Description: markflyhigh commented on issue #6707: [BEAM-5058] Run
basic ITs in Python Precommit
URL: https://github.com/apache/beam/pull/6707#issuecomment-430395043
PreCommit passed.
@udim @aaltay Please take a look.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 155113)
Time Spent: 20m (was: 10m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
