[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[
https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=88578=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88578
]
ASF GitHub Bot logged work on BEAM-3257:
Author: ASF GitHub Bot
Created on: 06/Apr/18 19:29
Start Date: 06/Apr/18 19:29
Worklog Time Spent: 10m
Work Description: lukecwik closed pull request #5045: [BEAM-3257] Migrate
JUnit test archiving to Python Gradle build
URL: https://github.com/apache/beam/pull/5045
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy
b/.test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy
index d2fdef72ad3..255277d471a 100644
--- a/.test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy
+++ b/.test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy
@@ -41,6 +41,12 @@ job('beam_PreCommit_Python_GradleBuild') {
'--rerun-tasks',
]
+ // Publish all test results to Jenkins. Note that Nose documentation
+ // specifically mentions that it produces JUnit compatible test results.
+ publishers {
+archiveJunit('**/nosetests.xml')
+ }
+
def gradle_command_line = './gradlew ' + gradle_switches.join(' ') + '
:pythonPreCommit'
// Sets that this is a PreCommit job.
common_job_properties.setPreCommit(delegate, gradle_command_line, 'Run
Python Gradle PreCommit')
diff --git a/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
b/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
deleted file mode 100644
index f0429e4aa18..000
--- a/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
+++ /dev/null
@@ -1,57 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import common_job_properties
-
-// This is the Python precommit which runs a maven install, and the current set
-// of precommit tests.
-mavenJob('beam_PreCommit_Python_MavenInstall') {
- description('Runs an install of the current GitHub Pull Request.')
-
- previousNames('beam_PreCommit_MavenVerify')
-
- // Execute concurrent builds if necessary.
- concurrentBuild()
-
- // Set common parameters.
- common_job_properties.setTopLevelMainJobProperties(
-delegate,
-'master',
-150)
-
- // Set Maven parameters.
- common_job_properties.setMavenConfig(delegate)
-
- // Sets that this is a PreCommit job.
- common_job_properties.setPreCommit(delegate, 'mvn clean install -pl
sdks/python -am -amd', 'Run Python PreCommit')
-
- // Maven modules for this job: The Python SDK, its dependencies, and things
that depend on it,
- // excluding the container.
- goals([
-'--batch-mode',
-'--errors',
-'--activate-profiles release',
-'--projects sdks/python,!sdks/python/container',
-'--also-make',
-'--also-make-dependents',
-'-D pullRequest=$ghprbPullId',
-'help:effective-settings',
-'clean',
-'install',
- ].join(' '))
-}
diff --git a/sdks/python/setup.cfg b/sdks/python/setup.cfg
index 0f6acea2c09..041d6fa55f4 100644
--- a/sdks/python/setup.cfg
+++ b/sdks/python/setup.cfg
@@ -26,6 +26,9 @@ verbosity=2
# fast_coders_test and typecoders_test.
exclude=fast_coders_test|typecoders_test
+# Creates an xml file compatible with standard XUnit XML format.
+with-xunit=1
+
# Configurations to control coverage.py.
[coverage:run]
branch = True
diff --git a/sdks/python/tox.ini b/sdks/python/tox.ini
index ff88ac42fa8..3cc16823684 100644
--- a/sdks/python/tox.ini
+++ b/sdks/python/tox.ini
@@ -50,7 +50,7 @@ commands =
pip --version
{toxinidir}/run_tox_cleanup.sh
python apache_beam/examples/complete/autocomplete_test.py
- python setup.py test
+ python setup.py nosetests
{toxinidir}/run_tox_cleanup.sh
[testenv:py27-cython]
@@ -64,7 +64,7 @@ commands =
pip --version
{toxinidir}/run_tox_cleanup.sh
python apache_beam/examples/complete/autocomplete_test.py
- python setup.py
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[
https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=88577=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88577
]
ASF GitHub Bot logged work on BEAM-3257:
Author: ASF GitHub Bot
Created on: 06/Apr/18 19:26
Start Date: 06/Apr/18 19:26
Worklog Time Spent: 10m
Work Description: aaltay closed pull request #5042: [BEAM-3257] Publish
Python precommit test results to Jenkins, remove Maven based Python precommit.
URL: https://github.com/apache/beam/pull/5042
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy
b/.test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy
index d2fdef72ad3..255277d471a 100644
--- a/.test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy
+++ b/.test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy
@@ -41,6 +41,12 @@ job('beam_PreCommit_Python_GradleBuild') {
'--rerun-tasks',
]
+ // Publish all test results to Jenkins. Note that Nose documentation
+ // specifically mentions that it produces JUnit compatible test results.
+ publishers {
+archiveJunit('**/nosetests.xml')
+ }
+
def gradle_command_line = './gradlew ' + gradle_switches.join(' ') + '
:pythonPreCommit'
// Sets that this is a PreCommit job.
common_job_properties.setPreCommit(delegate, gradle_command_line, 'Run
Python Gradle PreCommit')
diff --git a/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
b/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
deleted file mode 100644
index f0429e4aa18..000
--- a/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
+++ /dev/null
@@ -1,57 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import common_job_properties
-
-// This is the Python precommit which runs a maven install, and the current set
-// of precommit tests.
-mavenJob('beam_PreCommit_Python_MavenInstall') {
- description('Runs an install of the current GitHub Pull Request.')
-
- previousNames('beam_PreCommit_MavenVerify')
-
- // Execute concurrent builds if necessary.
- concurrentBuild()
-
- // Set common parameters.
- common_job_properties.setTopLevelMainJobProperties(
-delegate,
-'master',
-150)
-
- // Set Maven parameters.
- common_job_properties.setMavenConfig(delegate)
-
- // Sets that this is a PreCommit job.
- common_job_properties.setPreCommit(delegate, 'mvn clean install -pl
sdks/python -am -amd', 'Run Python PreCommit')
-
- // Maven modules for this job: The Python SDK, its dependencies, and things
that depend on it,
- // excluding the container.
- goals([
-'--batch-mode',
-'--errors',
-'--activate-profiles release',
-'--projects sdks/python,!sdks/python/container',
-'--also-make',
-'--also-make-dependents',
-'-D pullRequest=$ghprbPullId',
-'help:effective-settings',
-'clean',
-'install',
- ].join(' '))
-}
diff --git a/sdks/python/tox.ini b/sdks/python/tox.ini
index ff88ac42fa8..41611384a80 100644
--- a/sdks/python/tox.ini
+++ b/sdks/python/tox.ini
@@ -50,7 +50,7 @@ commands =
pip --version
{toxinidir}/run_tox_cleanup.sh
python apache_beam/examples/complete/autocomplete_test.py
- python setup.py test
+ python setup.py nosetests --with-xunit
{toxinidir}/run_tox_cleanup.sh
[testenv:py27-cython]
@@ -64,7 +64,7 @@ commands =
pip --version
{toxinidir}/run_tox_cleanup.sh
python apache_beam/examples/complete/autocomplete_test.py
- python setup.py test
+ python setup.py nosetests --with-xunit
{toxinidir}/run_tox_cleanup.sh
[testenv:py27-gcp]
@@ -74,7 +74,7 @@ commands =
pip --version
{toxinidir}/run_tox_cleanup.sh
python apache_beam/examples/complete/autocomplete_test.py
- python setup.py test
+ python setup.py nosetests --with-xunit
{toxinidir}/run_tox_cleanup.sh
[testenv:py27-lint]
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[ https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=88576=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88576 ] ASF GitHub Bot logged work on BEAM-3257: Author: ASF GitHub Bot Created on: 06/Apr/18 19:26 Start Date: 06/Apr/18 19:26 Worklog Time Spent: 10m Work Description: aaltay commented on issue #5042: [BEAM-3257] Publish Python precommit test results to Jenkins, remove Maven based Python precommit. URL: https://github.com/apache/beam/pull/5042#issuecomment-379353336 Superseeded by #5045 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 88576) Time Spent: 3h 50m (was: 3h 40m) > Migrate Python Jenkins PreCommits to Gradle > --- > > Key: BEAM-3257 > URL: https://issues.apache.org/jira/browse/BEAM-3257 > Project: Beam > Issue Type: Sub-task > Components: build-system, testing >Reporter: Luke Cwik >Assignee: Udi Meiri >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > Code is here: > https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[ https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=88575=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88575 ] ASF GitHub Bot logged work on BEAM-3257: Author: ASF GitHub Bot Created on: 06/Apr/18 19:25 Start Date: 06/Apr/18 19:25 Worklog Time Spent: 10m Work Description: aaltay opened a new pull request #5045: [BEAM-3257] Migrate JUnit test archiving to Python Gradle build URL: https://github.com/apache/beam/pull/5045 Publish python test results to Jenkins. Delete Maven based Python precommit. Note that this change supersedes #4689 #5042 Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [ ] Write a pull request description that is detailed enough to understand: - [ ] What the pull request does - [ ] Why it does it - [ ] How it does it - [ ] Why this approach - [ ] Each commit in the pull request should have a meaningful subject line and body. - [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 88575) Time Spent: 3h 40m (was: 3.5h) > Migrate Python Jenkins PreCommits to Gradle > --- > > Key: BEAM-3257 > URL: https://issues.apache.org/jira/browse/BEAM-3257 > Project: Beam > Issue Type: Sub-task > Components: build-system, testing >Reporter: Luke Cwik >Assignee: Udi Meiri >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > Code is here: > https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[
https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=88571=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88571
]
ASF GitHub Bot logged work on BEAM-3257:
Author: ASF GitHub Bot
Created on: 06/Apr/18 19:05
Start Date: 06/Apr/18 19:05
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #5042:
[BEAM-3257] Publish Python precommit test results to Jenkins, remove Maven
based Python precommit.
URL: https://github.com/apache/beam/pull/5042#discussion_r179849434
##
File path: sdks/python/tox.ini
##
@@ -50,7 +50,7 @@ commands =
pip --version
{toxinidir}/run_tox_cleanup.sh
python apache_beam/examples/complete/autocomplete_test.py
- python setup.py test
+ python setup.py nosetests --with-xunit
Review comment:
I believe it will since they will overwrite each other. You would only get
the test results from the last run.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 88571)
Time Spent: 3.5h (was: 3h 20m)
> Migrate Python Jenkins PreCommits to Gradle
> ---
>
> Key: BEAM-3257
> URL: https://issues.apache.org/jira/browse/BEAM-3257
> Project: Beam
> Issue Type: Sub-task
> Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Code is here:
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[
https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=88557=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88557
]
ASF GitHub Bot logged work on BEAM-3257:
Author: ASF GitHub Bot
Created on: 06/Apr/18 18:37
Start Date: 06/Apr/18 18:37
Worklog Time Spent: 10m
Work Description: udim commented on a change in pull request #5042:
[BEAM-3257] Publish Python precommit test results to Jenkins, remove Maven
based Python precommit.
URL: https://github.com/apache/beam/pull/5042#discussion_r179842426
##
File path: sdks/python/tox.ini
##
@@ -50,7 +50,7 @@ commands =
pip --version
{toxinidir}/run_tox_cleanup.sh
python apache_beam/examples/complete/autocomplete_test.py
- python setup.py test
+ python setup.py nosetests --with-xunit
Review comment:
Will each run's `nosetests.xml` output file overwrite the others'?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 88557)
Time Spent: 3h 20m (was: 3h 10m)
> Migrate Python Jenkins PreCommits to Gradle
> ---
>
> Key: BEAM-3257
> URL: https://issues.apache.org/jira/browse/BEAM-3257
> Project: Beam
> Issue Type: Sub-task
> Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Code is here:
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[ https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=88554=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88554 ] ASF GitHub Bot logged work on BEAM-3257: Author: ASF GitHub Bot Created on: 06/Apr/18 18:35 Start Date: 06/Apr/18 18:35 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #5042: [BEAM-3257] Publish Python precommit test results to Jenkins, remove Maven based Python precommit. URL: https://github.com/apache/beam/pull/5042#issuecomment-379339872 CC: @udim @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 88554) Time Spent: 3h 10m (was: 3h) > Migrate Python Jenkins PreCommits to Gradle > --- > > Key: BEAM-3257 > URL: https://issues.apache.org/jira/browse/BEAM-3257 > Project: Beam > Issue Type: Sub-task > Components: build-system, testing >Reporter: Luke Cwik >Assignee: Udi Meiri >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > Code is here: > https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[
https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=87840=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87840
]
ASF GitHub Bot logged work on BEAM-3257:
Author: ASF GitHub Bot
Created on: 05/Apr/18 00:51
Start Date: 05/Apr/18 00:51
Worklog Time Spent: 10m
Work Description: lukecwik closed pull request #5010: [BEAM-3257] Add
Python precommit gradle config
URL: https://github.com/apache/beam/pull/5010
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy
b/.test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy
new file mode 100644
index 000..d2fdef72ad3
--- /dev/null
+++ b/.test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import common_job_properties
+
+// This is the Python precommit which runs a Gradle build, and the current set
+// of precommit tests.
+job('beam_PreCommit_Python_GradleBuild') {
+ description('Runs Python PreCommit tests for the current GitHub Pull
Request.')
+
+ // Execute concurrent builds if necessary.
+ concurrentBuild()
+
+ // Set common parameters.
+ common_job_properties.setTopLevelMainJobProperties(
+delegate,
+'master',
+90)
+
+ def gradle_switches = [
+// Gradle log verbosity enough to diagnose basic build issues
+"--info",
+// Continue the build even if there is a failure to show as many potential
failures as possible.
+'--continue',
+// Until we verify the build cache is working appropriately, force
rerunning all tasks
+'--rerun-tasks',
+ ]
+
+ def gradle_command_line = './gradlew ' + gradle_switches.join(' ') + '
:pythonPreCommit'
+ // Sets that this is a PreCommit job.
+ common_job_properties.setPreCommit(delegate, gradle_command_line, 'Run
Python Gradle PreCommit')
+ steps {
+gradle {
+ rootBuildScriptDir(common_job_properties.checkoutDir)
+ tasks(':pythonPreCommit')
+ for (String gradle_switch : gradle_switches) {
+switches(gradle_switch)
+ }
+}
+ }
+}
diff --git a/build.gradle b/build.gradle
index 602f8e11a33..7b8c60c169e 100644
--- a/build.gradle
+++ b/build.gradle
@@ -126,3 +126,9 @@ task goPreCommit() {
dependsOn ":rat"
dependsOn ":sdks:go:test"
}
+
+task pythonPreCommit() {
+ dependsOn ":rat"
+ dependsOn ":sdks:python:check"
+}
+
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index 2c6637a5373..be17644e8d3 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -25,75 +25,123 @@ apply plugin: "base"
task test {}
check.dependsOn test
-task setupTest {
+def envdir = "${project.buildDir}/gradleenv"
+
+task setupVirtualenv {
doLast {
+exec {
+ commandLine 'virtualenv', "${envdir}"
+}
exec {
executable 'sh'
- args '-c', 'which tox || pip install --user --upgrade tox'
+ args '-c', ". ${envdir}/bin/activate && pip install --upgrade tox"
}
}
+ outputs.files("${envdir}/bin/tox")
}
-task sdist {
+task sdist(dependsOn: 'setupVirtualenv') {
doLast {
exec {
- commandLine 'python', 'setup.py', 'sdist', '--formats', 'zip,gztar',
'--dist-dir', project.buildDir
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && python setup.py sdist --formats
zip,gztar --dist-dir ${project.buildDir}"
}
}
}
-task cleanPython {
+task cleanPython(dependsOn: 'setupVirtualenv') {
doLast {
exec {
- commandLine 'python', 'setup.py', 'clean'
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && python setup.py clean"
}
}
}
clean.dependsOn cleanPython
-task buildPython {
+task buildPython(dependsOn: 'setupVirtualenv') {
doLast {
println 'Building Python Dependencies'
exec {
- commandLine 'python', 'setup.py', 'build',
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[ https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=87826=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87826 ] ASF GitHub Bot logged work on BEAM-3257: Author: ASF GitHub Bot Created on: 04/Apr/18 23:59 Start Date: 04/Apr/18 23:59 Worklog Time Spent: 10m Work Description: udim commented on issue #5010: [BEAM-3257] Add Python precommit gradle config URL: https://github.com/apache/beam/pull/5010#issuecomment-378781506 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 87826) Time Spent: 2h 50m (was: 2h 40m) > Migrate Python Jenkins PreCommits to Gradle > --- > > Key: BEAM-3257 > URL: https://issues.apache.org/jira/browse/BEAM-3257 > Project: Beam > Issue Type: Sub-task > Components: build-system, testing >Reporter: Luke Cwik >Assignee: Udi Meiri >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > Code is here: > https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[ https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=87652=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87652 ] ASF GitHub Bot logged work on BEAM-3257: Author: ASF GitHub Bot Created on: 04/Apr/18 17:22 Start Date: 04/Apr/18 17:22 Worklog Time Spent: 10m Work Description: udim commented on issue #5010: [BEAM-3257] Add Python precommit gradle config URL: https://github.com/apache/beam/pull/5010#issuecomment-378678881 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 87652) Time Spent: 2h 40m (was: 2.5h) > Migrate Python Jenkins PreCommits to Gradle > --- > > Key: BEAM-3257 > URL: https://issues.apache.org/jira/browse/BEAM-3257 > Project: Beam > Issue Type: Sub-task > Components: build-system, testing >Reporter: Luke Cwik >Assignee: Udi Meiri >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > Code is here: > https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[ https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=87632=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87632 ] ASF GitHub Bot logged work on BEAM-3257: Author: ASF GitHub Bot Created on: 04/Apr/18 16:44 Start Date: 04/Apr/18 16:44 Worklog Time Spent: 10m Work Description: udim commented on issue #5010: [BEAM-3257] Add Python precommit gradle config URL: https://github.com/apache/beam/pull/5010#issuecomment-378667422 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 87632) Time Spent: 2.5h (was: 2h 20m) > Migrate Python Jenkins PreCommits to Gradle > --- > > Key: BEAM-3257 > URL: https://issues.apache.org/jira/browse/BEAM-3257 > Project: Beam > Issue Type: Sub-task > Components: build-system, testing >Reporter: Luke Cwik >Assignee: Udi Meiri >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > > Code is here: > https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[ https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=87329=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87329 ] ASF GitHub Bot logged work on BEAM-3257: Author: ASF GitHub Bot Created on: 03/Apr/18 23:12 Start Date: 03/Apr/18 23:12 Worklog Time Spent: 10m Work Description: udim commented on issue #5010: [BEAM-3257] Add Python precommit gradle config URL: https://github.com/apache/beam/pull/5010#issuecomment-378428368 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 87329) Time Spent: 2h 20m (was: 2h 10m) > Migrate Python Jenkins PreCommits to Gradle > --- > > Key: BEAM-3257 > URL: https://issues.apache.org/jira/browse/BEAM-3257 > Project: Beam > Issue Type: Sub-task > Components: build-system, testing >Reporter: Luke Cwik >Assignee: Udi Meiri >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > Code is here: > https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[
https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=87325=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87325
]
ASF GitHub Bot logged work on BEAM-3257:
Author: ASF GitHub Bot
Created on: 03/Apr/18 23:02
Start Date: 03/Apr/18 23:02
Worklog Time Spent: 10m
Work Description: udim commented on a change in pull request #5010:
[BEAM-3257] Add Python precommit gradle config
URL: https://github.com/apache/beam/pull/5010#discussion_r178985870
##
File path: sdks/python/build.gradle
##
@@ -25,75 +25,123 @@ apply plugin: "base"
task test {}
check.dependsOn test
-task setupTest {
+def envdir = "${project.buildDir}/gradleenv"
+
+task setupVirtualenv {
doLast {
+exec {
+ commandLine 'virtualenv', "${envdir}"
+}
exec {
executable 'sh'
- args '-c', 'which tox || pip install --user --upgrade tox'
+ args '-c', "source ${envdir}/bin/activate && pip install --upgrade tox"
}
}
+ outputs.files("$buildDir/.gradleenv/bin/tox")
}
-task sdist {
+task sdist (dependsOn: 'setupVirtualenv') {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 87325)
> Migrate Python Jenkins PreCommits to Gradle
> ---
>
> Key: BEAM-3257
> URL: https://issues.apache.org/jira/browse/BEAM-3257
> Project: Beam
> Issue Type: Sub-task
> Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Code is here:
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[
https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=87324=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87324
]
ASF GitHub Bot logged work on BEAM-3257:
Author: ASF GitHub Bot
Created on: 03/Apr/18 23:02
Start Date: 03/Apr/18 23:02
Worklog Time Spent: 10m
Work Description: udim commented on a change in pull request #5010:
[BEAM-3257] Add Python precommit gradle config
URL: https://github.com/apache/beam/pull/5010#discussion_r178985705
##
File path: .test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy
##
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import common_job_properties
+
+// This is the Python precommit which runs a Gradle build, and the current set
+// of precommit tests.
+job('beam_PreCommit_Python_GradleBuild') {
+ description('Runs Python PreCommit tests for the current GitHub Pull
Request.')
+
+ // Execute concurrent builds if necessary.
+ concurrentBuild()
+
+ // Set common parameters.
+ common_job_properties.setTopLevelMainJobProperties(
+delegate,
+'master',
+90)
+
+ // Publish all test results to Jenkins
+ publishers {
+archiveJunit('**/build/test-results/**/*.xml')
Review comment:
removed
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 87324)
Time Spent: 2h (was: 1h 50m)
> Migrate Python Jenkins PreCommits to Gradle
> ---
>
> Key: BEAM-3257
> URL: https://issues.apache.org/jira/browse/BEAM-3257
> Project: Beam
> Issue Type: Sub-task
> Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Code is here:
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[
https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=87326=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87326
]
ASF GitHub Bot logged work on BEAM-3257:
Author: ASF GitHub Bot
Created on: 03/Apr/18 23:02
Start Date: 03/Apr/18 23:02
Worklog Time Spent: 10m
Work Description: udim commented on a change in pull request #5010:
[BEAM-3257] Add Python precommit gradle config
URL: https://github.com/apache/beam/pull/5010#discussion_r178985158
##
File path: sdks/python/build.gradle
##
@@ -25,75 +25,123 @@ apply plugin: "base"
task test {}
check.dependsOn test
-task setupTest {
+def envdir = "${project.buildDir}/gradleenv"
+
+task setupVirtualenv {
doLast {
+exec {
+ commandLine 'virtualenv', "${envdir}"
+}
exec {
executable 'sh'
- args '-c', 'which tox || pip install --user --upgrade tox'
+ args '-c', "source ${envdir}/bin/activate && pip install --upgrade tox"
}
}
+ outputs.files("$buildDir/.gradleenv/bin/tox")
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 87326)
Time Spent: 2h 10m (was: 2h)
> Migrate Python Jenkins PreCommits to Gradle
> ---
>
> Key: BEAM-3257
> URL: https://issues.apache.org/jira/browse/BEAM-3257
> Project: Beam
> Issue Type: Sub-task
> Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Code is here:
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[
https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=87320=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87320
]
ASF GitHub Bot logged work on BEAM-3257:
Author: ASF GitHub Bot
Created on: 03/Apr/18 22:42
Start Date: 03/Apr/18 22:42
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #5010:
[BEAM-3257] Add Python precommit gradle config
URL: https://github.com/apache/beam/pull/5010#discussion_r178982930
##
File path: .test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy
##
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import common_job_properties
+
+// This is the Python precommit which runs a Gradle build, and the current set
+// of precommit tests.
+job('beam_PreCommit_Python_GradleBuild') {
+ description('Runs Python PreCommit tests for the current GitHub Pull
Request.')
+
+ // Execute concurrent builds if necessary.
+ concurrentBuild()
+
+ // Set common parameters.
+ common_job_properties.setTopLevelMainJobProperties(
+delegate,
+'master',
+90)
+
+ // Publish all test results to Jenkins
+ publishers {
+archiveJunit('**/build/test-results/**/*.xml')
Review comment:
This doesn't seem to make sense. We should be publishing the python tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 87320)
Time Spent: 1h 50m (was: 1h 40m)
> Migrate Python Jenkins PreCommits to Gradle
> ---
>
> Key: BEAM-3257
> URL: https://issues.apache.org/jira/browse/BEAM-3257
> Project: Beam
> Issue Type: Sub-task
> Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Code is here:
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[
https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=87318=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87318
]
ASF GitHub Bot logged work on BEAM-3257:
Author: ASF GitHub Bot
Created on: 03/Apr/18 22:42
Start Date: 03/Apr/18 22:42
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #5010:
[BEAM-3257] Add Python precommit gradle config
URL: https://github.com/apache/beam/pull/5010#discussion_r178982403
##
File path: sdks/python/build.gradle
##
@@ -25,75 +25,123 @@ apply plugin: "base"
task test {}
check.dependsOn test
-task setupTest {
+def envdir = "${project.buildDir}/gradleenv"
+
+task setupVirtualenv {
doLast {
+exec {
+ commandLine 'virtualenv', "${envdir}"
+}
exec {
executable 'sh'
- args '-c', 'which tox || pip install --user --upgrade tox'
+ args '-c', "source ${envdir}/bin/activate && pip install --upgrade tox"
}
}
+ outputs.files("$buildDir/.gradleenv/bin/tox")
}
-task sdist {
+task sdist (dependsOn: 'setupVirtualenv') {
Review comment:
nit: remove the space between task name and arguments since its a method
call like `sdist(dependsOn: 'setupVirtualenv') {` here and below.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 87318)
Time Spent: 1.5h (was: 1h 20m)
> Migrate Python Jenkins PreCommits to Gradle
> ---
>
> Key: BEAM-3257
> URL: https://issues.apache.org/jira/browse/BEAM-3257
> Project: Beam
> Issue Type: Sub-task
> Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Code is here:
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[
https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=87319=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87319
]
ASF GitHub Bot logged work on BEAM-3257:
Author: ASF GitHub Bot
Created on: 03/Apr/18 22:42
Start Date: 03/Apr/18 22:42
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #5010:
[BEAM-3257] Add Python precommit gradle config
URL: https://github.com/apache/beam/pull/5010#discussion_r178981315
##
File path: sdks/python/build.gradle
##
@@ -25,75 +25,123 @@ apply plugin: "base"
task test {}
check.dependsOn test
-task setupTest {
+def envdir = "${project.buildDir}/gradleenv"
+
+task setupVirtualenv {
doLast {
+exec {
+ commandLine 'virtualenv', "${envdir}"
+}
exec {
executable 'sh'
- args '-c', 'which tox || pip install --user --upgrade tox'
+ args '-c', "source ${envdir}/bin/activate && pip install --upgrade tox"
}
}
+ outputs.files("$buildDir/.gradleenv/bin/tox")
Review comment:
re-use the definition of envdir here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 87319)
Time Spent: 1h 40m (was: 1.5h)
> Migrate Python Jenkins PreCommits to Gradle
> ---
>
> Key: BEAM-3257
> URL: https://issues.apache.org/jira/browse/BEAM-3257
> Project: Beam
> Issue Type: Sub-task
> Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Code is here:
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[ https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=87294=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87294 ] ASF GitHub Bot logged work on BEAM-3257: Author: ASF GitHub Bot Created on: 03/Apr/18 21:44 Start Date: 03/Apr/18 21:44 Worklog Time Spent: 10m Work Description: udim commented on issue #5010: [BEAM-3257] Add Python precommit gradle config URL: https://github.com/apache/beam/pull/5010#issuecomment-378409367 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 87294) Time Spent: 1h 20m (was: 1h 10m) > Migrate Python Jenkins PreCommits to Gradle > --- > > Key: BEAM-3257 > URL: https://issues.apache.org/jira/browse/BEAM-3257 > Project: Beam > Issue Type: Sub-task > Components: build-system, testing >Reporter: Luke Cwik >Assignee: Udi Meiri >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Code is here: > https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[ https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=87290=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87290 ] ASF GitHub Bot logged work on BEAM-3257: Author: ASF GitHub Bot Created on: 03/Apr/18 21:37 Start Date: 03/Apr/18 21:37 Worklog Time Spent: 10m Work Description: udim commented on issue #5010: [BEAM-3257] Add Python precommit gradle config URL: https://github.com/apache/beam/pull/5010#issuecomment-378407647 update seed job This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 87290) Time Spent: 1h 10m (was: 1h) > Migrate Python Jenkins PreCommits to Gradle > --- > > Key: BEAM-3257 > URL: https://issues.apache.org/jira/browse/BEAM-3257 > Project: Beam > Issue Type: Sub-task > Components: build-system, testing >Reporter: Luke Cwik >Assignee: Udi Meiri >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > Code is here: > https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[ https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=87280=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87280 ] ASF GitHub Bot logged work on BEAM-3257: Author: ASF GitHub Bot Created on: 03/Apr/18 21:15 Start Date: 03/Apr/18 21:15 Worklog Time Spent: 10m Work Description: udim commented on issue #5010: [BEAM-3257] Update Python Gradle tasks to run in a venv. URL: https://github.com/apache/beam/pull/5010#issuecomment-378401914 R: @lukecwik This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 87280) Time Spent: 1h (was: 50m) > Migrate Python Jenkins PreCommits to Gradle > --- > > Key: BEAM-3257 > URL: https://issues.apache.org/jira/browse/BEAM-3257 > Project: Beam > Issue Type: Sub-task > Components: build-system, testing >Reporter: Luke Cwik >Assignee: Udi Meiri >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Code is here: > https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[ https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=87278=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87278 ] ASF GitHub Bot logged work on BEAM-3257: Author: ASF GitHub Bot Created on: 03/Apr/18 21:11 Start Date: 03/Apr/18 21:11 Worklog Time Spent: 10m Work Description: udim opened a new pull request #5010: [BEAM-3257] Update Python Gradle tasks to run in a venv. URL: https://github.com/apache/beam/pull/5010 DESCRIPTION HERE Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [ ] Write a pull request description that is detailed enough to understand: - [ ] What the pull request does - [ ] Why it does it - [ ] How it does it - [ ] Why this approach - [ ] Each commit in the pull request should have a meaningful subject line and body. - [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 87278) Time Spent: 50m (was: 40m) > Migrate Python Jenkins PreCommits to Gradle > --- > > Key: BEAM-3257 > URL: https://issues.apache.org/jira/browse/BEAM-3257 > Project: Beam > Issue Type: Sub-task > Components: build-system, testing >Reporter: Luke Cwik >Assignee: Udi Meiri >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Code is here: > https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[
https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=87245=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87245
]
ASF GitHub Bot logged work on BEAM-3257:
Author: ASF GitHub Bot
Created on: 03/Apr/18 20:10
Start Date: 03/Apr/18 20:10
Worklog Time Spent: 10m
Work Description: alanmyrvold closed pull request #5006: [BEAM-3257]
Migrate Python Jenkins PreCommits to Gradle
URL: https://github.com/apache/beam/pull/5006
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.test-infra/jenkins/job_beam_PreCommit_Python_GradleInstall.groovy
b/.test-infra/jenkins/job_beam_PreCommit_Python_GradleInstall.groovy
new file mode 100644
index 000..20643654f31
--- /dev/null
+++ b/.test-infra/jenkins/job_beam_PreCommit_Python_GradleInstall.groovy
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import common_job_properties
+
+// This is the Python precommit which runs a gradle build, and the current set
+// of precommit tests.
+job('beam_PreCommit_Python_GradleInstall') {
+ description('Runs Python PreCommit tests for the current GitHub Pull
Request.')
+
+ // Execute concurrent builds if necessary.
+ concurrentBuild()
+
+ // Set common parameters.
+ common_job_properties.setTopLevelMainJobProperties(
+delegate,
+'master',
+150)
+
+ def gradle_switches = [
+// Gradle log verbosity enough to diagnose basic build issues
+"--info",
+// Continue the build even if there is a failure to show as many potential
failures as possible.
+'--continue',
+// Until we verify the build cache is working appropriately, force
rerunning all tasks
+'--rerun-tasks',
+ ]
+
+ def gradle_command_line = './gradlew ' + gradle_switches.join(' ') + '
:pythonPreCommit'
+ // Sets that this is a PreCommit job.
+ common_job_properties.setPreCommit(delegate, gradle_command_line, 'Run
Python Gradle PreCommit')
+ steps {
+gradle {
+ rootBuildScriptDir(common_job_properties.checkoutDir)
+ tasks(':pythonPreCommit')
+ for (String gradle_switch : gradle_switches) {
+switches(gradle_switch)
+ }
+}
+ }
+}
diff --git a/build.gradle b/build.gradle
index ad58db61eeb..6dae8225dbc 100644
--- a/build.gradle
+++ b/build.gradle
@@ -100,6 +100,7 @@ rat {
"**/dist/**/*",
"**/distribute-*/**/*",
"**/env/**/*",
+"sdks/python/.coverage",
"sdks/python/**/*.c",
"sdks/python/**/*.so",
"sdks/python/**/*.egg",
@@ -107,6 +108,7 @@ rat {
"sdks/python/NOTICE",
"sdks/python/README.md",
"sdks/python/apache_beam/portability/api/*pb2*.*",
+"sdks/python/apache_beam/portability/common_urns.py",
// .gitignore: Ignore IntelliJ files.
"**/.idea/**/*",
@@ -175,3 +177,11 @@ task goPreCommit() {
dependsOn ":rat"
dependsOn ":sdks:go:test"
}
+
+task pythonPreCommit() {
+ dependsOn ":rat"
+ dependsOn ":sdks:python:build"
+ dependsOn ":sdks:python:sdist"
+ dependsOn ":sdks:python:check"
+ dependsOn ":sdks:python:test"
+}
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 87245)
Time Spent: 40m (was: 0.5h)
> Migrate Python Jenkins PreCommits to Gradle
> ---
>
> Key: BEAM-3257
> URL: https://issues.apache.org/jira/browse/BEAM-3257
> Project: Beam
> Issue Type: Sub-task
> Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 40m
> Remaining Estimate:
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[ https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=87232=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87232 ] ASF GitHub Bot logged work on BEAM-3257: Author: ASF GitHub Bot Created on: 03/Apr/18 20:01 Start Date: 03/Apr/18 20:01 Worklog Time Spent: 10m Work Description: alanmyrvold commented on issue #5006: [BEAM-3257] Migrate Python Jenkins PreCommits to Gradle URL: https://github.com/apache/beam/pull/5006#issuecomment-378377370 Run Python Gradle PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 87232) Time Spent: 0.5h (was: 20m) > Migrate Python Jenkins PreCommits to Gradle > --- > > Key: BEAM-3257 > URL: https://issues.apache.org/jira/browse/BEAM-3257 > Project: Beam > Issue Type: Sub-task > Components: build-system, testing >Reporter: Luke Cwik >Assignee: Udi Meiri >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Code is here: > https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[ https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=87199=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87199 ] ASF GitHub Bot logged work on BEAM-3257: Author: ASF GitHub Bot Created on: 03/Apr/18 18:44 Start Date: 03/Apr/18 18:44 Worklog Time Spent: 10m Work Description: alanmyrvold commented on issue #5006: [BEAM-3257] Migrate Python Jenkins PreCommits to Gradle URL: https://github.com/apache/beam/pull/5006#issuecomment-378355321 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 87199) Time Spent: 20m (was: 10m) > Migrate Python Jenkins PreCommits to Gradle > --- > > Key: BEAM-3257 > URL: https://issues.apache.org/jira/browse/BEAM-3257 > Project: Beam > Issue Type: Sub-task > Components: build-system, testing >Reporter: Luke Cwik >Assignee: Udi Meiri >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Code is here: > https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle
[ https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=87198=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87198 ] ASF GitHub Bot logged work on BEAM-3257: Author: ASF GitHub Bot Created on: 03/Apr/18 18:44 Start Date: 03/Apr/18 18:44 Worklog Time Spent: 10m Work Description: alanmyrvold opened a new pull request #5006: [BEAM-3257] Migrate Python Jenkins PreCommits to Gradle URL: https://github.com/apache/beam/pull/5006 Migrate Python Jenkins PreCommits to Gradle Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [ ] Write a pull request description that is detailed enough to understand: - [ ] What the pull request does - [ ] Why it does it - [ ] How it does it - [ ] Why this approach - [ ] Each commit in the pull request should have a meaningful subject line and body. - [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 87198) Time Spent: 10m Remaining Estimate: 0h > Migrate Python Jenkins PreCommits to Gradle > --- > > Key: BEAM-3257 > URL: https://issues.apache.org/jira/browse/BEAM-3257 > Project: Beam > Issue Type: Sub-task > Components: build-system, testing >Reporter: Luke Cwik >Assignee: Udi Meiri >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Code is here: > https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
