[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[ https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=191841&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-191841 ] ASF GitHub Bot logged work on BEAM-6024: Author: ASF GitHub Bot Created on: 29/Jan/19 18:12 Start Date: 29/Jan/19 18:12 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #7423: [BEAM-6024] Build and verify Python 3 container image with Gradle URL: https://github.com/apache/beam/pull/7423 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 191841) Time Spent: 3h 50m (was: 3h 40m) > Gradle setupVirtualenv supports Python 3 > > > Key: BEAM-6024 > URL: https://issues.apache.org/jira/browse/BEAM-6024 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Mark Liu >Assignee: Mark Liu >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > Need to depend on Python 3 virtualenv in few places: > - Build Dataflow worker container in Python 3 > - Run ValidatesRunner and integration tests on Jenkins -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[
https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=191228&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-191228
]
ASF GitHub Bot logged work on BEAM-6024:
Author: ASF GitHub Bot
Created on: 28/Jan/19 19:41
Start Date: 28/Jan/19 19:41
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #7423: [BEAM-6024]
Build Python 3 container image with Gradle
URL: https://github.com/apache/beam/pull/7423#discussion_r251564298
##
File path:
.test-infra/jenkins/job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
##
@@ -31,5 +31,6 @@
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Py_ValCont',
// Execute shell command to test Python SDK.
steps {
shell('cd ' + commonJobProperties.checkoutDir + ' && bash
sdks/python/container/run_validatescontainer.sh')
+shell('cd ' + commonJobProperties.checkoutDir + ' && bash
sdks/python/container/run_validatescontainer_py3.sh')
Review comment:
Ok, we can add a TODO[...] and take care of this later.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 191228)
Time Spent: 3h 40m (was: 3.5h)
> Gradle setupVirtualenv supports Python 3
>
>
> Key: BEAM-6024
> URL: https://issues.apache.org/jira/browse/BEAM-6024
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-harness
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Need to depend on Python 3 virtualenv in few places:
> - Build Dataflow worker container in Python 3
> - Run ValidatesRunner and integration tests on Jenkins
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[ https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=191129&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-191129 ] ASF GitHub Bot logged work on BEAM-6024: Author: ASF GitHub Bot Created on: 28/Jan/19 17:08 Start Date: 28/Jan/19 17:08 Worklog Time Spent: 10m Work Description: markflyhigh commented on issue #7423: [BEAM-6024] Build Python 3 container image with Gradle URL: https://github.com/apache/beam/pull/7423#issuecomment-458216860 @tvalentyn PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 191129) Time Spent: 3.5h (was: 3h 20m) > Gradle setupVirtualenv supports Python 3 > > > Key: BEAM-6024 > URL: https://issues.apache.org/jira/browse/BEAM-6024 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Mark Liu >Assignee: Mark Liu >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > Need to depend on Python 3 virtualenv in few places: > - Build Dataflow worker container in Python 3 > - Run ValidatesRunner and integration tests on Jenkins -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[ https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=190260&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-190260 ] ASF GitHub Bot logged work on BEAM-6024: Author: ASF GitHub Bot Created on: 26/Jan/19 00:06 Start Date: 26/Jan/19 00:06 Worklog Time Spent: 10m Work Description: markflyhigh commented on issue #7423: [BEAM-6024] Build Python 3 container image with Gradle URL: https://github.com/apache/beam/pull/7423#issuecomment-457776801 Run Python Dataflow ValidatesContainer This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 190260) Time Spent: 3h 20m (was: 3h 10m) > Gradle setupVirtualenv supports Python 3 > > > Key: BEAM-6024 > URL: https://issues.apache.org/jira/browse/BEAM-6024 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Mark Liu >Assignee: Mark Liu >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > Need to depend on Python 3 virtualenv in few places: > - Build Dataflow worker container in Python 3 > - Run ValidatesRunner and integration tests on Jenkins -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[ https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=190257&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-190257 ] ASF GitHub Bot logged work on BEAM-6024: Author: ASF GitHub Bot Created on: 25/Jan/19 23:57 Start Date: 25/Jan/19 23:57 Worklog Time Spent: 10m Work Description: markflyhigh commented on issue #7423: [BEAM-6024] Build Python 3 container image with Gradle URL: https://github.com/apache/beam/pull/7423#issuecomment-457775244 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 190257) Time Spent: 3h 10m (was: 3h) > Gradle setupVirtualenv supports Python 3 > > > Key: BEAM-6024 > URL: https://issues.apache.org/jira/browse/BEAM-6024 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Mark Liu >Assignee: Mark Liu >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > Need to depend on Python 3 virtualenv in few places: > - Build Dataflow worker container in Python 3 > - Run ValidatesRunner and integration tests on Jenkins -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[ https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=190140&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-190140 ] ASF GitHub Bot logged work on BEAM-6024: Author: ASF GitHub Bot Created on: 25/Jan/19 18:42 Start Date: 25/Jan/19 18:42 Worklog Time Spent: 10m Work Description: markflyhigh commented on issue #7423: [BEAM-6024] Build Python 3 container image with Gradle URL: https://github.com/apache/beam/pull/7423#issuecomment-457677868 from console output ([link](https://builds.apache.org/job/beam_PostCommit_Py_ValCont_PR/12/console)): ``` 00:02:46.412 > Task :beam-sdks-python-container:docker 01:40:11.674 Build timed out (after 100 minutes). Marking the build as aborted. ``` Gradle process was stuck when building the first python 2 image which took ~1h37mins. No detail logs can explain that. I think we can add `--info` to [gradle command](https://github.com/apache/beam/blob/master/sdks/python/container/run_validatescontainer.sh#L52) to enable more logs. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 190140) Time Spent: 3h (was: 2h 50m) > Gradle setupVirtualenv supports Python 3 > > > Key: BEAM-6024 > URL: https://issues.apache.org/jira/browse/BEAM-6024 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Mark Liu >Assignee: Mark Liu >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > Need to depend on Python 3 virtualenv in few places: > - Build Dataflow worker container in Python 3 > - Run ValidatesRunner and integration tests on Jenkins -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[ https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=190124&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-190124 ] ASF GitHub Bot logged work on BEAM-6024: Author: ASF GitHub Bot Created on: 25/Jan/19 18:11 Start Date: 25/Jan/19 18:11 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #7423: [BEAM-6024] Build Python 3 container image with Gradle URL: https://github.com/apache/beam/pull/7423#issuecomment-457666108 100 min sounds doable for building two containers and running two wordcount tests... or is there more to this suite? But parallelizing would help. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 190124) Time Spent: 2h 50m (was: 2h 40m) > Gradle setupVirtualenv supports Python 3 > > > Key: BEAM-6024 > URL: https://issues.apache.org/jira/browse/BEAM-6024 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Mark Liu >Assignee: Mark Liu >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > Need to depend on Python 3 virtualenv in few places: > - Build Dataflow worker container in Python 3 > - Run ValidatesRunner and integration tests on Jenkins -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[ https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=190123&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-190123 ] ASF GitHub Bot logged work on BEAM-6024: Author: ASF GitHub Bot Created on: 25/Jan/19 18:06 Start Date: 25/Jan/19 18:06 Worklog Time Spent: 10m Work Description: markflyhigh commented on issue #7423: [BEAM-6024] Build Python 3 container image with Gradle URL: https://github.com/apache/beam/pull/7423#issuecomment-457664338 `Google Cloud Dataflow Runner Python ValidatesContainer Tests` failed due to exceed Jenkins timeout (100mins). Should find a way to run build+tests in parallel or in separate Jenkins job. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 190123) Time Spent: 2h 40m (was: 2.5h) > Gradle setupVirtualenv supports Python 3 > > > Key: BEAM-6024 > URL: https://issues.apache.org/jira/browse/BEAM-6024 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Mark Liu >Assignee: Mark Liu >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > Need to depend on Python 3 virtualenv in few places: > - Build Dataflow worker container in Python 3 > - Run ValidatesRunner and integration tests on Jenkins -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[ https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=189863&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-189863 ] ASF GitHub Bot logged work on BEAM-6024: Author: ASF GitHub Bot Created on: 25/Jan/19 06:46 Start Date: 25/Jan/19 06:46 Worklog Time Spent: 10m Work Description: markflyhigh commented on issue #7423: [BEAM-6024] Build Python 3 container image with Gradle URL: https://github.com/apache/beam/pull/7423#issuecomment-457473035 Run Python Dataflow ValidatesContainer This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 189863) Time Spent: 2.5h (was: 2h 20m) > Gradle setupVirtualenv supports Python 3 > > > Key: BEAM-6024 > URL: https://issues.apache.org/jira/browse/BEAM-6024 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Mark Liu >Assignee: Mark Liu >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > > Need to depend on Python 3 virtualenv in few places: > - Build Dataflow worker container in Python 3 > - Run ValidatesRunner and integration tests on Jenkins -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[ https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=189857&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-189857 ] ASF GitHub Bot logged work on BEAM-6024: Author: ASF GitHub Bot Created on: 25/Jan/19 06:05 Start Date: 25/Jan/19 06:05 Worklog Time Spent: 10m Work Description: markflyhigh commented on issue #7423: [BEAM-6024] Build Python 3 container image with Gradle URL: https://github.com/apache/beam/pull/7423#issuecomment-457466225 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 189857) Time Spent: 2h 20m (was: 2h 10m) > Gradle setupVirtualenv supports Python 3 > > > Key: BEAM-6024 > URL: https://issues.apache.org/jira/browse/BEAM-6024 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Mark Liu >Assignee: Mark Liu >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > Need to depend on Python 3 virtualenv in few places: > - Build Dataflow worker container in Python 3 > - Run ValidatesRunner and integration tests on Jenkins -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[ https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=189723&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-189723 ] ASF GitHub Bot logged work on BEAM-6024: Author: ASF GitHub Bot Created on: 24/Jan/19 22:51 Start Date: 24/Jan/19 22:51 Worklog Time Spent: 10m Work Description: markflyhigh commented on issue #7423: [BEAM-6024] Build Python 3 container image with Gradle URL: https://github.com/apache/beam/pull/7423#issuecomment-457388800 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 189723) Time Spent: 2h 10m (was: 2h) > Gradle setupVirtualenv supports Python 3 > > > Key: BEAM-6024 > URL: https://issues.apache.org/jira/browse/BEAM-6024 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Mark Liu >Assignee: Mark Liu >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > Need to depend on Python 3 virtualenv in few places: > - Build Dataflow worker container in Python 3 > - Run ValidatesRunner and integration tests on Jenkins -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[ https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=189560&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-189560 ] ASF GitHub Bot logged work on BEAM-6024: Author: ASF GitHub Bot Created on: 24/Jan/19 17:06 Start Date: 24/Jan/19 17:06 Worklog Time Spent: 10m Work Description: markflyhigh commented on issue #7423: [BEAM-6024] Build Python 3 container image with Gradle URL: https://github.com/apache/beam/pull/7423#issuecomment-457276420 Run Python Dataflow ValidatesContainer This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 189560) Time Spent: 2h (was: 1h 50m) > Gradle setupVirtualenv supports Python 3 > > > Key: BEAM-6024 > URL: https://issues.apache.org/jira/browse/BEAM-6024 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Mark Liu >Assignee: Mark Liu >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > Need to depend on Python 3 virtualenv in few places: > - Build Dataflow worker container in Python 3 > - Run ValidatesRunner and integration tests on Jenkins -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[
https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=189306&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-189306
]
ASF GitHub Bot logged work on BEAM-6024:
Author: ASF GitHub Bot
Created on: 24/Jan/19 04:17
Start Date: 24/Jan/19 04:17
Worklog Time Spent: 10m
Work Description: markflyhigh commented on pull request #7423:
[BEAM-6024] Build Python 3 container image with Gradle
URL: https://github.com/apache/beam/pull/7423#discussion_r250461928
##
File path:
.test-infra/jenkins/job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
##
@@ -31,5 +31,6 @@
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Py_ValCont',
// Execute shell command to test Python SDK.
steps {
shell('cd ' + commonJobProperties.checkoutDir + ' && bash
sdks/python/container/run_validatescontainer.sh')
+shell('cd ' + commonJobProperties.checkoutDir + ' && bash
sdks/python/container/run_validatescontainer_py3.sh')
Review comment:
I'm not sure if Gradle build step on each other if we build containers at
same time. Probably a better approach is to rewrite run scripts in Gradle since
building container is a gradle task and docker has gradle plugin.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 189306)
Time Spent: 1h 50m (was: 1h 40m)
> Gradle setupVirtualenv supports Python 3
>
>
> Key: BEAM-6024
> URL: https://issues.apache.org/jira/browse/BEAM-6024
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-harness
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Need to depend on Python 3 virtualenv in few places:
> - Build Dataflow worker container in Python 3
> - Run ValidatesRunner and integration tests on Jenkins
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[
https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=186627&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-186627
]
ASF GitHub Bot logged work on BEAM-6024:
Author: ASF GitHub Bot
Created on: 17/Jan/19 23:50
Start Date: 17/Jan/19 23:50
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #7423: [BEAM-6024]
Build Python 3 container image with Gradle
URL: https://github.com/apache/beam/pull/7423#discussion_r248884716
##
File path:
.test-infra/jenkins/job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
##
@@ -31,5 +31,6 @@
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Py_ValCont',
// Execute shell command to test Python SDK.
steps {
shell('cd ' + commonJobProperties.checkoutDir + ' && bash
sdks/python/container/run_validatescontainer.sh')
+shell('cd ' + commonJobProperties.checkoutDir + ' && bash
sdks/python/container/run_validatescontainer_py3.sh')
Review comment:
I would suggest spending 30 min to see if it's possible to start this in
parallel, and if it works as intended. But can also keep as is.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 186627)
Time Spent: 1.5h (was: 1h 20m)
> Gradle setupVirtualenv supports Python 3
>
>
> Key: BEAM-6024
> URL: https://issues.apache.org/jira/browse/BEAM-6024
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-harness
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Need to depend on Python 3 virtualenv in few places:
> - Build Dataflow worker container in Python 3
> - Run ValidatesRunner and integration tests on Jenkins
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[
https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=186628&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-186628
]
ASF GitHub Bot logged work on BEAM-6024:
Author: ASF GitHub Bot
Created on: 17/Jan/19 23:50
Start Date: 17/Jan/19 23:50
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #7423: [BEAM-6024]
Build Python 3 container image with Gradle
URL: https://github.com/apache/beam/pull/7423#discussion_r248882024
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1554,9 +1554,20 @@ class BeamModulePlugin implements Plugin {
project.ext.envdir =
"${project.rootProject.buildDir}/gradleenv/${project.name.hashCode()}"
project.ext.pythonRootDir = "${project.rootDir}/sdks/python"
+ // This is current supported Python3 version. It should match the one in
+ // sdks/python/container/py3/Dockerfile
+ final PYTHON3_VERSION = 35
Review comment:
Should this be 35 or `3.5`?
virtualenv documentation says:
```
-p PYTHON_EXE, --python=PYTHON_EXE
The Python interpreter to use, e.g.,
--python=python3.5 will use the python3.5 interpreter
to create the new environment. The default is the
interpreter that virtualenv was installed with
(/usr/bin/python3.5)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 186628)
Time Spent: 1h 40m (was: 1.5h)
> Gradle setupVirtualenv supports Python 3
>
>
> Key: BEAM-6024
> URL: https://issues.apache.org/jira/browse/BEAM-6024
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-harness
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Need to depend on Python 3 virtualenv in few places:
> - Build Dataflow worker container in Python 3
> - Run ValidatesRunner and integration tests on Jenkins
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[ https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=185892&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185892 ] ASF GitHub Bot logged work on BEAM-6024: Author: ASF GitHub Bot Created on: 16/Jan/19 17:24 Start Date: 16/Jan/19 17:24 Worklog Time Spent: 10m Work Description: markflyhigh commented on issue #7423: [BEAM-6024] Build Python 3 container image with Gradle URL: https://github.com/apache/beam/pull/7423#issuecomment-454865616 PTAL @tvalentyn This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185892) Time Spent: 1h 20m (was: 1h 10m) > Gradle setupVirtualenv supports Python 3 > > > Key: BEAM-6024 > URL: https://issues.apache.org/jira/browse/BEAM-6024 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Mark Liu >Assignee: Mark Liu >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Need to depend on Python 3 virtualenv in few places: > - Build Dataflow worker container in Python 3 > - Run ValidatesRunner and integration tests on Jenkins -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[
https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=185540&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185540
]
ASF GitHub Bot logged work on BEAM-6024:
Author: ASF GitHub Bot
Created on: 16/Jan/19 01:49
Start Date: 16/Jan/19 01:49
Worklog Time Spent: 10m
Work Description: markflyhigh commented on pull request #7423:
[BEAM-6024] Build Python 3 container image with Gradle
URL: https://github.com/apache/beam/pull/7423#discussion_r248127767
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1523,7 +1523,14 @@ artifactId=${project.name}
project.task('setupVirtualenv') {
doLast {
- project.exec { commandLine 'virtualenv', "${project.ext.envdir}" }
+ def virtualenvCmd = [
+'virtualenv',
+"${project.ext.envdir}",
+ ]
+ if (project.hasProperty('python3')) {
+virtualenvCmd += '--python=python3'
Review comment:
You are right. We should consider other version of python 3 in the future.
Currently I don't have a good answer yet. So let's use `python3` in general and
specify `--python=python35` (I think this is the right format) in virtualenv
cmd. A constant with comments sounds good.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 185540)
Time Spent: 1h 10m (was: 1h)
> Gradle setupVirtualenv supports Python 3
>
>
> Key: BEAM-6024
> URL: https://issues.apache.org/jira/browse/BEAM-6024
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-harness
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Need to depend on Python 3 virtualenv in few places:
> - Build Dataflow worker container in Python 3
> - Run ValidatesRunner and integration tests on Jenkins
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[
https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=185536&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185536
]
ASF GitHub Bot logged work on BEAM-6024:
Author: ASF GitHub Bot
Created on: 16/Jan/19 01:33
Start Date: 16/Jan/19 01:33
Worklog Time Spent: 10m
Work Description: markflyhigh commented on pull request #7423:
[BEAM-6024] Build Python 3 container image with Gradle
URL: https://github.com/apache/beam/pull/7423#discussion_r248125099
##
File path:
.test-infra/jenkins/job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
##
@@ -31,5 +31,6 @@
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Py_ValCont',
// Execute shell command to test Python SDK.
steps {
shell('cd ' + commonJobProperties.checkoutDir + ' && bash
sdks/python/container/run_validatescontainer.sh')
+shell('cd ' + commonJobProperties.checkoutDir + ' && bash
sdks/python/container/run_validatescontainer_py3.sh')
Review comment:
Jenkins DSL may have a way to run multiple steps in parallel:
https://stackoverflow.com/a/36893393
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 185536)
Time Spent: 50m (was: 40m)
> Gradle setupVirtualenv supports Python 3
>
>
> Key: BEAM-6024
> URL: https://issues.apache.org/jira/browse/BEAM-6024
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-harness
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Need to depend on Python 3 virtualenv in few places:
> - Build Dataflow worker container in Python 3
> - Run ValidatesRunner and integration tests on Jenkins
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[
https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=185537&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185537
]
ASF GitHub Bot logged work on BEAM-6024:
Author: ASF GitHub Bot
Created on: 16/Jan/19 01:33
Start Date: 16/Jan/19 01:33
Worklog Time Spent: 10m
Work Description: markflyhigh commented on pull request #7423:
[BEAM-6024] Build Python 3 container image with Gradle
URL: https://github.com/apache/beam/pull/7423#discussion_r248125099
##
File path:
.test-infra/jenkins/job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
##
@@ -31,5 +31,6 @@
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Py_ValCont',
// Execute shell command to test Python SDK.
steps {
shell('cd ' + commonJobProperties.checkoutDir + ' && bash
sdks/python/container/run_validatescontainer.sh')
+shell('cd ' + commonJobProperties.checkoutDir + ' && bash
sdks/python/container/run_validatescontainer_py3.sh')
Review comment:
Jenkins DSL may have a way to run multiple steps in parallel:
https://stackoverflow.com/a/36893393. But I never tried before.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 185537)
Time Spent: 1h (was: 50m)
> Gradle setupVirtualenv supports Python 3
>
>
> Key: BEAM-6024
> URL: https://issues.apache.org/jira/browse/BEAM-6024
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-harness
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Need to depend on Python 3 virtualenv in few places:
> - Build Dataflow worker container in Python 3
> - Run ValidatesRunner and integration tests on Jenkins
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[
https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=184293&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184293
]
ASF GitHub Bot logged work on BEAM-6024:
Author: ASF GitHub Bot
Created on: 11/Jan/19 18:37
Start Date: 11/Jan/19 18:37
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #7423: [BEAM-6024]
Build Python 3 container image with Gradle
URL: https://github.com/apache/beam/pull/7423#discussion_r247209615
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1523,7 +1523,14 @@ artifactId=${project.name}
project.task('setupVirtualenv') {
doLast {
- project.exec { commandLine 'virtualenv', "${project.ext.envdir}" }
+ def virtualenvCmd = [
+'virtualenv',
+"${project.ext.envdir}",
+ ]
+ if (project.hasProperty('python3')) {
+virtualenvCmd += '--python=python3'
Review comment:
In the future, I'd like us to be able to customize which Python 3 version to
use (3.5, 3.6, etc). Dockerfile can take an argument for the FROM clause as
well. This can happen in the future PR, but for the scope of this PR I'd like
to make sure that we use the same Python 3 version to build the SDK as we use
in the Dockerfile. So we I think should pass `--python=python3.5`, but we could
make this a constant somewhere up in the file and make a note that the version
should match the version we [inside the Docker
container](https://github.com/apache/beam/blob/3a531323322b3f5bbd695b64744a0603a1c97607/sdks/python/container/py3/Dockerfile#L19).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184293)
Time Spent: 40m (was: 0.5h)
> Gradle setupVirtualenv supports Python 3
>
>
> Key: BEAM-6024
> URL: https://issues.apache.org/jira/browse/BEAM-6024
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-harness
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Need to depend on Python 3 virtualenv in few places:
> - Build Dataflow worker container in Python 3
> - Run ValidatesRunner and integration tests on Jenkins
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[
https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=184292&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184292
]
ASF GitHub Bot logged work on BEAM-6024:
Author: ASF GitHub Bot
Created on: 11/Jan/19 18:37
Start Date: 11/Jan/19 18:37
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #7423: [BEAM-6024]
Build Python 3 container image with Gradle
URL: https://github.com/apache/beam/pull/7423#discussion_r247207378
##
File path:
.test-infra/jenkins/job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
##
@@ -31,5 +31,6 @@
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Py_ValCont',
// Execute shell command to test Python SDK.
steps {
shell('cd ' + commonJobProperties.checkoutDir + ' && bash
sdks/python/container/run_validatescontainer.sh')
+shell('cd ' + commonJobProperties.checkoutDir + ' && bash
sdks/python/container/run_validatescontainer_py3.sh')
Review comment:
Is it possible to run these steps in parallel?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 184292)
Time Spent: 40m (was: 0.5h)
> Gradle setupVirtualenv supports Python 3
>
>
> Key: BEAM-6024
> URL: https://issues.apache.org/jira/browse/BEAM-6024
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-harness
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Need to depend on Python 3 virtualenv in few places:
> - Build Dataflow worker container in Python 3
> - Run ValidatesRunner and integration tests on Jenkins
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[ https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=183480&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-183480 ] ASF GitHub Bot logged work on BEAM-6024: Author: ASF GitHub Bot Created on: 10/Jan/19 01:22 Start Date: 10/Jan/19 01:22 Worklog Time Spent: 10m Work Description: markflyhigh commented on issue #7423: [BEAM-6024] Build Python 3 container image with Gradle URL: https://github.com/apache/beam/pull/7423#issuecomment-452934776 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 183480) Time Spent: 20m (was: 10m) > Gradle setupVirtualenv supports Python 3 > > > Key: BEAM-6024 > URL: https://issues.apache.org/jira/browse/BEAM-6024 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Mark Liu >Assignee: Mark Liu >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Need to depend on Python 3 virtualenv in few places: > - Build Dataflow worker container in Python 3 > - Run ValidatesRunner and integration tests on Jenkins -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[ https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=183481&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-183481 ] ASF GitHub Bot logged work on BEAM-6024: Author: ASF GitHub Bot Created on: 10/Jan/19 01:27 Start Date: 10/Jan/19 01:27 Worklog Time Spent: 10m Work Description: markflyhigh commented on issue #7423: [BEAM-6024] Build Python 3 container image with Gradle URL: https://github.com/apache/beam/pull/7423#issuecomment-452935807 Run Python Dataflow ValidatesContainer This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 183481) Time Spent: 0.5h (was: 20m) > Gradle setupVirtualenv supports Python 3 > > > Key: BEAM-6024 > URL: https://issues.apache.org/jira/browse/BEAM-6024 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Mark Liu >Assignee: Mark Liu >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Need to depend on Python 3 virtualenv in few places: > - Build Dataflow worker container in Python 3 > - Run ValidatesRunner and integration tests on Jenkins -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6024) Gradle setupVirtualenv supports Python 3
[ https://issues.apache.org/jira/browse/BEAM-6024?focusedWorklogId=181318&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-181318 ] ASF GitHub Bot logged work on BEAM-6024: Author: ASF GitHub Bot Created on: 04/Jan/19 22:56 Start Date: 04/Jan/19 22:56 Worklog Time Spent: 10m Work Description: markflyhigh commented on pull request #7423: [BEAM-6024] Build Python 3 container image with Gradle URL: https://github.com/apache/beam/pull/7423 With this change, people can build Python 3 container image with single command: `./gradlew :beam-sdks-python-container-py3:docker -Ppython3` Changes are: - `:beam-sdks-python:setupVirtualenv` will create a python 3 virtualenv instead of python 2 (by default) if `-Ppython3` is specified. - pip install requires "--process-dependency-links" flag when running in Python 3 due to changes to dill in https://github.com/apache/beam/pull/7104. The flag can be removed once BEAM-6135 is done. This also affects Python 3 Dockerfile. - Add validation of building Python 3 container to one of existing Jenkins PostCommit. @tvalentyn Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apa
