[
https://issues.apache.org/jira/browse/SPARK-25079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675866#comment-16675866
]
Bryan Cutler edited comment on SPARK-25079 at 11/5/18 11:09 PM:
----------------------------------------------------------------
Sounds like a good plan [~shaneknapp]\! The instances of python3.4 in
SparkLauncherSuite and SparkSubmitSuite look like they are for testing that the
env vars PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON can be set to different
Python versions. So I don't think they need to be changed, but if you want to
maybe change python3.4 -> python3.5 and also python3.5 -> python3.6 to keep
them different. Everything else looks good!
was (Author: bryanc):
Sounds like a good plan [~shaneknapp] ! The instances of python3.4 in
SparkLauncherSuite and SparkSubmitSuite look like they are for testing that the
env vars PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON can be set to different
Python versions. So I don't think they need to be changed, but if you want to
maybe change python3.4 -> python3.5 and also python3.5 -> python3.6 to keep
them different. Everything else looks good!
> [PYTHON] upgrade python 3.4 -> 3.5
> ----------------------------------
>
> Key: SPARK-25079
> URL: https://issues.apache.org/jira/browse/SPARK-25079
> Project: Spark
> Issue Type: Improvement
> Components: Build, PySpark
> Affects Versions: 2.3.1
> Reporter: shane knapp
> Assignee: shane knapp
> Priority: Major
>
> for the impending arrow upgrade
> (https://issues.apache.org/jira/browse/SPARK-23874) we need to bump python
> 3.4 -> 3.5.
> i have been testing this here:
> [https://amplab.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/|https://amplab.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/69]
> my methodology:
> 1) upgrade python + arrow to 3.5 and 0.10.0
> 2) run python tests
> 3) when i'm happy that Things Won't Explode Spectacularly, pause jenkins and
> upgrade centos workers to python3.5
> 4) simultaneously do the following:
> - create a symlink in /home/anaconda/envs/py3k/bin for python3.4 that
> points to python3.5 (this is currently being tested here:
> [https://amplab.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/69)]
> - push a change to python/run-tests.py replacing 3.4 with 3.5
> 5) once the python3.5 change to run-tests.py is merged, we will need to
> back-port this to all existing branches
> 6) then and only then can i remove the python3.4 -> python3.5 symlink
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
