[
https://issues.apache.org/jira/browse/SPARK-25079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820364#comment-16820364
]
shane knapp commented on SPARK-25079:
-------------------------------------
from my email to dev@:
ok.
after much wailing and gnashing of teeth (and conversations w/[~bryanc]), i
think we're coming to a general idea of how python testing will soon work!
i propose the following:
py27: master, 2.3, 2.4
py36 + pandas 0.19.2 + pyarrow 0.8.0: 2.3, 2.4
py36 + pandas 0.24.2 + pyarrow 0.12.1: master
all of the above combinations have been tested (locally) and pass. i will need
to create/deploy the new 2.3/4 branch python envs and then test my two PRs
against them.
the good:
1) this, IMO, will get us to a place where we can get all spark python tests
using py36 as quickly as possible w/o needing to backport and spend a ton of
time fixing 2.3/4 tests.
2) there is literally *one* hardcoded path (in dev/run-tests.py) that needs to
be updated on 2.3/4 to point to a different python env than 'py3k'.
the bad:
1) three python envs to deal with (with the env supporting 2.3 and 2.4
remaining relatively static).
since the 'good' definitely outweighs the 'bad', my vote is for 'good'. ;)
also: i am putting my foot down and we won't be testing against more than
three python envs!
> [PYTHON] upgrade python 3.4 -> 3.6
> ----------------------------------
>
> Key: SPARK-25079
> URL: https://issues.apache.org/jira/browse/SPARK-25079
> Project: Spark
> Issue Type: Improvement
> Components: Build, PySpark
> Affects Versions: 2.3.1
> Reporter: shane knapp
> Assignee: shane knapp
> Priority: Major
>
> for the impending arrow upgrade
> (https://issues.apache.org/jira/browse/SPARK-23874) we need to bump python
> 3.4 -> 3.5.
> i have been testing this here:
> [https://amplab.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/|https://amplab.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/69]
> my methodology:
> 1) upgrade python + arrow to 3.5 and 0.10.0
> 2) run python tests
> 3) when i'm happy that Things Won't Explode Spectacularly, pause jenkins and
> upgrade centos workers to python3.5
> 4) simultaneously do the following:
> - create a symlink in /home/anaconda/envs/py3k/bin for python3.4 that
> points to python3.5 (this is currently being tested here:
> [https://amplab.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/69)]
> - push a change to python/run-tests.py replacing 3.4 with 3.5
> 5) once the python3.5 change to run-tests.py is merged, we will need to
> back-port this to all existing branches
> 6) then and only then can i remove the python3.4 -> python3.5 symlink
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
