The first concern for Spark will probably be to ensure that we still build and test against Python 2.6, since that's the minimum version of Python we support.
Otherwise this seems OK. We use numpy and other Python packages in PySpark, but I don't think we're pinned to any particular version of those packages. Nick On Mon Feb 23 2015 at 2:15:19 PM shane knapp <skn...@berkeley.edu> wrote: > good morning, developers! > > TL;DR: > > i will be installing anaconda and setting it in the system PATH so that > your python will default to 2.7, as well as it taking over management of > all of the sci-py packages. this is potentially a big change, so i'll be > testing locally on my staging instance before deployment to the wide world. > > deployment is *tentatively* next monday, march 2nd. > > a little background: > > the jenkins test infra is currently (and happily) managed by a set of tools > that allow me to set up and deploy new workers, manage their packages and > make sure that all spark and research projects can happily and successfully > build. > > we're currently at the state where ~50 or so packages are installed and > configured on each worker. this is getting a little cumbersome, as the > package-to-build dep tree is getting pretty large. > > the biggest offender is the science-based python infrastructure. > everything is blindly installed w/yum and pip, so it's hard to control > *exactly* what version of any given library is as compared to what's on a > dev's laptop. > > the solution: > > anaconda (https://store.continuum.io/cshop/anaconda/)! everything is > centralized! i can manage specific versions much easier! > > what this means to you: > > * python 2.7 will be the default system python. > * 2.6 will still be installed and available (/usr/bin/python or > /usr/bin/python/2.6) > > what you need to do: > * install anaconda, have it update your PATH > * build locally and try to fix any bugs (for spark, this "should just > work") > * if you have problems, reach out to me and i'll see what i can do to help. > if we can't get your stuff running under python2.7, we can default to 2.6 > via a job config change. > > what i will be doing: > * setting up anaconda on my staging instance and spot-testing a lot of > builds before deployment > > please let me know if there are any issues/concerns... i'll be posting > updates this week and will let everyone know if there are any changes to > the Plan[tm]. > > your friendly devops engineer, > > shane >
