[jenkins infra -- pls read ] installing anaconda, moving default python from 2.6 -> 2.7
good morning, developers! TL;DR: i will be installing anaconda and setting it in the system PATH so that your python will default to 2.7, as well as it taking over management of all of the sci-py packages. this is potentially a big change, so i'll be testing locally on my staging instance before deployment to the wide world. deployment is *tentatively* next monday, march 2nd. a little background: the jenkins test infra is currently (and happily) managed by a set of tools that allow me to set up and deploy new workers, manage their packages and make sure that all spark and research projects can happily and successfully build. we're currently at the state where ~50 or so packages are installed and configured on each worker. this is getting a little cumbersome, as the package-to-build dep tree is getting pretty large. the biggest offender is the science-based python infrastructure. everything is blindly installed w/yum and pip, so it's hard to control *exactly* what version of any given library is as compared to what's on a dev's laptop. the solution: anaconda (https://store.continuum.io/cshop/anaconda/)! everything is centralized! i can manage specific versions much easier! what this means to you: * python 2.7 will be the default system python. * 2.6 will still be installed and available (/usr/bin/python or /usr/bin/python/2.6) what you need to do: * install anaconda, have it update your PATH * build locally and try to fix any bugs (for spark, this "should just work") * if you have problems, reach out to me and i'll see what i can do to help. if we can't get your stuff running under python2.7, we can default to 2.6 via a job config change. what i will be doing: * setting up anaconda on my staging instance and spot-testing a lot of builds before deployment please let me know if there are any issues/concerns... i'll be posting updates this week and will let everyone know if there are any changes to the Plan[tm]. your friendly devops engineer, shane
Re: [jenkins infra -- pls read ] installing anaconda, moving default python from 2.6 -> 2.7
The first concern for Spark will probably be to ensure that we still build and test against Python 2.6, since that's the minimum version of Python we support. Otherwise this seems OK. We use numpy and other Python packages in PySpark, but I don't think we're pinned to any particular version of those packages. Nick On Mon Feb 23 2015 at 2:15:19 PM shane knapp wrote: > good morning, developers! > > TL;DR: > > i will be installing anaconda and setting it in the system PATH so that > your python will default to 2.7, as well as it taking over management of > all of the sci-py packages. this is potentially a big change, so i'll be > testing locally on my staging instance before deployment to the wide world. > > deployment is *tentatively* next monday, march 2nd. > > a little background: > > the jenkins test infra is currently (and happily) managed by a set of tools > that allow me to set up and deploy new workers, manage their packages and > make sure that all spark and research projects can happily and successfully > build. > > we're currently at the state where ~50 or so packages are installed and > configured on each worker. this is getting a little cumbersome, as the > package-to-build dep tree is getting pretty large. > > the biggest offender is the science-based python infrastructure. > everything is blindly installed w/yum and pip, so it's hard to control > *exactly* what version of any given library is as compared to what's on a > dev's laptop. > > the solution: > > anaconda (https://store.continuum.io/cshop/anaconda/)! everything is > centralized! i can manage specific versions much easier! > > what this means to you: > > * python 2.7 will be the default system python. > * 2.6 will still be installed and available (/usr/bin/python or > /usr/bin/python/2.6) > > what you need to do: > * install anaconda, have it update your PATH > * build locally and try to fix any bugs (for spark, this "should just > work") > * if you have problems, reach out to me and i'll see what i can do to help. > if we can't get your stuff running under python2.7, we can default to 2.6 > via a job config change. > > what i will be doing: > * setting up anaconda on my staging instance and spot-testing a lot of > builds before deployment > > please let me know if there are any issues/concerns... i'll be posting > updates this week and will let everyone know if there are any changes to > the Plan[tm]. > > your friendly devops engineer, > > shane >
Re: [jenkins infra -- pls read ] installing anaconda, moving default python from 2.6 -> 2.7
On Mon, Feb 23, 2015 at 11:36 AM, Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > The first concern for Spark will probably be to ensure that we still build > and test against Python 2.6, since that's the minimum version of Python we > support. > > sounds good... we can set up separate 2.6 builds on specific versions... this could allow you to easily differentiate between "baseline" and "latest and greatest" if you wanted. it'll have a little bit more administrative overhead, due to more jobs needing configs, but offers more flexibility. let me know what you think. > Otherwise this seems OK. We use numpy and other Python packages in > PySpark, but I don't think we're pinned to any particular version of those > packages. > > cool. i'll start mucking about and let you guys know how it goes. shane
Re: [jenkins infra -- pls read ] installing anaconda, moving default python from 2.6 -> 2.7
i'm going to punt on this until after the next spark 1.3 release (2-3 weeks?). since i'll be installing a bunch of other packages (including mongodb), i'd rather wait and be safe. :) the full install list is forthcoming, and i'll update the spark infra wiki w/what's installed on the workers. shane On Mon, Feb 23, 2015 at 11:13 AM, shane knapp wrote: > good morning, developers! > > TL;DR: > > i will be installing anaconda and setting it in the system PATH so that > your python will default to 2.7, as well as it taking over management of > all of the sci-py packages. this is potentially a big change, so i'll be > testing locally on my staging instance before deployment to the wide world. > > deployment is *tentatively* next monday, march 2nd. > > a little background: > > the jenkins test infra is currently (and happily) managed by a set of > tools that allow me to set up and deploy new workers, manage their packages > and make sure that all spark and research projects can happily and > successfully build. > > we're currently at the state where ~50 or so packages are installed and > configured on each worker. this is getting a little cumbersome, as the > package-to-build dep tree is getting pretty large. > > the biggest offender is the science-based python infrastructure. > everything is blindly installed w/yum and pip, so it's hard to control > *exactly* what version of any given library is as compared to what's on a > dev's laptop. > > the solution: > > anaconda (https://store.continuum.io/cshop/anaconda/)! everything is > centralized! i can manage specific versions much easier! > > what this means to you: > > * python 2.7 will be the default system python. > * 2.6 will still be installed and available (/usr/bin/python or > /usr/bin/python/2.6) > > what you need to do: > * install anaconda, have it update your PATH > * build locally and try to fix any bugs (for spark, this "should just > work") > * if you have problems, reach out to me and i'll see what i can do to > help. if we can't get your stuff running under python2.7, we can default > to 2.6 via a job config change. > > what i will be doing: > * setting up anaconda on my staging instance and spot-testing a lot of > builds before deployment > > please let me know if there are any issues/concerns... i'll be posting > updates this week and will let everyone know if there are any changes to > the Plan[tm]. > > your friendly devops engineer, > > shane >
