HI Andy, I built an anaconda/spark AMI a few months ago. I'm still iterating on it so if things break please report them. If you want to give it awhirl: ./spark-ec2 -k my_key -i ~/.ssh/mykey.rsa -a ami-3ecd0c56
The nice thing about anaconda is that it come pre-baked with ipython-notebook, matplotlib, scipy stack, and many other libraries. --Ben On Mon, Sep 29, 2014 at 12:45 PM, Andy Davidson < a...@santacruzintegration.com> wrote: > Hi Nicholas > > Yes out of the box PySpark works. My problem is I am using iPython note > book and matlabplot is not found. It seems that out of the box the cluster > has an old version of python and iPython notebook. It was suggested I > upgrade iPython because the new version include matlabplot. This upgrade > requires going to python 2.7. The python upgrade and iPython upgrade seemed > to work how ever I am still getting my original problem > > ERROR: Line magic function `%matplotlib` not found > > I also posted to the iPython-dev mail list. So far I have not found a > solution. Maybe I’ll have to switch to a different graphing package > > Thanks > > Andy > > From: Nicholas Chammas <nicholas.cham...@gmail.com> > Date: Saturday, September 27, 2014 at 4:49 PM > To: Andrew Davidson <a...@santacruzintegration.com> > Cc: "user@spark.apache.org" <user@spark.apache.org> > Subject: Re: iPython notebook ec2 cluster matlabplot not found? > > Can you first confirm that the regular PySpark shell works on your > cluster? Without upgrading to 2.7. That is, you log on to your master using > spark-ec2 > login and run bin/pyspark successfully without any special flags. > > And as far as I can tell, you should be able to use IPython at 2.6, so I’d > next confirm that that is working before throwing the 2.7 upgrade into the > mix. > > Also, when upgrading or installing things, try doing so for all the nodes > in your cluster using pssh. If you install stuff just on the master > without somehow transferring it to the slaves, that will be problematic. > > Finally, there is an open pull request > <https://github.com/apache/spark/pull/2554> related to IPython that may > be relevant, though I haven’t looked at it too closely. > > Nick > > > On Sat, Sep 27, 2014 at 7:33 PM, Andy Davidson < > a...@santacruzintegration.com> wrote: > >> Hi >> >> I am having a heck of time trying to get python to work correctly on my >> cluster created using the spark-ec2 script >> >> The following link was really helpful >> https://issues.apache.org/jira/browse/SPARK-922 >> >> >> I am still running into problem with matplotlib. (it works fine on my >> mac). I can not figure out how to get libagg, freetype, or Qhull >> dependencies installed. >> >> Has anyone else run into this problem? >> >> Thanks >> >> Andy >> >> sudo yum install freetype-devel >> >> sudo yum install libpng-devel >> >> sudo pip2.7 install six >> >> sudo pip2.7 install python-dateutil >> >> sudo pip2.7 install pyparsing >> >> sudo pip2.7 install pycxx >> >> >> sudo pip2.7 install matplotlib >> >> ec2-user@ip-172-31-15-87 ~]$ sudo pip2.7 install matplotlib >> >> Downloading/unpacking matplotlib >> >> Downloading matplotlib-1.4.0.tar.gz (51.2MB): 51.2MB downloaded >> >> Running setup.py (path:/tmp/pip_build_root/matplotlib/setup.py) >> egg_info for package matplotlib >> >> >> ============================================================================ >> >> Edit setup.cfg to change the build options >> >> >> >> BUILDING MATPLOTLIB >> >> matplotlib: yes [1.4.0] >> >> python: yes [2.7.5 (default, Sep 15 2014, 17:30:20) >> [GCC >> >> 4.8.2 20140120 (Red Hat 4.8.2-16)]] >> >> platform: yes [linux2] >> >> >> >> REQUIRED DEPENDENCIES AND EXTENSIONS >> >> numpy: yes [version 1.9.0] >> >> six: yes [using six version 1.8.0] >> >> dateutil: yes [using dateutil version 2.2] >> >> tornado: yes [using tornado version 4.0.2] >> >> pyparsing: yes [using pyparsing version 2.0.2] >> >> pycxx: yes [Couldn't import. Using local copy.] >> >> libagg: yes [pkg-config information for 'libagg' >> could not >> >> be found. Using local copy.] >> >> freetype: no [Requires freetype2 2.4 or later. Found >> >> 2.3.11.] >> >> png: yes [version 1.2.49] >> >> qhull: yes [pkg-config information for 'qhull' could >> not be >> >> found. Using local copy.] >> >> >> >> OPTIONAL SUBPACKAGES >> >> sample_data: yes [installing] >> >> toolkits: yes [installing] >> >> tests: yes [using nose version 1.3.4 / mock is >> required to >> >> run the matplotlib test suite. >> pip/easy_install may >> >> attempt to install it after matplotlib.] >> >> toolkits_tests: yes [using nose version 1.3.4 / mock is >> required to >> >> run the matplotlib test suite. >> pip/easy_install may >> >> attempt to install it after matplotlib.] >> >> >> >> OPTIONAL BACKEND EXTENSIONS >> >> macosx: no [Mac OS-X only] >> >> qt5agg: no [PyQt5 not found] >> >> qt4agg: no [PyQt4 not found] >> >> pyside: no [PySide not found] >> >> gtk3agg: no [Requires pygobject to be installed.] >> >> gtk3cairo: no [Requires cairocffi or pycairo to be >> installed.] >> >> gtkagg: no [Requires pygtk] >> >> tkagg: no [TKAgg requires Tkinter.] >> >> wxagg: no [requires wxPython] >> >> gtk: no [Requires pygtk] >> >> agg: yes [installing] >> >> cairo: no [cairocffi or pycairo not found] >> >> windowing: no [Microsoft Windows only] >> >> >> >> OPTIONAL LATEX DEPENDENCIES >> >> dvipng: no >> >> ghostscript: yes [version 8.70] >> >> latex: yes [version 3.141592] >> >> pdftops: no >> >> >> >> >> ============================================================================ >> >> * The following required packages can not be >> built: >> >> * freetype >> >> Complete output from command python setup.py egg_info: >> >> >> ============================================================================ >> >> >> Edit setup.cfg to change the build options >> >> >> >> >> BUILDING MATPLOTLIB >> >> >> matplotlib: yes [1.4.0] >> >> >> python: yes [2.7.5 (default, Sep 15 2014, 17:30:20) [GCC >> >> >> 4.8.2 20140120 (Red Hat 4.8.2-16)]] >> >> >> platform: yes [linux2] >> >> >> >> >> REQUIRED DEPENDENCIES AND EXTENSIONS >> >> >> numpy: yes [version 1.9.0] >> >> >> six: yes [using six version 1.8.0] >> >> >> dateutil: yes [using dateutil version 2.2] >> >> >> tornado: yes [using tornado version 4.0.2] >> >> >> pyparsing: yes [using pyparsing version 2.0.2] >> >> >> pycxx: yes [Couldn't import. Using local copy.] >> >> >> libagg: yes [pkg-config information for 'libagg' could not >> >> >> be found. Using local copy.] >> >> >> freetype: no [Requires freetype2 2.4 or later. Found >> >> >> 2.3.11.] >> >> >> png: yes [version 1.2.49] >> >> >> qhull: yes [pkg-config information for 'qhull' could not >> be >> >> >> found. Using local copy.] >> >> >> >> >> OPTIONAL SUBPACKAGES >> >> >> sample_data: yes [installing] >> >> >> toolkits: yes [installing] >> >> >> tests: yes [using nose version 1.3.4 / mock is required >> to >> >> >> run the matplotlib test suite. pip/easy_install >> may >> >> >> attempt to install it after matplotlib.] >> >> >> toolkits_tests: yes [using nose version 1.3.4 / mock is required >> to >> >> >> run the matplotlib test suite. pip/easy_install >> may >> >> >> attempt to install it after matplotlib.] >> >> >> >> >> OPTIONAL BACKEND EXTENSIONS >> >> >> macosx: no [Mac OS-X only] >> >> >> qt5agg: no [PyQt5 not found] >> >> >> qt4agg: no [PyQt4 not found] >> >> >> pyside: no [PySide not found] >> >> >> gtk3agg: no [Requires pygobject to be installed.] >> >> >> gtk3cairo: no [Requires cairocffi or pycairo to be >> installed.] >> >> >> gtkagg: no [Requires pygtk] >> >> >> tkagg: no [TKAgg requires Tkinter.] >> >> >> wxagg: no [requires wxPython] >> >> >> gtk: no [Requires pygtk] >> >> >> agg: yes [installing] >> >> >> cairo: no [cairocffi or pycairo not found] >> >> >> windowing: no [Microsoft Windows only] >> >> >> >> >> OPTIONAL LATEX DEPENDENCIES >> >> >> dvipng: no >> >> >> ghostscript: yes [version 8.70] >> >> >> latex: yes [version 3.141592] >> >> >> pdftops: no >> >> >> >> >> >> ============================================================================ >> >> >> * The following required packages can not be >> built: >> >> >> * freetype >> >> >> ---------------------------------------- >> >> Cleaning up... >> >> Command python setup.py egg_info failed with error code 1 in >> /tmp/pip_build_root/matplotlib >> >> Storing debug log for failure in /root/.pip/pip.log >> >> ec2-user@ip-172-31-15-87 ~]$ >> >> >
