problem with spark-ec2 launch script Re: spark-ec2 ERROR: Line magic function `%matplotlib` not found
Hi Davies
The real issue is about cluster management. I am new to the spark world and
am not a system administrator. It seem like the problem is with the
spark-ec2 launch script. It is installing old version of python
In the mean time I am trying to figure out how I can manually install the
correct version on all the machines in my cluster
Thanks
Andy
From: Davies Liu dav...@databricks.com
Date: Thursday, September 25, 2014 at 9:58 PM
To: Andrew Davidson a...@santacruzintegration.com
Cc: user@spark.apache.org user@spark.apache.org
Subject: Re: spark-ec2 ERROR: Line magic function `%matplotlib` not found
Maybe you have Python 2.7 on master but Python 2.6 in cluster,
you should upgrade python to 2.7 in cluster, or use python 2.6 in
master by set PYSPARK_PYTHON=python2.6
On Thu, Sep 25, 2014 at 5:11 PM, Andy Davidson
a...@santacruzintegration.com wrote:
Hi
I am running into trouble using iPython notebook on my cluster. Use the
following command to set the cluster up
$ ./spark-ec2 --key-pair=$KEY_PAIR --identity-file=$KEY_FILE
--region=$REGION --slaves=$NUM_SLAVES launch $CLUSTER_NAME
On master I launch python as follows
$ IPYTHON_OPTS=notebook --pylab inline --no-browser --port=7000
$SPARK_HOME/bin/pyspark
It looks like the problem is the cluster is using an old version of python
and python. Any idea how I can easily upgrade ? The following version works
on my mac
Thanks
Andy
{'commit_hash': '681fd77',
'commit_source': 'installation',
'default_encoding': 'UTF-8',
'ipython_path': '/Library/Python/2.7/site-packages/IPython',
'ipython_version': '2.1.0',
'os_name': 'posix',
'platform': 'Darwin-13.3.0-x86_64-i386-64bit',
'sys_executable': '/usr/bin/python',
'sys_platform': 'darwin',
'sys_version': '2.7.5 (default, Mar 9 2014, 22:15:05) \n[GCC 4.2.1
Compatible Apple LLVM 5.0 (clang-500.0.68)]¹}
Re: problem with spark-ec2 launch script Re: spark-ec2 ERROR: Line magic function `%matplotlib` not found
Are you able to use the regular PySpark shell on your EC2 cluster? That
would be the first thing to confirm is working.
I don’t know whether the version of Python on the cluster would affect
whether IPython works or not, but if you want to try manually upgrading
Python on a cluster launched by spark-ec2, there are some instructions in
the comments here https://issues.apache.org/jira/browse/SPARK-922 for
doing so.
Nick
On Fri, Sep 26, 2014 at 2:18 PM, Andy Davidson
a...@santacruzintegration.com wrote:
Hi Davies
The real issue is about cluster management. I am new to the spark world
and am not a system administrator. It seem like the problem is with the
spark-ec2 launch script. It is installing old version of python
In the mean time I am trying to figure out how I can manually install the
correct version on all the machines in my cluster
Thanks
Andy
From: Davies Liu dav...@databricks.com
Date: Thursday, September 25, 2014 at 9:58 PM
To: Andrew Davidson a...@santacruzintegration.com
Cc: user@spark.apache.org user@spark.apache.org
Subject: Re: spark-ec2 ERROR: Line magic function `%matplotlib` not found
Maybe you have Python 2.7 on master but Python 2.6 in cluster,
you should upgrade python to 2.7 in cluster, or use python 2.6 in
master by set PYSPARK_PYTHON=python2.6
On Thu, Sep 25, 2014 at 5:11 PM, Andy Davidson
a...@santacruzintegration.com wrote:
Hi
I am running into trouble using iPython notebook on my cluster. Use the
following command to set the cluster up
$ ./spark-ec2 --key-pair=$KEY_PAIR --identity-file=$KEY_FILE
--region=$REGION --slaves=$NUM_SLAVES launch $CLUSTER_NAME
On master I launch python as follows
$ IPYTHON_OPTS=notebook --pylab inline --no-browser --port=7000
$SPARK_HOME/bin/pyspark
It looks like the problem is the cluster is using an old version of python
and python. Any idea how I can easily upgrade ? The following version works
on my mac
Thanks
Andy
{'commit_hash': '681fd77',
'commit_source': 'installation',
'default_encoding': 'UTF-8',
'ipython_path': '/Library/Python/2.7/site-packages/IPython',
'ipython_version': '2.1.0',
'os_name': 'posix',
'platform': 'Darwin-13.3.0-x86_64-i386-64bit',
'sys_executable': '/usr/bin/python',
'sys_platform': 'darwin',
'sys_version': '2.7.5 (default, Mar 9 2014, 22:15:05) \n[GCC 4.2.1
Compatible Apple LLVM 5.0 (clang-500.0.68)]’}
Re: problem with spark-ec2 launch script Re: spark-ec2 ERROR: Line magic function `%matplotlib` not found
Many many thanks
Andy
From: Nicholas Chammas nicholas.cham...@gmail.com
Date: Friday, September 26, 2014 at 11:24 AM
To: Andrew Davidson a...@santacruzintegration.com
Cc: Davies Liu dav...@databricks.com, user@spark.apache.org
user@spark.apache.org
Subject: Re: problem with spark-ec2 launch script Re: spark-ec2 ERROR: Line
magic function `%matplotlib` not found
Are you able to use the regular PySpark shell on your EC2 cluster? That would
be the first thing to confirm is working.
I don’t know whether the version of Python on the cluster would affect whether
IPython works or not, but if you want to try manually upgrading Python on a
cluster launched by spark-ec2, there are some instructions in the comments
here https://issues.apache.org/jira/browse/SPARK-922 for doing so.
Nick
On Fri, Sep 26, 2014 at 2:18 PM, Andy Davidson a...@santacruzintegration.com
wrote:
Hi Davies
The real issue is about cluster management. I am new to the spark world and
am not a system administrator. It seem like the problem is with the
spark-ec2 launch script. It is installing old version of python
In the mean time I am trying to figure out how I can manually install the
correct version on all the machines in my cluster
Thanks
Andy
From: Davies Liu dav...@databricks.com
Date: Thursday, September 25, 2014 at 9:58 PM
To: Andrew Davidson a...@santacruzintegration.com
Cc: user@spark.apache.org user@spark.apache.org
Subject: Re: spark-ec2 ERROR: Line magic function `%matplotlib` not found
Maybe you have Python 2.7 on master but Python 2.6 in cluster,
you should upgrade python to 2.7 in cluster, or use python 2.6 in
master by set PYSPARK_PYTHON=python2.6
On Thu, Sep 25, 2014 at 5:11 PM, Andy Davidson
a...@santacruzintegration.com wrote:
Hi
I am running into trouble using iPython notebook on my cluster. Use the
following command to set the cluster up
$ ./spark-ec2 --key-pair=$KEY_PAIR --identity-file=$KEY_FILE
--region=$REGION --slaves=$NUM_SLAVES launch $CLUSTER_NAME
On master I launch python as follows
$ IPYTHON_OPTS=notebook --pylab inline --no-browser --port=7000
$SPARK_HOME/bin/pyspark
It looks like the problem is the cluster is using an old version of python
and python. Any idea how I can easily upgrade ? The following version
works
on my mac
Thanks
Andy
{'commit_hash': '681fd77',
'commit_source': 'installation',
'default_encoding': 'UTF-8',
'ipython_path': '/Library/Python/2.7/site-packages/IPython',
'ipython_version': '2.1.0',
'os_name': 'posix',
'platform': 'Darwin-13.3.0-x86_64-i386-64bit',
'sys_executable': '/usr/bin/python',
'sys_platform': 'darwin',
'sys_version': '2.7.5 (default, Mar 9 2014, 22:15:05) \n[GCC 4.2.1
Compatible Apple LLVM 5.0 (clang-500.0.68)]’}
spark-ec2 ERROR: Line magic function `%matplotlib` not found
Hi
I am running into trouble using iPython notebook on my cluster. Use the
following command to set the cluster up
$ ./spark-ec2 --key-pair=$KEY_PAIR --identity-file=$KEY_FILE
--region=$REGION --slaves=$NUM_SLAVES launch $CLUSTER_NAME
On master I launch python as follows
$ IPYTHON_OPTS=notebook --pylab inline --no-browser --port=7000
$SPARK_HOME/bin/pyspark
It looks like the problem is the cluster is using an old version of python
and python. Any idea how I can easily upgrade ? The following version works
on my mac
Thanks
Andy
{'commit_hash': '681fd77',
'commit_source': 'installation',
'default_encoding': 'UTF-8',
'ipython_path': '/Library/Python/2.7/site-packages/IPython',
'ipython_version': '2.1.0',
'os_name': 'posix',
'platform': 'Darwin-13.3.0-x86_64-i386-64bit',
'sys_executable': '/usr/bin/python',
'sys_platform': 'darwin',
'sys_version': '2.7.5 (default, Mar 9 2014, 22:15:05) \n[GCC 4.2.1
Compatible Apple LLVM 5.0 (clang-500.0.68)]¹}
Re: spark-ec2 ERROR: Line magic function `%matplotlib` not found
Maybe you have Python 2.7 on master but Python 2.6 in cluster,
you should upgrade python to 2.7 in cluster, or use python 2.6 in
master by set PYSPARK_PYTHON=python2.6
On Thu, Sep 25, 2014 at 5:11 PM, Andy Davidson
a...@santacruzintegration.com wrote:
Hi
I am running into trouble using iPython notebook on my cluster. Use the
following command to set the cluster up
$ ./spark-ec2 --key-pair=$KEY_PAIR --identity-file=$KEY_FILE
--region=$REGION --slaves=$NUM_SLAVES launch $CLUSTER_NAME
On master I launch python as follows
$ IPYTHON_OPTS=notebook --pylab inline --no-browser --port=7000
$SPARK_HOME/bin/pyspark
It looks like the problem is the cluster is using an old version of python
and python. Any idea how I can easily upgrade ? The following version works
on my mac
Thanks
Andy
{'commit_hash': '681fd77',
'commit_source': 'installation',
'default_encoding': 'UTF-8',
'ipython_path': '/Library/Python/2.7/site-packages/IPython',
'ipython_version': '2.1.0',
'os_name': 'posix',
'platform': 'Darwin-13.3.0-x86_64-i386-64bit',
'sys_executable': '/usr/bin/python',
'sys_platform': 'darwin',
'sys_version': '2.7.5 (default, Mar 9 2014, 22:15:05) \n[GCC 4.2.1
Compatible Apple LLVM 5.0 (clang-500.0.68)]’}
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
