Hi there,
*I have already downloaded Pre-built spark-1.1.0, I want to run
pyspark by try typing ./bin/pyspark but I got the following error:*
*
*
*scala shell is up and working fine*
hduser@master:~/Downloads/spark-1.1.0$ ./bin/spark-shell
Java HotSpot(TM) Client VM warning: ignoring option MaxPermSize=128m;
support was removed in 8.0
Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
.....
.....
14/11/18 04:33:13 INFO AkkaUtils: Connecting to HeartbeatReceiver:
akka.tcp://sparkDriver@master:34937/user/HeartbeatReceiver
14/11/18 04:33:13 INFO SparkILoop: Created spark context..
Spark context available as sc.
scala> hduser@master:~/Downloads/spark-1.1.0$
*
*
*But python shell does not work:*
hduser@master:~/Downloads/spark-1.1.0$
hduser@master:~/Downloads/spark-1.1.0$
hduser@master:~/Downloads/spark-1.1.0$ ./bin/pyspark
Python 2.7.3 (default, Feb 27 2014, 20:00:17)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Java HotSpot(TM) Client VM warning: ignoring option MaxPermSize=128m;
support was removed in 8.0
Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
14/11/18 04:36:06 INFO SecurityManager: Changing view acls to: hduser,
14/11/18 04:36:06 INFO SecurityManager: Changing modify acls to: hduser,
14/11/18 04:36:06 INFO SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view
permissions: Set(hduser, ); users with modify permissions: Set(hduser, )
14/11/18 04:36:06 INFO Slf4jLogger: Slf4jLogger started
14/11/18 04:36:06 INFO Remoting: Starting remoting
14/11/18 04:36:06 INFO Remoting: Remoting started; listening on
addresses :[akka.tcp://sparkDriver@master:52317]
14/11/18 04:36:06 INFO Remoting: Remoting now listens on addresses:
[akka.tcp://sparkDriver@master:52317]
14/11/18 04:36:06 INFO Utils: Successfully started service
'sparkDriver' on port 52317.
14/11/18 04:36:06 INFO SparkEnv: Registering MapOutputTracker
14/11/18 04:36:06 INFO SparkEnv: Registering BlockManagerMaster
14/11/18 04:36:06 INFO DiskBlockManager: Created local directory at
/tmp/spark-local-20141118043606-c346
14/11/18 04:36:07 INFO Utils: Successfully started service 'Connection
manager for block manager' on port 47507.
14/11/18 04:36:07 INFO ConnectionManager: Bound socket to port 47507
with id = ConnectionManagerId(master,47507)
14/11/18 04:36:07 INFO MemoryStore: MemoryStore started with capacity
267.3 MB
14/11/18 04:36:07 INFO BlockManagerMaster: Trying to register BlockManager
14/11/18 04:36:07 INFO BlockManagerMasterActor: Registering block
manager master:47507 with 267.3 MB RAM
14/11/18 04:36:07 INFO BlockManagerMaster: Registered BlockManager
14/11/18 04:36:07 INFO HttpFileServer: HTTP File server directory is
/tmp/spark-8b29544a-c74b-4a3e-88e0-13801c8dcc65
14/11/18 04:36:07 INFO HttpServer: Starting HTTP Server
14/11/18 04:36:07 INFO Utils: Successfully started service 'HTTP file
server' on port 40029.
14/11/18 04:36:12 INFO Utils: Successfully started service 'SparkUI'
on port 4040.
14/11/18 04:36:12 INFO SparkUI: Started SparkUI at http://master:4040
<http://master:4040/>
14/11/18 04:36:12 INFO AkkaUtils: Connecting to HeartbeatReceiver:
akka.tcp://sparkDriver@master:52317/user/HeartbeatReceiver
14/11/18 04:36:12 INFO SparkUI: Stopped Spark web UI at
http://master:4040 <http://master:4040/>
14/11/18 04:36:12 INFO DAGScheduler: Stopping DAGScheduler
14/11/18 04:36:13 INFO MapOutputTrackerMasterActor:
MapOutputTrackerActor stopped!
14/11/18 04:36:13 INFO ConnectionManager: Selector thread was interrupted!
14/11/18 04:36:13 INFO ConnectionManager: ConnectionManager stopped
14/11/18 04:36:13 INFO MemoryStore: MemoryStore cleared
14/11/18 04:36:13 INFO BlockManager: BlockManager stopped
14/11/18 04:36:13 INFO BlockManagerMaster: BlockManagerMaster stopped
14/11/18 04:36:13 INFO RemoteActorRefProvider$RemotingTerminator:
Shutting down remote daemon.
14/11/18 04:36:13 INFO SparkContext: Successfully stopped SparkContext
14/11/18 04:36:13 INFO RemoteActorRefProvider$RemotingTerminator:
Remote daemon shut down; proceeding with flushing remote transports.
14/11/18 04:36:13 INFO Remoting: Remoting shut down
14/11/18 04:36:13 INFO RemoteActorRefProvider$RemotingTerminator:
Remoting shut down.
Traceback (most recent call last):
File "/home/hduser/Downloads/spark-1.1.0/python/pyspark/shell.py",
line 44, in <module>
sc = SparkContext(appName="PySparkShell", pyFiles=add_files)
File "/home/hduser/Downloads/spark-1.1.0/python/pyspark/context.py",
line 107, in __init__
conf)
File "/home/hduser/Downloads/spark-1.1.0/python/pyspark/context.py",
line 159, in _do_init
self._accumulatorServer = accumulators._start_update_server()
File
"/home/hduser/Downloads/spark-1.1.0/python/pyspark/accumulators.py",
line 251, in _start_update_server
server = AccumulatorServer(("localhost", 0), _UpdateRequestHandler)
File "/usr/lib/python2.7/SocketServer.py", line 408, in __init__
self.server_bind()
File "/usr/lib/python2.7/SocketServer.py", line 419, in server_bind
self.socket.bind(self.server_address)
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
socket.gaierror: [Errno -5] No address associated with hostname
>>> sc.parallelize(range(1000)).count()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'sc' is not defined
>>> sc
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'sc' is not defined
>>> spark
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'spark' is not defined
>>>
Best Regards
.......................................................
Amin Mohebbi
PhD candidate in Software Engineering
at university of Malaysia
Tel : +60 18 2040 017
E-Mail : tp025...@ex.apiit.edu.my
amin_...@me.com
