Re: Issue in executing Spark Application from Eclipse
Hey Stuti, Did you start your standalone Master and Workers? You can do this through sbin/start-all.sh (see http://spark.apache.org/docs/latest/spark-standalone.html). Otherwise, I would recommend launching your application from the command line through bin/spark-submit. I am not sure if we officially support launching Spark applications from an IDE, because spark-submit handles very specific cases of how we set up class paths and JVM memory etc. -Andrew 2014-12-03 22:05 GMT-08:00 Stuti Awasthi stutiawas...@hcl.com: Hi All, I have a standalone Spark(1.1) cluster on one machine and I have installed scala Eclipse IDE (scala 2.10) on my desktop. I am trying to execute a spark code to execute over my standalone cluster but getting errors. Please guide me to resolve this. Code: val logFile = File Path present on desktop // Should be some file on your system val conf = new SparkConf().setAppName(Simple Application).setMaster(spark://IP:PORT).setSparkHome(/home/stuti/Spark/spark-1.1.0-bin-hadoop1); val sc = new SparkContext(conf) println(sc.master) // Print correct master val logData = sc.textFile(logFile, 2).cache() println(logData.count) // throws error Error : Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 14/12/04 11:05:38 INFO SecurityManager: Changing view acls to: stutiawasthi, 14/12/04 11:05:38 INFO SecurityManager: Changing modify acls to: stutiawasthi, 14/12/04 11:05:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(stutiawasthi, ); users with modify permissions: Set(stutiawasthi, ) 14/12/04 11:05:39 INFO Slf4jLogger: Slf4jLogger started 14/12/04 11:05:39 INFO Remoting: Starting remoting 14/12/04 11:05:40 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@HOSTNAME_DESKTOP:62308] 14/12/04 11:05:40 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@HOSTNAME_DESKTOP:62308] 14/12/04 11:05:40 INFO Utils: Successfully started service 'sparkDriver' on port 62308. 14/12/04 11:05:40 INFO SparkEnv: Registering MapOutputTracker 14/12/04 11:05:40 INFO SparkEnv: Registering BlockManagerMaster 14/12/04 11:05:40 INFO DiskBlockManager: Created local directory at C:\Users\STUTIA~1\AppData\Local\Temp\spark-local-20141204110540-ad60 14/12/04 11:05:40 INFO Utils: Successfully started service 'Connection manager for block manager' on port 62311. 14/12/04 11:05:40 INFO ConnectionManager: Bound socket to port 62311 with id = ConnectionManagerId(HOSTNAME_DESKTOP,62311) 14/12/04 11:05:41 INFO MemoryStore: MemoryStore started with capacity 133.6 MB 14/12/04 11:05:41 INFO BlockManagerMaster: Trying to register BlockManager 14/12/04 11:05:41 INFO BlockManagerMasterActor: Registering block manager HOSTNAME_DESKTOP:62311 with 133.6 MB RAM 14/12/04 11:05:41 INFO BlockManagerMaster: Registered BlockManager 14/12/04 11:05:41 INFO HttpFileServer: HTTP File server directory is C:\Users\STUTIA~1\AppData\Local\Temp\spark-b65e69f4-69b9-4bb2-b41f-67165909e4c7 14/12/04 11:05:41 INFO HttpServer: Starting HTTP Server 14/12/04 11:05:41 INFO Utils: Successfully started service 'HTTP file server' on port 62312. 14/12/04 11:05:42 INFO Utils: Successfully started service 'SparkUI' on port 4040. 14/12/04 11:05:42 INFO SparkUI: Started SparkUI at http:// HOSTNAME_DESKTOP:4040 14/12/04 11:05:43 INFO AppClient$ClientActor: Connecting to master spark://10.112.67.80:7077... 14/12/04 11:05:43 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 spark://10.112.67.80:7077 14/12/04 11:05:44 WARN SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes 14/12/04 11:05:45 INFO MemoryStore: ensureFreeSpace(31447) called with curMem=0, maxMem=140142182 14/12/04 11:05:45 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 30.7 KB, free 133.6 MB) 14/12/04 11:05:45 INFO MemoryStore: ensureFreeSpace(3631) called with curMem=31447, maxMem=140142182 14/12/04 11:05:45 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.5 KB, free 133.6 MB) 14/12/04 11:05:45 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on HOSTNAME_DESKTOP:62311 (size: 3.5 KB, free: 133.6 MB) 14/12/04 11:05:45 INFO BlockManagerMaster: Updated info of block broadcast_0_piece0 14/12/04 11:05:45 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/12/04 11:05:45 WARN LoadSnappy: Snappy native library not loaded 14/12/04 11:05:46 INFO FileInputFormat: Total input paths to process : 1 14/12/04 11:05:46 INFO SparkContext: Starting job: count at Test.scala:15 14/12/04 11:05:46 INFO DAGScheduler: Got job 0 (count at Test.scala:15) with 2 output partitions
Issue in executing Spark Application from Eclipse
Hi All, I have a standalone Spark(1.1) cluster on one machine and I have installed scala Eclipse IDE (scala 2.10) on my desktop. I am trying to execute a spark code to execute over my standalone cluster but getting errors. Please guide me to resolve this. Code: val logFile = File Path present on desktop // Should be some file on your system val conf = new SparkConf().setAppName(Simple Application).setMaster(spark://IP:PORT).setSparkHome(/home/stuti/Spark/spark-1.1.0-bin-hadoop1); val sc = new SparkContext(conf) println(sc.master) // Print correct master val logData = sc.textFile(logFile, 2).cache() println(logData.count) // throws error Error : Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 14/12/04 11:05:38 INFO SecurityManager: Changing view acls to: stutiawasthi, 14/12/04 11:05:38 INFO SecurityManager: Changing modify acls to: stutiawasthi, 14/12/04 11:05:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(stutiawasthi, ); users with modify permissions: Set(stutiawasthi, ) 14/12/04 11:05:39 INFO Slf4jLogger: Slf4jLogger started 14/12/04 11:05:39 INFO Remoting: Starting remoting 14/12/04 11:05:40 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@HOSTNAME_DESKTOP:62308] 14/12/04 11:05:40 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@HOSTNAME_DESKTOP:62308] 14/12/04 11:05:40 INFO Utils: Successfully started service 'sparkDriver' on port 62308. 14/12/04 11:05:40 INFO SparkEnv: Registering MapOutputTracker 14/12/04 11:05:40 INFO SparkEnv: Registering BlockManagerMaster 14/12/04 11:05:40 INFO DiskBlockManager: Created local directory at C:\Users\STUTIA~1\AppData\Local\Temp\spark-local-20141204110540-ad60 14/12/04 11:05:40 INFO Utils: Successfully started service 'Connection manager for block manager' on port 62311. 14/12/04 11:05:40 INFO ConnectionManager: Bound socket to port 62311 with id = ConnectionManagerId(HOSTNAME_DESKTOP,62311) 14/12/04 11:05:41 INFO MemoryStore: MemoryStore started with capacity 133.6 MB 14/12/04 11:05:41 INFO BlockManagerMaster: Trying to register BlockManager 14/12/04 11:05:41 INFO BlockManagerMasterActor: Registering block manager HOSTNAME_DESKTOP:62311 with 133.6 MB RAM 14/12/04 11:05:41 INFO BlockManagerMaster: Registered BlockManager 14/12/04 11:05:41 INFO HttpFileServer: HTTP File server directory is C:\Users\STUTIA~1\AppData\Local\Temp\spark-b65e69f4-69b9-4bb2-b41f-67165909e4c7 14/12/04 11:05:41 INFO HttpServer: Starting HTTP Server 14/12/04 11:05:41 INFO Utils: Successfully started service 'HTTP file server' on port 62312. 14/12/04 11:05:42 INFO Utils: Successfully started service 'SparkUI' on port 4040. 14/12/04 11:05:42 INFO SparkUI: Started SparkUI at http://HOSTNAME_DESKTOP:4040 14/12/04 11:05:43 INFO AppClient$ClientActor: Connecting to master spark://10.112.67.80:7077... 14/12/04 11:05:43 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 spark://10.112.67.80:7077 14/12/04 11:05:44 WARN SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes 14/12/04 11:05:45 INFO MemoryStore: ensureFreeSpace(31447) called with curMem=0, maxMem=140142182 14/12/04 11:05:45 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 30.7 KB, free 133.6 MB) 14/12/04 11:05:45 INFO MemoryStore: ensureFreeSpace(3631) called with curMem=31447, maxMem=140142182 14/12/04 11:05:45 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.5 KB, free 133.6 MB) 14/12/04 11:05:45 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on HOSTNAME_DESKTOP:62311 (size: 3.5 KB, free: 133.6 MB) 14/12/04 11:05:45 INFO BlockManagerMaster: Updated info of block broadcast_0_piece0 14/12/04 11:05:45 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/12/04 11:05:45 WARN LoadSnappy: Snappy native library not loaded 14/12/04 11:05:46 INFO FileInputFormat: Total input paths to process : 1 14/12/04 11:05:46 INFO SparkContext: Starting job: count at Test.scala:15 14/12/04 11:05:46 INFO DAGScheduler: Got job 0 (count at Test.scala:15) with 2 output partitions (allowLocal=false) 14/12/04 11:05:46 INFO DAGScheduler: Final stage: Stage 0(count at Test.scala:15) 14/12/04 11:05:46 INFO DAGScheduler: Parents of final stage: List() 14/12/04 11:05:46 INFO DAGScheduler: Missing parents: List() 14/12/04 11:05:46 INFO DAGScheduler: Submitting Stage 0 (D:/Workspace/Spark/Test/README MappedRDD[1] at textFile at Test.scala:14), which has no missing parents 14/12/04 11:05:46 INFO MemoryStore: ensureFreeSpace(2408) called with curMem=35078, maxMem=140142182
Re: Issue in executing Spark Application from Eclipse
It seems you provided master url as spark://10.112.67.80:7077 , i think you should give spark://ubuntu:7077 instead. Thanks Best Regards On Thu, Dec 4, 2014 at 11:35 AM, Stuti Awasthi stutiawas...@hcl.com wrote: Hi All, I have a standalone Spark(1.1) cluster on one machine and I have installed scala Eclipse IDE (scala 2.10) on my desktop. I am trying to execute a spark code to execute over my standalone cluster but getting errors. Please guide me to resolve this. Code: val logFile = File Path present on desktop // Should be some file on your system val conf = new SparkConf().setAppName(Simple Application).setMaster(spark://IP:PORT).setSparkHome(/home/stuti/Spark/spark-1.1.0-bin-hadoop1); val sc = new SparkContext(conf) println(sc.master) // Print correct master val logData = sc.textFile(logFile, 2).cache() println(logData.count) // throws error Error : Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 14/12/04 11:05:38 INFO SecurityManager: Changing view acls to: stutiawasthi, 14/12/04 11:05:38 INFO SecurityManager: Changing modify acls to: stutiawasthi, 14/12/04 11:05:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(stutiawasthi, ); users with modify permissions: Set(stutiawasthi, ) 14/12/04 11:05:39 INFO Slf4jLogger: Slf4jLogger started 14/12/04 11:05:39 INFO Remoting: Starting remoting 14/12/04 11:05:40 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@HOSTNAME_DESKTOP:62308] 14/12/04 11:05:40 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@HOSTNAME_DESKTOP:62308] 14/12/04 11:05:40 INFO Utils: Successfully started service 'sparkDriver' on port 62308. 14/12/04 11:05:40 INFO SparkEnv: Registering MapOutputTracker 14/12/04 11:05:40 INFO SparkEnv: Registering BlockManagerMaster 14/12/04 11:05:40 INFO DiskBlockManager: Created local directory at C:\Users\STUTIA~1\AppData\Local\Temp\spark-local-20141204110540-ad60 14/12/04 11:05:40 INFO Utils: Successfully started service 'Connection manager for block manager' on port 62311. 14/12/04 11:05:40 INFO ConnectionManager: Bound socket to port 62311 with id = ConnectionManagerId(HOSTNAME_DESKTOP,62311) 14/12/04 11:05:41 INFO MemoryStore: MemoryStore started with capacity 133.6 MB 14/12/04 11:05:41 INFO BlockManagerMaster: Trying to register BlockManager 14/12/04 11:05:41 INFO BlockManagerMasterActor: Registering block manager HOSTNAME_DESKTOP:62311 with 133.6 MB RAM 14/12/04 11:05:41 INFO BlockManagerMaster: Registered BlockManager 14/12/04 11:05:41 INFO HttpFileServer: HTTP File server directory is C:\Users\STUTIA~1\AppData\Local\Temp\spark-b65e69f4-69b9-4bb2-b41f-67165909e4c7 14/12/04 11:05:41 INFO HttpServer: Starting HTTP Server 14/12/04 11:05:41 INFO Utils: Successfully started service 'HTTP file server' on port 62312. 14/12/04 11:05:42 INFO Utils: Successfully started service 'SparkUI' on port 4040. 14/12/04 11:05:42 INFO SparkUI: Started SparkUI at http:// HOSTNAME_DESKTOP:4040 14/12/04 11:05:43 INFO AppClient$ClientActor: Connecting to master spark://10.112.67.80:7077... 14/12/04 11:05:43 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 spark://10.112.67.80:7077 14/12/04 11:05:44 WARN SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes 14/12/04 11:05:45 INFO MemoryStore: ensureFreeSpace(31447) called with curMem=0, maxMem=140142182 14/12/04 11:05:45 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 30.7 KB, free 133.6 MB) 14/12/04 11:05:45 INFO MemoryStore: ensureFreeSpace(3631) called with curMem=31447, maxMem=140142182 14/12/04 11:05:45 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.5 KB, free 133.6 MB) 14/12/04 11:05:45 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on HOSTNAME_DESKTOP:62311 (size: 3.5 KB, free: 133.6 MB) 14/12/04 11:05:45 INFO BlockManagerMaster: Updated info of block broadcast_0_piece0 14/12/04 11:05:45 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/12/04 11:05:45 WARN LoadSnappy: Snappy native library not loaded 14/12/04 11:05:46 INFO FileInputFormat: Total input paths to process : 1 14/12/04 11:05:46 INFO SparkContext: Starting job: count at Test.scala:15 14/12/04 11:05:46 INFO DAGScheduler: Got job 0 (count at Test.scala:15) with 2 output partitions (allowLocal=false) 14/12/04 11:05:46 INFO DAGScheduler: Final stage: Stage 0(count at Test.scala:15) 14/12/04 11:05:46 INFO DAGScheduler: Parents of final stage: List() 14/12/04 11:05:46 INFO DAGScheduler: Missing parents: List() 14/12/04 11:05:46 INFO DAGScheduler: Submitting Stage 0
