Error "Driver disassociated" while running the spark job
I got this error from the executor's stderr: [akka.tcp://sparkDriver@saturn00:49464] disassociated! Shutting down. What is the reason of "Actor not found"? // Yoonmin Nam - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Error while running sparkSQL application in the cluster-mode environment
Hello, I tried to execute a simple spark application using sparkSQL. At first try, it worked as I exepcted but after then, it doesn't run and shows an stderr like below: Spark Executor Command: "java" "-cp" "::/opt/spark-1.0.2-bin-hadoop2/conf:/opt/spark-1.0.2-bin-hadoop2/lib/spark-assembly-1.0.2-hadoop2.4.0.jar:/opt/hadoop2/etc/hadoop:/opt/hadoop2/etc/hadoop" "-XX:MaxPermSize=128m" "-Xms14336M" "-Xmx14336M" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "akka.tcp://spark@saturn00:35894/user/CoarseGrainedScheduler" "9" "saturn09" "4" "akka.tcp://sparkWorker@saturn09:45636/user/Worker" "app-20140908223656-" 14/09/08 22:36:57 INFO spark.SecurityManager: Changing view acls to: root 14/09/08 22:36:57 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root) 14/09/08 22:36:57 INFO slf4j.Slf4jLogger: Slf4jLogger started 14/09/08 22:36:57 INFO Remoting: Starting remoting 14/09/08 22:36:57 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutor@saturn09:44260] 14/09/08 22:36:57 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkExecutor@saturn09:44260] 14/09/08 22:36:57 INFO executor.CoarseGrainedExecutorBackend: Connecting to driver: akka.tcp://spark@saturn00:35894/user/CoarseGrainedScheduler 14/09/08 22:36:57 INFO worker.WorkerWatcher: Connecting to worker akka.tcp://sparkWorker@saturn09:45636/user/Worker 14/09/08 22:36:57 INFO worker.WorkerWatcher: Successfully connected to akka.tcp://sparkWorker@saturn09:45636/user/Wo rker 14/09/08 22:36:57 INFO executor.CoarseGrainedExecutorBackend: Successfully registered with driver 14/09/08 22:36:57 INFO spark.SecurityManager: Changing view acls to: root 14/09/08 22:36:57 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root) 14/09/08 22:36:58 INFO slf4j.Slf4jLogger: Slf4jLogger started 14/09/08 22:36:58 INFO Remoting: Starting remoting 14/09/08 22:36:58 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@saturn09:39880] 14/09/08 22:36:58 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@saturn09:39880] 14/09/08 22:36:58 INFO spark.SparkEnv: Connecting to MapOutputTracker: akka.tcp://spark@saturn00:35894/user/MapOutputTracker 14/09/08 22:36:58 INFO spark.SparkEnv: Connecting to BlockManagerMaster: akka.tcp://spark@saturn00:35894/user/BlockManagerMaster 14/09/08 22:36:58 INFO storage.DiskBlockManager: Created local directory at /hadoop/spark/spark-local-20140908223658-5699 14/09/08 22:36:58 INFO storage.MemoryStore: MemoryStore started with capacity 4.0 GB. 14/09/08 22:36:58 INFO network.ConnectionManager: Bound socket to port 49090 with id = ConnectionManagerId(saturn09,49090) 14/09/08 22:36:58 INFO storage.BlockManagerMaster: Trying to register BlockManager 14/09/08 22:36:58 INFO storage.BlockManagerMaster: Registered BlockManager 14/09/08 22:36:58 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-379704ff-05f2-4c93-8814-ffbe1cc8cd53 14/09/08 22:36:58 INFO spark.HttpServer: Starting HTTP Server 14/09/08 22:36:58 INFO server.Server: jetty-8.y.z-SNAPSHOT 14/09/08 22:36:58 INFO server.AbstractConnector: Started mailto:SocketConnector@0.0.0.0:40257";>SocketConnector@0.0.0.0:40257 [akka.tcp://spark@saturn00:35894] disassociated! Shutting down. Here, saturn00 is a master and there are 10 nodes in my cluster (saturn01~saturn10) At the last message of the error, what is the meaning of "Driver Disassociated?" How can I resolve this issue? Thanks // Yoonmin Nam // Yoonmin Nam - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
spark application in cluster mode doesn't run correctly
Hello, I tried to execute a simple spark application using sparkSQL. At first try, it worked as I exepcted but after then, it doesn't run and shows an stderr like below: Spark Executor Command: "java" "-cp" "::/opt/spark-1.0.2-bin-hadoop2/conf:/opt/spark-1.0.2-bin-hadoop2/lib/spark-assembly-1.0.2-hadoop2.4.0.jar:/opt/hadoop2/etc/hadoop:/opt/hadoop2/etc/hadoop" "-XX:MaxPermSize=128m" "-Xms14336M" "-Xmx14336M" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "akka.tcp://spark@saturn00:35894/user/CoarseGrainedScheduler" "9" "saturn09" "4" "akka.tcp://sparkWorker@saturn09:45636/user/Worker" "app-20140908223656-" 14/09/08 22:36:57 INFO spark.SecurityManager: Changing view acls to: root14/09/08 22:36:57 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root)14/09/08 22:36:57 INFO slf4j.Slf4jLogger: Slf4jLogger started14/09/08 22:36:57 INFO Remoting: Starting remoting14/09/08 22:36:57 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutor@saturn09:44260]14/09/08 22:36:57 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkExecutor@saturn09:44260]14/09/08 22:36:57 INFO executor.CoarseGrainedExecutorBackend: Connecting to driver: akka.tcp://spark@saturn00:35894/user/CoarseGrainedScheduler14/09/08 22:36:57 INFO worker.WorkerWatcher: Connecting to worker akka.tcp://sparkWorker@saturn09:45636/user/Worker14/09/08 22:36:57 INFO worker.WorkerWatcher: Successfully connected to akka.tcp://sparkWorker@saturn09:45636/user/Worker14/09/08 22:36:57 INFO executor.CoarseGrainedExecutorBackend: Succe ssfully registered with driver14/09/08 22:36:57 INFO spark.SecurityManager: Changing view acls to: root14/09/08 22:36:57 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root)14/09/08 22:36:58 INFO slf4j.Slf4jLogger: Slf4jLogger started14/09/08 22:36:58 INFO Remoting: Starting remoting14/09/08 22:36:58 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@saturn09:39880]14/09/08 22:36:58 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@saturn09:39880]14/09/08 22:36:58 INFO spark.SparkEnv: Connecting to MapOutputTracker: akka.tcp://spark@saturn00:35894/user/MapOutputTracker14/09/08 22:36:58 INFO spark.SparkEnv: Connecting to BlockManagerMaster: akka.tcp://spark@saturn00:35894/user/BlockManagerMaster14/09/08 22:36:58 INFO storage.DiskBlockManager: Created local directory at /hadoop/spark/spark-local-20140908223658-569914/09/08 22:36:58 INFO storage.MemoryStore: MemoryStore s tarted with capacity 4.0 GB.14/09/08 22:36:58 INFO network.ConnectionManager: Bound socket to port 49090 with id = ConnectionManagerId(saturn09,49090)14/09/08 22:36:58 INFO storage.BlockManagerMaster: Trying to register BlockManager14/09/08 22:36:58 INFO storage.BlockManagerMaster: Registered BlockManager14/09/08 22:36:58 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-379704ff-05f2-4c93-8814-ffbe1cc8cd5314/09/08 22:36:58 INFO spark.HttpServer: Starting HTTP Server14/09/08 22:36:58 INFO server.Server: jetty-8.y.z-SNAPSHOT14/09/08 22:36:58 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:40257 [akka.tcp://spark@saturn00:35894] disassociated! Shutting down. Here, saturn00 is a master and there are 10 nodes in my cluster (saturn01~saturn10) At the last message of the error, what is the meaning of "Driver Disassociated?" How can I resolve this issue? Thanks // Yoonmin Nam - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Why spark on yarn applicationmaster cannot get a proper resourcemanager address from yarnconfiguration?
Hello, I tried to submit a spark job to yarn cluster, there is an error occured with those messages: [root@saturn00 bin]# ./spark-submit --class SparkHiveJoin --master yarn-cluster --num-executors 10 --executor-memory 12g --executor-cores 1 spark.jarSpark assembly has been built with Hive, including Datanucleus jars on classpathWarning: Ignoring non-spark config property: yarn.resourcemanager.address=10.150.20.22:8032Warning: Ignoring non-spark config property: yarn.resourcemanager.address=10.150.20.22:803214/09/04 11:01:27 INFO client.RMProxy: Connecting to ResourceManager at /10.150.20.22:803214/09/04 11:01:27 INFO yarn.Client: Got Cluster metric info from ApplicationsManager (ASM), number of NodeManagers: 1014/09/04 11:01:27 INFO yarn.Client: Queue info ... queueName: default, queueCurrentCapacity: 0.0, queueMaxCapacity: 1.0, queueApplicationCount = 0, queueChildQueueCount = 014/09/04 11:01:27 INFO yarn.Client: Max mem capabililty of a single resource in this cluster 1228814 /09/04 11:01:27 INFO yarn.Client: Preparing Local resources14/09/04 11:01:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable14/09/04 11:01:27 INFO yarn.Client: Uploading file:/opt/spark-1.0.2-bin-hadoop2/bin/spark.jar to hdfs://10.150.20.22:8020/user/root/.sparkStaging/application_1409759471992_0007/spark.jar14/09/04 11:01:28 INFO yarn.Client: Uploading file:/opt/spark-1.0.2-bin-hadoop2/lib/spark-assembly-1.0.2-hadoop2.4.0.jar to hdfs://10.150.20.22:8020/user/root/.sparkStaging/application_1409759471992_0007/spark-assembly-1.0.2-hadoop2.4.0.jar14/09/04 11:01:30 INFO yarn.Client: Setting up the launch environment14/09/04 11:01:30 INFO yarn.Client: Setting up container launch context, , /stderr)14/09/04 11:01:30 INFO yarn.Client: Submitting application to ASM14/09/04 11:01:30 INFO impl.YarnClientImpl: Submitted application application_1409759471992_000714/09/04 11:01:31 INFO yarn.Client: Application repo rt from ASM:application identifier: application_1409759471992_0007 appId: 7clientToAMToken: null appDiagnostics: appMasterHost: N/A appQueue: default appMasterRpcPort: -1appStartTime: 1409796090132 yarnAppState: ACCEPTED distributedFinalState: UNDEFINED appTrackingUrl: http://saturn00:8088/proxy/application_1409759471992_0007/ appUser: root14/09/04 11:01:32 INFO yarn.Client: Application report from ASM: application identifier: application_1409759471992_0007 appId: 7 clientToAMToken: null appDiagnostics: appMasterHost: N/A appQueue: default appMasterRpcPort: -1appStartTime: 1409796090132 yarnAppState: ACCEPTED distributedFinalState: UNDEFINEDappTrackingUrl: http://saturn00:8088/proxy/application_1409759471992_0007/ appUser: root So, I found the log of applicationmaster of spark on yarn job, here is the logs: 14/09/04 11:01:33 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1409759471992_0007_01 14/09/04 11:01:33 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8030 14/09/04 11:01:33 INFO SecurityManager: Changing view acls to: root 14/09/04 11:01:33 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root) 14/09/04 11:01:33 INFO ApplicationMaster: Starting the user JAR in a separate Thread 14/09/04 11:01:33 INFO ApplicationMaster: Waiting for Spark context initialization 14/09/04 11:01:33 INFO ApplicationMaster: Waiting for Spark context initialization ... 0 14/09/04 11:01:33 WARN SparkConf: In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN). 14/09/04 11:01:33 INFO SecurityManager: Changing view acls to: root 14/09/04 11:01:33 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root) 14/09/04 11:01:33 INFO Slf4jLogger: Slf4jLogger started 14/09/04 11:01:33 INFO Remoting: Starting remoting 14/09/04 11:01:33 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@saturn02:58444] 14/09/04 11:01:33 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@saturn02:58444] 14/09/04 11:01:33 INFO SparkEnv: Registering MapOutputTracker 14/09/04 11:01:33 INFO SparkEnv: Registering BlockManagerMaster 14/09/04 11:01:33 INFO DiskBlockManager: Created local directory at /home/hadoop/logs/hadoop_tmp/usercache/root/appcache/application_1409759471992_0007/spark-local-20140904110133-a55e 14/09/04 11:01:33 INFO MemoryStore: MemoryStore started with capacity 273.1 MB. 14/09/04 11:01:33 INFO ConnectionManager: Bound socket to port 43903 with id = ConnectionManagerId(saturn02,43903) 14/09/04 11:01:33 INFO BlockManagerMaster: Trying to register BlockManager 14/09/04 11:01:33 INFO BlockManagerInfo: Registering block manager satu
