I recently moved my Spark installation from one Linux user to another one, i.e. changed the folder and ownership of the files. That was everything, no other settings were changed or different machines used.
However, now it suddenly takes three minutes to have all executors in the Spark shell ready, instead of about 10 seconds as it was before. They'll be listed in the "Application: Spark shell" page, but not in "Application Detail UI" - "Executors"-Tab. Only the master driver and executor are there. Can anyone think of an explanation for this? Here's a log of the startup, you can see that the delay is exactly three minutes: spark.logConf=true spark.master=spark://idp19:10000 spark.repl.class.uri=http://172.16.0.19:46457 spark.speculation=true 14/08/08 14:54:29 INFO spark.SecurityManager: Changing view acls to: myuser 14/08/08 14:54:29 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(myuser) 4.400: [GC (Metadata GC Threshold) [PSYoungGen: 188763K->30233K(1223168K)] 248423K->89902K(4019712K), 0.0468201 secs] [Times: user=0.26 sys=0.04, real=0.05 secs] 4.447: [Full GC (Metadata GC Threshold) [PSYoungGen: 30233K->0K(1223168K)] [ParOldGen: 59668K->77437K(2796544K)] 89902K->77437K(4019712K), [Metaspace: 34757K->34757K(1079296K)], 0.1397987 secs] [Times: user=0.77 sys=0.02, real=0.14 secs] 14/08/08 14:54:30 INFO slf4j.Slf4jLogger: Slf4jLogger started 14/08/08 14:54:30 INFO Remoting: Starting remoting 14/08/08 14:54:30 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sp...@idp19.foo.bar:55806] 14/08/08 14:54:30 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sp...@idp19.foo.bar:55806] 14/08/08 14:54:30 INFO spark.SparkEnv: Registering MapOutputTracker 14/08/08 14:54:30 INFO spark.SparkEnv: Registering BlockManagerMaster 14/08/08 14:54:30 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-local-20140808145430-8ede 14/08/08 14:54:30 INFO network.ConnectionManager: Bound socket to port 51497 with id = ConnectionManagerId(idp19.foo.bar,51497) 14/08/08 14:54:30 INFO storage.MemoryStore: MemoryStore started with capacity 2.1 GB 14/08/08 14:54:30 INFO storage.BlockManagerMaster: Trying to register BlockManager 14/08/08 14:54:30 INFO storage.BlockManagerMasterActor: Registering block manager idp19.foo.bar:51497 with 2.1 GB RAM 14/08/08 14:54:30 INFO storage.BlockManagerMaster: Registered BlockManager 14/08/08 14:54:30 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-76fec37d-3812-4034-94e7-0456a4bc76dc 14/08/08 14:54:30 INFO spark.HttpServer: Starting HTTP Server 14/08/08 14:54:30 INFO server.Server: jetty-8.y.z-SNAPSHOT 14/08/08 14:54:30 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:56276 14/08/08 14:54:30 INFO server.Server: jetty-8.y.z-SNAPSHOT 14/08/08 14:54:30 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 14/08/08 14:54:30 INFO ui.SparkUI: Started SparkUI at http://idp19.foo.bar:4040 14/08/08 14:54:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/08/08 14:54:31 WARN hdfs.DomainSocketFactory: The short-circuit local reads feature is disabled because libhadoop cannot be loaded. 14/08/08 14:54:31 INFO scheduler.EventLoggingListener: Logging events to hdfs://idp11:9100/user/myuser/logs/spark-shell-1407534870777 14/08/08 14:54:31 INFO scheduler.TaskSchedulerImpl: Starting speculative execution thread 14/08/08 14:54:31 INFO client.AppClient$ClientActor: Connecting to master spark://idp19:10000... 14/08/08 14:54:31 INFO repl.SparkILoop: Created spark context.. Spark context available as sc. scala> 14/08/08 14:54:31 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20140808145431-0001 14/08/08 14:54:31 INFO client.AppClient$ClientActor: Executor added: app-20140808145431-0001/0 on worker-20140808145140-idp09.foo.bar-11000 (idp09.foo.bar:11000) with 8 cores 14/08/08 14:54:31 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20140808145431-0001/0 on hostPort idp09.foo.bar:11000 with 8 cores, 25.0 GB RAM 14/08/08 14:54:31 INFO client.AppClient$ClientActor: Executor added: app-20140808145431-0001/1 on worker-20140808145427-idp42.foo.bar-11000 (idp42.foo.bar:11000) with 8 cores 14/08/08 14:54:31 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20140808145431-0001/1 on hostPort idp42.foo.bar:11000 with 8 cores, 25.0 GB RAM 14/08/08 14:54:31 INFO client.AppClient$ClientActor: Executor added: app-20140808145431-0001/2 on worker-20140808145049-idp41.foo.bar-11000 (idp41.foo.bar:11000) with 8 cores 14/08/08 14:54:31 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20140808145431-0001/2 on hostPort idp41.foo.bar:11000 with 8 cores, 25.0 GB RAM 14/08/08 14:54:31 INFO client.AppClient$ClientActor: Executor added: app-20140808145431-0001/3 on worker-20140808144952-idp19.foo.bar-11000 (idp19.foo.bar:11000) with 8 cores 14/08/08 14:54:31 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20140808145431-0001/3 on hostPort idp19.foo.bar:11000 with 8 cores, 25.0 GB RAM 14/08/08 14:54:31 INFO client.AppClient$ClientActor: Executor added: app-20140808145431-0001/4 on worker-20140808145044-idp11.foo.bar-11000 (idp11.foo.bar:11000) with 8 cores 14/08/08 14:54:31 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20140808145431-0001/4 on hostPort idp11.foo.bar:11000 with 8 cores, 25.0 GB RAM 14/08/08 14:54:31 INFO client.AppClient$ClientActor: Executor updated: app-20140808145431-0001/0 is now RUNNING 14/08/08 14:54:31 INFO client.AppClient$ClientActor: Executor updated: app-20140808145431-0001/2 is now RUNNING 14/08/08 14:54:31 INFO client.AppClient$ClientActor: Executor updated: app-20140808145431-0001/1 is now RUNNING 14/08/08 14:54:31 INFO client.AppClient$ClientActor: Executor updated: app-20140808145431-0001/3 is now RUNNING 14/08/08 14:54:31 INFO client.AppClient$ClientActor: Executor updated: app-20140808145431-0001/4 is now RUNNING 6.905: [GC (Metadata GC Threshold) [PSYoungGen: 281152K->24461K(1223168K)] 358590K->101971K(4019712K), 0.0331379 secs] [Times: user=0.07 sys=0.01, real=0.03 secs] 6.939: [Full GC (Metadata GC Threshold) [PSYoungGen: 24461K->0K(1223168K)] [ParOldGen: 77509K->88552K(2796544K)] 101971K->88552K(4019712K), [Metaspace: 57918K->57918K(1097728K)], 0.4788130 secs] [Times: user=2.57 sys=0.02, real=0.48 secs] 14/08/08 14:54:34 INFO cluster.SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkexecu...@idp19.foo.bar:33460/user/Executor#358355180] with ID 3 14/08/08 14:54:34 INFO storage.BlockManagerMasterActor: Registering block manager idp19.foo.bar:60334 with 12.9 GB RAM 14/08/08 14:57:34 INFO cluster.SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkexecu...@idp09.foo.bar:44179/user/Executor#1400029746] with ID 0 14/08/08 14:57:34 INFO cluster.SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkexecu...@idp42.foo.bar:51985/user/Executor#-2108280549] with ID 1 14/08/08 14:57:34 INFO cluster.SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkexecu...@idp11.foo.bar:52008/user/Executor#-1865221922] with ID 4 14/08/08 14:57:34 INFO cluster.SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkexecu...@idp41.foo.bar:38568/user/Executor#1804089900] with ID 2 14/08/08 14:57:34 INFO storage.BlockManagerMasterActor: Registering block manager idp09.foo.bar:52232 with 12.9 GB RAM 14/08/08 14:57:34 INFO storage.BlockManagerMasterActor: Registering block manager idp42.foo.bar:39259 with 12.9 GB RAM 14/08/08 14:57:34 INFO storage.BlockManagerMasterActor: Registering block manager idp11.foo.bar:37555 with 12.9 GB RAM 14/08/08 14:57:34 INFO storage.BlockManagerMasterActor: Registering block manager idp41.foo.bar:48988 with 12.9 GB RAM -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Executors-for-Spark-shell-take-much-longer-to-be-ready-tp11822.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
