Here is the master log: Spark Command: /usr/lib/jvm/java-1.7.0/bin/java -cp ::/root/ephemeral-hdfs/conf:/root/spark/sbin/../conf:/root/spark/lib/spark-assembly-1.3.0-SNAPSHOT-hadoop1.0.4.jar -XX:MaxPermSize=128m -Dspark.akka.logLifecycleEvents=true -Xms512m -Xmx512m org.apache.spark.deploy.master.Master --ip ec2-54-148-20-235.us-west-2.compute.amazonaws.com --port 7077 --webui-port 8080 ========================================
15/01/13 16:25:54 INFO master.Master: Registered signal handlers for [TERM, HUP, INT] 15/01/13 16:25:54 INFO spark.SecurityManager: Changing view acls to: root 15/01/13 16:25:54 INFO spark.SecurityManager: Changing modify acls to: root 15/01/13 16:25:54 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root) 15/01/13 16:25:55 INFO slf4j.Slf4jLogger: Slf4jLogger started 15/01/13 16:25:55 INFO Remoting: Starting remoting 15/01/13 16:25:55 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkmas...@ec2-54-148-20-235.us-west-2.compute.amazonaws.com:7077] 15/01/13 16:25:55 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkmas...@ec2-54-148-20-235.us-west-2.compute.amazonaws.com:7077] 15/01/13 16:25:55 INFO util.Utils: Successfully started service 'sparkMaster' on port 7077. 15/01/13 16:25:55 INFO master.Master: Starting Spark master at spark://ec2-54-148-20-235.us-west-2.compute.amazonaws.com:7077 15/01/13 16:25:55 INFO master.Master: Running Spark version 1.3.0-SNAPSHOT 15/01/13 16:25:55 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/01/13 16:25:55 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:8080 15/01/13 16:25:55 INFO util.Utils: Successfully started service 'MasterUI' on port 8080. 15/01/13 16:25:55 INFO ui.MasterWebUI: Started MasterWebUI at http://ec2-54-148-20-235.us-west-2.compute.amazonaws.com:8080 15/01/13 16:25:55 INFO master.Master: I have been elected leader! New state: ALIVE 15/01/13 16:26:17 INFO master.Master: Registering worker ip-172-31-14-253.us-west-2.compute.internal:36568 with 32 cores, 57.7 GB RAM 15/01/13 16:26:17 INFO master.Master: Registering worker ip-172-31-10-227.us-west-2.compute.internal:50140 with 32 cores, 57.7 GB RAM 15/01/13 16:26:17 INFO master.Master: Registering worker ip-172-31-9-37.us-west-2.compute.internal:34991 with 32 cores, 57.7 GB RAM 15/01/13 16:26:17 INFO master.Master: Registering worker ip-172-31-2-174.us-west-2.compute.internal:58511 with 32 cores, 57.7 GB RAM 15/01/13 16:31:49 INFO master.Master: Registering app test2 15/01/13 16:31:49 INFO master.Master: Registered app test2 with ID app-20150113163149-0000 15/01/13 16:31:49 INFO master.Master: Launching executor app-20150113163149-0000/0 on worker worker-20150113162617-ip-172-31-2-174.us-west-2.compute.internal-58511 15/01/13 16:31:49 INFO master.Master: Launching executor app-20150113163149-0000/1 on worker worker-20150113162617-ip-172-31-14-253.us-west-2.compute.internal-36568 15/01/13 16:31:49 INFO master.Master: Launching executor app-20150113163149-0000/2 on worker worker-20150113162617-ip-172-31-9-37.us-west-2.compute.internal-34991 15/01/13 16:31:49 INFO master.Master: Launching executor app-20150113163149-0000/3 on worker worker-20150113162617-ip-172-31-10-227.us-west-2.compute.internal-50140 15/01/13 16:32:50 INFO master.Master: akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:37612 got disassociated, removing it. 15/01/13 16:32:50 INFO master.Master: Removing app app-20150113163149-0000 15/01/13 16:32:50 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:37612] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 15/01/13 16:32:50 INFO actor.LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%40172.31.10.58%3A35980-5#873817571] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 15/01/13 16:32:50 WARN master.Master: Application test2 is still in progress. 15/01/13 16:32:50 INFO master.Master: akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:37612 got disassociated, removing it. 15/01/13 16:32:51 WARN master.Master: Got status update for unknown executor app-20150113163149-0000/1 15/01/13 16:32:51 WARN master.Master: Got status update for unknown executor app-20150113163149-0000/3 15/01/13 16:32:51 WARN master.Master: Got status update for unknown executor app-20150113163149-0000/2 15/01/13 16:32:52 WARN master.Master: Got status update for unknown executor app-20150113163149-0000/0 15/01/13 16:36:01 INFO master.Master: Registering app test2 15/01/13 16:36:01 INFO master.Master: Registered app test2 with ID app-20150113163601-0001 15/01/13 16:36:01 INFO master.Master: Launching executor app-20150113163601-0001/0 on worker worker-20150113162617-ip-172-31-2-174.us-west-2.compute.internal-58511 15/01/13 16:36:01 INFO master.Master: Launching executor app-20150113163601-0001/1 on worker worker-20150113162617-ip-172-31-14-253.us-west-2.compute.internal-36568 15/01/13 16:36:01 INFO master.Master: Launching executor app-20150113163601-0001/2 on worker worker-20150113162617-ip-172-31-9-37.us-west-2.compute.internal-34991 15/01/13 16:36:01 INFO master.Master: Launching executor app-20150113163601-0001/3 on worker worker-20150113162617-ip-172-31-10-227.us-west-2.compute.internal-50140 15/01/13 16:37:02 INFO master.Master: akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:45081 got disassociated, removing it. 15/01/13 16:37:02 INFO master.Master: Removing app app-20150113163601-0001 15/01/13 16:37:02 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:45081] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 15/01/13 16:37:02 WARN master.Master: Application test2 is still in progress. 15/01/13 16:37:02 INFO actor.LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%40172.31.10.58%3A36006-6#490113048] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 15/01/13 16:37:02 INFO master.Master: akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:45081 got disassociated, removing it. 15/01/13 16:37:03 WARN master.Master: Got status update for unknown executor app-20150113163601-0001/3 15/01/13 16:37:03 WARN master.Master: Got status update for unknown executor app-20150113163601-0001/2 15/01/13 16:37:03 WARN master.Master: Got status update for unknown executor app-20150113163601-0001/0 15/01/13 16:37:04 WARN master.Master: Got status update for unknown executor app-20150113163601-0001/1 15/01/13 17:18:44 INFO master.Master: Registering app test2 15/01/13 17:18:44 INFO master.Master: Registered app test2 with ID app-20150113171844-0002 15/01/13 17:18:44 INFO master.Master: Launching executor app-20150113171844-0002/0 on worker worker-20150113162617-ip-172-31-2-174.us-west-2.compute.internal-58511 15/01/13 17:18:44 INFO master.Master: Launching executor app-20150113171844-0002/1 on worker worker-20150113162617-ip-172-31-14-253.us-west-2.compute.internal-36568 15/01/13 17:18:44 INFO master.Master: Launching executor app-20150113171844-0002/2 on worker worker-20150113162617-ip-172-31-9-37.us-west-2.compute.internal-34991 15/01/13 17:18:44 INFO master.Master: Launching executor app-20150113171844-0002/3 on worker worker-20150113162617-ip-172-31-10-227.us-west-2.compute.internal-50140 15/01/13 17:19:44 INFO master.Master: akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:48007 got disassociated, removing it. 15/01/13 17:19:44 INFO master.Master: Removing app app-20150113171844-0002 15/01/13 17:19:44 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:48007] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 15/01/13 17:19:44 WARN master.Master: Application test2 is still in progress. 15/01/13 17:19:44 INFO actor.LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%40172.31.10.58%3A36195-7#-1186518698] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 15/01/13 17:19:44 INFO master.Master: akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:48007 got disassociated, removing it. 15/01/13 17:19:46 WARN master.Master: Got status update for unknown executor app-20150113171844-0002/0 15/01/13 17:19:46 WARN master.Master: Got status update for unknown executor app-20150113171844-0002/2 15/01/13 17:19:46 WARN master.Master: Got status update for unknown executor app-20150113171844-0002/3 15/01/13 17:19:46 WARN master.Master: Got status update for unknown executor app-20150113171844-0002/1 15/01/13 18:59:36 INFO master.Master: Registering app test2 15/01/13 18:59:36 INFO master.Master: Registered app test2 with ID app-20150113185936-0003 15/01/13 18:59:36 INFO master.Master: Launching executor app-20150113185936-0003/0 on worker worker-20150113162617-ip-172-31-2-174.us-west-2.compute.internal-58511 15/01/13 18:59:36 INFO master.Master: Launching executor app-20150113185936-0003/1 on worker worker-20150113162617-ip-172-31-14-253.us-west-2.compute.internal-36568 15/01/13 18:59:36 INFO master.Master: Launching executor app-20150113185936-0003/2 on worker worker-20150113162617-ip-172-31-9-37.us-west-2.compute.internal-34991 15/01/13 18:59:36 INFO master.Master: Launching executor app-20150113185936-0003/3 on worker worker-20150113162617-ip-172-31-10-227.us-west-2.compute.internal-50140 15/01/13 18:59:36 INFO master.Master: akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:36546 got disassociated, removing it. 15/01/13 18:59:36 INFO master.Master: Removing app app-20150113185936-0003 15/01/13 18:59:36 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:36546] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 15/01/13 18:59:36 INFO actor.LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%40172.31.10.58%3A36639-8#-134281562] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 15/01/13 18:59:36 WARN master.Master: No event logs found for application test2 in file:///tmp/spark-events/app-20150113185936-0003. 15/01/13 18:59:36 INFO master.Master: akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:36546 got disassociated, removing it. 15/01/13 18:59:36 WARN master.Master: Got status update for unknown executor app-20150113185936-0003/1 15/01/13 18:59:36 WARN master.Master: Got status update for unknown executor app-20150113185936-0003/3 15/01/13 18:59:36 WARN master.Master: Got status update for unknown executor app-20150113185936-0003/0 15/01/13 18:59:36 WARN master.Master: Got status update for unknown executor app-20150113185936-0003/2 15/01/13 19:00:25 INFO master.Master: Registering app test2 15/01/13 19:00:25 INFO master.Master: Registered app test2 with ID app-20150113190025-0004 15/01/13 19:00:25 INFO master.Master: Launching executor app-20150113190025-0004/0 on worker worker-20150113162617-ip-172-31-2-174.us-west-2.compute.internal-58511 15/01/13 19:00:25 INFO master.Master: Launching executor app-20150113190025-0004/1 on worker worker-20150113162617-ip-172-31-14-253.us-west-2.compute.internal-36568 15/01/13 19:00:25 INFO master.Master: Launching executor app-20150113190025-0004/2 on worker worker-20150113162617-ip-172-31-9-37.us-west-2.compute.internal-34991 15/01/13 19:00:25 INFO master.Master: Launching executor app-20150113190025-0004/3 on worker worker-20150113162617-ip-172-31-10-227.us-west-2.compute.internal-50140 15/01/13 19:01:25 INFO master.Master: akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:40167 got disassociated, removing it. 15/01/13 19:01:25 INFO master.Master: Removing app app-20150113190025-0004 15/01/13 19:01:25 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:40167] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 15/01/13 19:01:25 INFO actor.LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%40172.31.10.58%3A36648-9#-1156014080] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 15/01/13 19:01:25 WARN master.Master: Application test2 is still in progress. 15/01/13 19:01:25 INFO master.Master: akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:40167 got disassociated, removing it. 15/01/13 19:01:27 WARN master.Master: Got status update for unknown executor app-20150113190025-0004/2 15/01/13 19:01:27 WARN master.Master: Got status update for unknown executor app-20150113190025-0004/1 15/01/13 19:01:27 WARN master.Master: Got status update for unknown executor app-20150113190025-0004/3 15/01/13 19:01:28 WARN master.Master: Got status update for unknown executor app-20150113190025-0004/0 15/01/13 20:17:43 INFO master.Master: Registering app test2 15/01/13 20:17:43 INFO master.Master: Registered app test2 with ID app-20150113201743-0005 15/01/13 20:17:43 INFO master.Master: Launching executor app-20150113201743-0005/0 on worker worker-20150113162617-ip-172-31-2-174.us-west-2.compute.internal-58511 15/01/13 20:17:43 INFO master.Master: Launching executor app-20150113201743-0005/1 on worker worker-20150113162617-ip-172-31-14-253.us-west-2.compute.internal-36568 15/01/13 20:17:43 INFO master.Master: Launching executor app-20150113201743-0005/2 on worker worker-20150113162617-ip-172-31-9-37.us-west-2.compute.internal-34991 15/01/13 20:17:43 INFO master.Master: Launching executor app-20150113201743-0005/3 on worker worker-20150113162617-ip-172-31-10-227.us-west-2.compute.internal-50140 15/01/13 20:18:44 INFO master.Master: akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:36658 got disassociated, removing it. 15/01/13 20:18:44 INFO master.Master: Removing app app-20150113201743-0005 15/01/13 20:18:44 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:36658] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 15/01/13 20:18:44 WARN master.Master: Application test2 is still in progress. 15/01/13 20:18:44 INFO actor.LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%40172.31.10.58%3A36986-10#781099781] was not delivered. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 15/01/13 20:18:44 INFO master.Master: akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:36658 got disassociated, removing it. 15/01/13 20:18:45 WARN master.Master: Got status update for unknown executor app-20150113201743-0005/2 15/01/13 20:18:45 WARN master.Master: Got status update for unknown executor app-20150113201743-0005/0 15/01/13 20:18:45 WARN master.Master: Got status update for unknown executor app-20150113201743-0005/1 15/01/13 20:18:45 WARN master.Master: Got status update for unknown executor app-20150113201743-0005/3
