hello spark community, i have build an application with geomesa, accumulo and spark. if it run on spark local mode, it is working, but on spark cluster not. in short it says: No space left on device. Asked to remove non-existent executor XY. I´m confused, because there were many GB´s of free space. do i need to change my configuration or what else can i do? thanks in advance.
here is the complete exception: og4j:WARN No appenders could be found for logger (org.apache.accumulo.fate.zookeeper.ZooSession). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 15/07/23 13:26:39 INFO SparkContext: Running Spark version 1.3.0 15/07/23 13:26:39 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/07/23 13:26:39 INFO SecurityManager: Changing view acls to: marcel 15/07/23 13:26:39 INFO SecurityManager: Changing modify acls to: marcel 15/07/23 13:26:39 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(marcel); users with modify permissions: Set(marcel) 15/07/23 13:26:39 INFO Slf4jLogger: Slf4jLogger started 15/07/23 13:26:40 INFO Remoting: Starting remoting 15/07/23 13:26:40 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@node1-scads02:52478] 15/07/23 13:26:40 INFO Utils: Successfully started service 'sparkDriver' on port 52478. 15/07/23 13:26:40 INFO SparkEnv: Registering MapOutputTracker 15/07/23 13:26:40 INFO SparkEnv: Registering BlockManagerMaster 15/07/23 13:26:40 INFO DiskBlockManager: Created local directory at /tmp/spark-ca9319d4-68a2-4add-a21a-48b13ae9cf81/blockmgr-cbf8af23-e113-4732-8c2c-7413ad237b3b 15/07/23 13:26:40 INFO MemoryStore: MemoryStore started with capacity 1916.2 MB 15/07/23 13:26:40 INFO HttpFileServer: HTTP File server directory is /tmp/spark-9d4a04d5-3535-49e0-a859-d278a0cc7bf8/httpd-1882aafc-45fe-4490-803d-c04fc67510a2 15/07/23 13:26:40 INFO HttpServer: Starting HTTP Server 15/07/23 13:26:40 INFO Server: jetty-8.y.z-SNAPSHOT 15/07/23 13:26:40 INFO AbstractConnector: Started SocketConnector@0.0.0.0:56499 15/07/23 13:26:40 INFO Utils: Successfully started service 'HTTP file server' on port 56499. 15/07/23 13:26:40 INFO SparkEnv: Registering OutputCommitCoordinator 15/07/23 13:26:40 INFO Server: jetty-8.y.z-SNAPSHOT 15/07/23 13:26:40 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 15/07/23 13:26:40 INFO Utils: Successfully started service 'SparkUI' on port 4040. 15/07/23 13:26:40 INFO SparkUI: Started SparkUI at http://node1-scads02:4040 15/07/23 13:26:40 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@node1-scads02:7077/user/Master... 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20150723132640-0000 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added: app-20150723132640-0000/0 on worker-20150723132524-node3-scads06-7078 (node3-scads06:7078) with 8 cores 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150723132640-0000/0 on hostPort node3-scads06:7078 with 8 cores, 512.0 MB RAM 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added: app-20150723132640-0000/1 on worker-20150723132513-node2-scads05-7078 (node2-scads05:7078) with 8 cores 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150723132640-0000/1 on hostPort node2-scads05:7078 with 8 cores, 512.0 MB RAM 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/0 is now RUNNING 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/1 is now RUNNING 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/0 is now LOADING 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/1 is now LOADING 15/07/23 13:26:40 INFO NettyBlockTransferService: Server created on 45786 15/07/23 13:26:40 INFO BlockManagerMaster: Trying to register BlockManager 15/07/23 13:26:40 INFO BlockManagerMasterActor: Registering block manager node1-scads02:45786 with 1916.2 MB RAM, BlockManagerId(<driver>, node1-scads02, 45786) 15/07/23 13:26:40 INFO BlockManagerMaster: Registered BlockManager 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/0 is now FAILED (java.io.IOException: No space left on device) 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Executor app-20150723132640-0000/0 removed: java.io.IOException: No space left on device 15/07/23 13:26:40 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 0 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added: app-20150723132640-0000/2 on worker-20150723132524-node3-scads06-7078 (node3-scads06:7078) with 8 cores 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150723132640-0000/2 on hostPort node3-scads06:7078 with 8 cores, 512.0 MB RAM 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/1 is now FAILED (java.io.IOException: No space left on device) 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Executor app-20150723132640-0000/1 removed: java.io.IOException: No space left on device 15/07/23 13:26:40 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 1 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added: app-20150723132640-0000/3 on worker-20150723132513-node2-scads05-7078 (node2-scads05:7078) with 8 cores 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150723132640-0000/3 on hostPort node2-scads05:7078 with 8 cores, 512.0 MB RAM 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/2 is now LOADING 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/3 is now LOADING 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/2 is now RUNNING 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/3 is now RUNNING 15/07/23 13:26:41 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/2 is now FAILED (java.io.IOException: No space left on device) 15/07/23 13:26:41 INFO SparkDeploySchedulerBackend: Executor app-20150723132640-0000/2 removed: java.io.IOException: No space left on device 15/07/23 13:26:41 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 2...
