This exception is so ugly!!! The screen is full of these information when the program runs a long time, and they will not fail the job. I comment it in the source code. I think this information is useless because the executor is already removed and I don't know what does the executor id mean. Should we remove this information forever? 15/07/23 13:26:41 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 2... 15/07/23 13:26:41 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 2...
------------------ 原始邮件 ------------------ 发件人: "Ted Yu";<yuzhih...@gmail.com>; 发送时间: 2015年7月26日(星期天) 晚上10:51 收件人: "Pa Rö"<paul.roewer1...@googlemail.com>; 抄送: "user"<u...@spark.apache.org>; 主题: Re: Asked to remove non-existent executor exception You can list the files in tmpfs in reverse chronological order and remove the oldest until you have enough space. Cheers On Sun, Jul 26, 2015 at 12:43 AM, Pa Rö <paul.roewer1...@googlemail.com> wrote: i has seen that the "tempfs" is full, how i can clear this? 2015-07-23 13:41 GMT+02:00 Pa Rö <paul.roewer1...@googlemail.com>: hello spark community, i have build an application with geomesa, accumulo and spark. if it run on spark local mode, it is working, but on spark cluster not. in short it says: No space left on device. Asked to remove non-existent executor XY. I´m confused, because there were many GB´s of free space. do i need to change my configuration or what else can i do? thanks in advance. here is the complete exception: og4j:WARN No appenders could be found for logger (org.apache.accumulo.fate.zookeeper.ZooSession). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 15/07/23 13:26:39 INFO SparkContext: Running Spark version 1.3.0 15/07/23 13:26:39 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/07/23 13:26:39 INFO SecurityManager: Changing view acls to: marcel 15/07/23 13:26:39 INFO SecurityManager: Changing modify acls to: marcel 15/07/23 13:26:39 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(marcel); users with modify permissions: Set(marcel) 15/07/23 13:26:39 INFO Slf4jLogger: Slf4jLogger started 15/07/23 13:26:40 INFO Remoting: Starting remoting 15/07/23 13:26:40 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@node1-scads02:52478] 15/07/23 13:26:40 INFO Utils: Successfully started service 'sparkDriver' on port 52478. 15/07/23 13:26:40 INFO SparkEnv: Registering MapOutputTracker 15/07/23 13:26:40 INFO SparkEnv: Registering BlockManagerMaster 15/07/23 13:26:40 INFO DiskBlockManager: Created local directory at /tmp/spark-ca9319d4-68a2-4add-a21a-48b13ae9cf81/blockmgr-cbf8af23-e113-4732-8c2c-7413ad237b3b 15/07/23 13:26:40 INFO MemoryStore: MemoryStore started with capacity 1916.2 MB 15/07/23 13:26:40 INFO HttpFileServer: HTTP File server directory is /tmp/spark-9d4a04d5-3535-49e0-a859-d278a0cc7bf8/httpd-1882aafc-45fe-4490-803d-c04fc67510a2 15/07/23 13:26:40 INFO HttpServer: Starting HTTP Server 15/07/23 13:26:40 INFO Server: jetty-8.y.z-SNAPSHOT 15/07/23 13:26:40 INFO AbstractConnector: Started SocketConnector@0.0.0.0:56499 15/07/23 13:26:40 INFO Utils: Successfully started service 'HTTP file server' on port 56499. 15/07/23 13:26:40 INFO SparkEnv: Registering OutputCommitCoordinator 15/07/23 13:26:40 INFO Server: jetty-8.y.z-SNAPSHOT 15/07/23 13:26:40 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 15/07/23 13:26:40 INFO Utils: Successfully started service 'SparkUI' on port 4040. 15/07/23 13:26:40 INFO SparkUI: Started SparkUI at http://node1-scads02:4040 15/07/23 13:26:40 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@node1-scads02:7077/user/Master... 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20150723132640-0000 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added: app-20150723132640-0000/0 on worker-20150723132524-node3-scads06-7078 (node3-scads06:7078) with 8 cores 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150723132640-0000/0 on hostPort node3-scads06:7078 with 8 cores, 512.0 MB RAM 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added: app-20150723132640-0000/1 on worker-20150723132513-node2-scads05-7078 (node2-scads05:7078) with 8 cores 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150723132640-0000/1 on hostPort node2-scads05:7078 with 8 cores, 512.0 MB RAM 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/0 is now RUNNING 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/1 is now RUNNING 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/0 is now LOADING 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/1 is now LOADING 15/07/23 13:26:40 INFO NettyBlockTransferService: Server created on 45786 15/07/23 13:26:40 INFO BlockManagerMaster: Trying to register BlockManager 15/07/23 13:26:40 INFO BlockManagerMasterActor: Registering block manager node1-scads02:45786 with 1916.2 MB RAM, BlockManagerId(<driver>, node1-scads02, 45786) 15/07/23 13:26:40 INFO BlockManagerMaster: Registered BlockManager 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/0 is now FAILED (java.io.IOException: No space left on device) 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Executor app-20150723132640-0000/0 removed: java.io.IOException: No space left on device 15/07/23 13:26:40 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 0 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added: app-20150723132640-0000/2 on worker-20150723132524-node3-scads06-7078 (node3-scads06:7078) with 8 cores 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150723132640-0000/2 on hostPort node3-scads06:7078 with 8 cores, 512.0 MB RAM 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/1 is now FAILED (java.io.IOException: No space left on device) 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Executor app-20150723132640-0000/1 removed: java.io.IOException: No space left on device 15/07/23 13:26:40 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 1 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added: app-20150723132640-0000/3 on worker-20150723132513-node2-scads05-7078 (node2-scads05:7078) with 8 cores 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150723132640-0000/3 on hostPort node2-scads05:7078 with 8 cores, 512.0 MB RAM 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/2 is now LOADING 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/3 is now LOADING 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/2 is now RUNNING 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/3 is now RUNNING 15/07/23 13:26:41 INFO AppClient$ClientActor: Executor updated: app-20150723132640-0000/2 is now FAILED (java.io.IOException: No space left on device) 15/07/23 13:26:41 INFO SparkDeploySchedulerBackend: Executor app-20150723132640-0000/2 removed: java.io.IOException: No space left on device 15/07/23 13:26:41 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 2...