Simply customize your log4j confit instead of modifying code if you don't want messages from that class.
Regards Mridul On Sunday, July 26, 2015, Sea <261810...@qq.com> wrote: > This exception is so ugly!!! The screen is full of these information when > the program runs a long time, and they will not fail the job. > > I comment it in the source code. I think this information is useless > because the executor is already removed and I don't know what does the > executor id mean. > > Should we remove this information forever? > > > > 15/07/23 13:26:41 ERROR SparkDeploySchedulerBackend: Asked to remove > non-existent executor 2... > > 15/07/23 13:26:41 ERROR SparkDeploySchedulerBackend: Asked to remove > non-existent executor 2... > > > > > > > > ------------------ 原始邮件 ------------------ > *发件人:* "Ted Yu";<yuzhih...@gmail.com > <javascript:_e(%7B%7D,'cvml','yuzhih...@gmail.com');>>; > *发送时间:* 2015年7月26日(星期天) 晚上10:51 > *收件人:* "Pa Rö"<paul.roewer1...@googlemail.com > <javascript:_e(%7B%7D,'cvml','paul.roewer1...@googlemail.com');>>; > *抄送:* "user"<u...@spark.apache.org > <javascript:_e(%7B%7D,'cvml','u...@spark.apache.org');>>; > *主题:* Re: Asked to remove non-existent executor exception > > You can list the files in tmpfs in reverse chronological order and remove > the oldest until you have enough space. > > Cheers > > On Sun, Jul 26, 2015 at 12:43 AM, Pa Rö <paul.roewer1...@googlemail.com > <javascript:_e(%7B%7D,'cvml','paul.roewer1...@googlemail.com');>> wrote: > >> i has seen that the "tempfs" is full, how i can clear this? >> >> 2015-07-23 13:41 GMT+02:00 Pa Rö <paul.roewer1...@googlemail.com >> <javascript:_e(%7B%7D,'cvml','paul.roewer1...@googlemail.com');>>: >> >>> hello spark community, >>> >>> i have build an application with geomesa, accumulo and spark. >>> if it run on spark local mode, it is working, but on spark >>> cluster not. in short it says: No space left on device. Asked to remove >>> non-existent executor XY. >>> I´m confused, because there were many GB´s of free space. do i need to >>> change my configuration or what else can i do? thanks in advance. >>> >>> here is the complete exception: >>> >>> og4j:WARN No appenders could be found for logger >>> (org.apache.accumulo.fate.zookeeper.ZooSession). >>> log4j:WARN Please initialize the log4j system properly. >>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig >>> for more info. >>> Using Spark's default log4j profile: >>> org/apache/spark/log4j-defaults.properties >>> 15/07/23 13:26:39 INFO SparkContext: Running Spark version 1.3.0 >>> 15/07/23 13:26:39 WARN NativeCodeLoader: Unable to load native-hadoop >>> library for your platform... using builtin-java classes where applicable >>> 15/07/23 13:26:39 INFO SecurityManager: Changing view acls to: marcel >>> 15/07/23 13:26:39 INFO SecurityManager: Changing modify acls to: marcel >>> 15/07/23 13:26:39 INFO SecurityManager: SecurityManager: authentication >>> disabled; ui acls disabled; users with view permissions: Set(marcel); users >>> with modify permissions: Set(marcel) >>> 15/07/23 13:26:39 INFO Slf4jLogger: Slf4jLogger started >>> 15/07/23 13:26:40 INFO Remoting: Starting remoting >>> 15/07/23 13:26:40 INFO Remoting: Remoting started; listening on >>> addresses :[akka.tcp://sparkDriver@node1-scads02:52478] >>> 15/07/23 13:26:40 INFO Utils: Successfully started service 'sparkDriver' >>> on port 52478. >>> 15/07/23 13:26:40 INFO SparkEnv: Registering MapOutputTracker >>> 15/07/23 13:26:40 INFO SparkEnv: Registering BlockManagerMaster >>> 15/07/23 13:26:40 INFO DiskBlockManager: Created local directory at >>> /tmp/spark-ca9319d4-68a2-4add-a21a-48b13ae9cf81/blockmgr-cbf8af23-e113-4732-8c2c-7413ad237b3b >>> 15/07/23 13:26:40 INFO MemoryStore: MemoryStore started with capacity >>> 1916.2 MB >>> 15/07/23 13:26:40 INFO HttpFileServer: HTTP File server directory is >>> /tmp/spark-9d4a04d5-3535-49e0-a859-d278a0cc7bf8/httpd-1882aafc-45fe-4490-803d-c04fc67510a2 >>> 15/07/23 13:26:40 INFO HttpServer: Starting HTTP Server >>> 15/07/23 13:26:40 INFO Server: jetty-8.y.z-SNAPSHOT >>> 15/07/23 13:26:40 INFO AbstractConnector: Started >>> SocketConnector@0.0.0.0:56499 >>> 15/07/23 13:26:40 INFO Utils: Successfully started service 'HTTP file >>> server' on port 56499. >>> 15/07/23 13:26:40 INFO SparkEnv: Registering OutputCommitCoordinator >>> 15/07/23 13:26:40 INFO Server: jetty-8.y.z-SNAPSHOT >>> 15/07/23 13:26:40 INFO AbstractConnector: Started >>> SelectChannelConnector@0.0.0.0:4040 >>> 15/07/23 13:26:40 INFO Utils: Successfully started service 'SparkUI' on >>> port 4040. >>> 15/07/23 13:26:40 INFO SparkUI: Started SparkUI at >>> http://node1-scads02:4040 >>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Connecting to master >>> akka.tcp://sparkMaster@node1-scads02:7077/user/Master... >>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Connected to Spark >>> cluster with app ID app-20150723132640-0000 >>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added: >>> app-20150723132640-0000/0 on worker-20150723132524-node3-scads06-7078 >>> (node3-scads06:7078) with 8 cores >>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID >>> app-20150723132640-0000/0 on hostPort node3-scads06:7078 with 8 cores, >>> 512.0 MB RAM >>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added: >>> app-20150723132640-0000/1 on worker-20150723132513-node2-scads05-7078 >>> (node2-scads05:7078) with 8 cores >>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID >>> app-20150723132640-0000/1 on hostPort node2-scads05:7078 with 8 cores, >>> 512.0 MB RAM >>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: >>> app-20150723132640-0000/0 is now RUNNING >>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: >>> app-20150723132640-0000/1 is now RUNNING >>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: >>> app-20150723132640-0000/0 is now LOADING >>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: >>> app-20150723132640-0000/1 is now LOADING >>> 15/07/23 13:26:40 INFO NettyBlockTransferService: Server created on 45786 >>> 15/07/23 13:26:40 INFO BlockManagerMaster: Trying to register >>> BlockManager >>> 15/07/23 13:26:40 INFO BlockManagerMasterActor: Registering block >>> manager node1-scads02:45786 with 1916.2 MB RAM, BlockManagerId(<driver>, >>> node1-scads02, 45786) >>> 15/07/23 13:26:40 INFO BlockManagerMaster: Registered BlockManager >>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: >>> app-20150723132640-0000/0 is now FAILED (java.io.IOException: No space left >>> on device) >>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Executor >>> app-20150723132640-0000/0 removed: java.io.IOException: No space left on >>> device >>> 15/07/23 13:26:40 ERROR SparkDeploySchedulerBackend: Asked to remove >>> non-existent executor 0 >>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added: >>> app-20150723132640-0000/2 on worker-20150723132524-node3-scads06-7078 >>> (node3-scads06:7078) with 8 cores >>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID >>> app-20150723132640-0000/2 on hostPort node3-scads06:7078 with 8 cores, >>> 512.0 MB RAM >>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: >>> app-20150723132640-0000/1 is now FAILED (java.io.IOException: No space left >>> on device) >>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Executor >>> app-20150723132640-0000/1 removed: java.io.IOException: No space left on >>> device >>> 15/07/23 13:26:40 ERROR SparkDeploySchedulerBackend: Asked to remove >>> non-existent executor 1 >>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added: >>> app-20150723132640-0000/3 on worker-20150723132513-node2-scads05-7078 >>> (node2-scads05:7078) with 8 cores >>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID >>> app-20150723132640-0000/3 on hostPort node2-scads05:7078 with 8 cores, >>> 512.0 MB RAM >>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: >>> app-20150723132640-0000/2 is now LOADING >>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: >>> app-20150723132640-0000/3 is now LOADING >>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: >>> app-20150723132640-0000/2 is now RUNNING >>> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: SchedulerBackend is >>> ready for scheduling beginning after reached minRegisteredResourcesRatio: >>> 0.0 >>> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated: >>> app-20150723132640-0000/3 is now RUNNING >>> 15/07/23 13:26:41 INFO AppClient$ClientActor: Executor updated: >>> app-20150723132640-0000/2 is now FAILED (java.io.IOException: No space left >>> on device) >>> 15/07/23 13:26:41 INFO SparkDeploySchedulerBackend: Executor >>> app-20150723132640-0000/2 removed: java.io.IOException: No space left on >>> device >>> 15/07/23 13:26:41 ERROR SparkDeploySchedulerBackend: Asked to remove >>> non-existent executor 2... >>> >> >> >
