This exception is so ugly!!! The screen is full of these information when the
program runs a long time, and they will not fail the job.
I comment it in the source code. I think this information is useless because
the executor is already removed and I don't know what does the executor id mean.
Should we remove this information forever?
15/07/23 13:26:41 ERROR SparkDeploySchedulerBackend: Asked to remove
non-existent executor 2...
15/07/23 13:26:41 ERROR SparkDeploySchedulerBackend: Asked to remove
non-existent executor 2...
------------------ 原始邮件 ------------------
发件人: "Ted Yu";<yuzhih...@gmail.com>;
发送时间: 2015年7月26日(星期天) 晚上10:51
收件人: "Pa Rö"<paul.roewer1...@googlemail.com>;
抄送: "user"<u...@spark.apache.org>;
主题: Re: Asked to remove non-existent executor exception
You can list the files in tmpfs in reverse chronological order and remove the
oldest until you have enough space.
Cheers
On Sun, Jul 26, 2015 at 12:43 AM, Pa Rö <paul.roewer1...@googlemail.com> wrote:
i has seen that the "tempfs" is full, how i can clear this?
2015-07-23 13:41 GMT+02:00 Pa Rö <paul.roewer1...@googlemail.com>:
hello spark community,
i have build an application with geomesa, accumulo and spark.
if it run on spark local mode, it is working, but on spark
cluster not. in short it says: No space left on device. Asked to remove
non-existent executor XY.
I´m confused, because there were many GB´s of free space. do i need to change
my configuration or what else can i do? thanks in advance.
here is the complete exception:
og4j:WARN No appenders could be found for logger
(org.apache.accumulo.fate.zookeeper.ZooSession).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/07/23 13:26:39 INFO SparkContext: Running Spark version 1.3.0
15/07/23 13:26:39 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
15/07/23 13:26:39 INFO SecurityManager: Changing view acls to: marcel
15/07/23 13:26:39 INFO SecurityManager: Changing modify acls to: marcel
15/07/23 13:26:39 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(marcel); users
with modify permissions: Set(marcel)
15/07/23 13:26:39 INFO Slf4jLogger: Slf4jLogger started
15/07/23 13:26:40 INFO Remoting: Starting remoting
15/07/23 13:26:40 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkDriver@node1-scads02:52478]
15/07/23 13:26:40 INFO Utils: Successfully started service 'sparkDriver' on
port 52478.
15/07/23 13:26:40 INFO SparkEnv: Registering MapOutputTracker
15/07/23 13:26:40 INFO SparkEnv: Registering BlockManagerMaster
15/07/23 13:26:40 INFO DiskBlockManager: Created local directory at
/tmp/spark-ca9319d4-68a2-4add-a21a-48b13ae9cf81/blockmgr-cbf8af23-e113-4732-8c2c-7413ad237b3b
15/07/23 13:26:40 INFO MemoryStore: MemoryStore started with capacity 1916.2 MB
15/07/23 13:26:40 INFO HttpFileServer: HTTP File server directory is
/tmp/spark-9d4a04d5-3535-49e0-a859-d278a0cc7bf8/httpd-1882aafc-45fe-4490-803d-c04fc67510a2
15/07/23 13:26:40 INFO HttpServer: Starting HTTP Server
15/07/23 13:26:40 INFO Server: jetty-8.y.z-SNAPSHOT
15/07/23 13:26:40 INFO AbstractConnector: Started SocketConnector@0.0.0.0:56499
15/07/23 13:26:40 INFO Utils: Successfully started service 'HTTP file server'
on port 56499.
15/07/23 13:26:40 INFO SparkEnv: Registering OutputCommitCoordinator
15/07/23 13:26:40 INFO Server: jetty-8.y.z-SNAPSHOT
15/07/23 13:26:40 INFO AbstractConnector: Started
SelectChannelConnector@0.0.0.0:4040
15/07/23 13:26:40 INFO Utils: Successfully started service 'SparkUI' on port
4040.
15/07/23 13:26:40 INFO SparkUI: Started SparkUI at http://node1-scads02:4040
15/07/23 13:26:40 INFO AppClient$ClientActor: Connecting to master
akka.tcp://sparkMaster@node1-scads02:7077/user/Master...
15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Connected to Spark cluster
with app ID app-20150723132640-0000
15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added:
app-20150723132640-0000/0 on worker-20150723132524-node3-scads06-7078
(node3-scads06:7078) with 8 cores
15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20150723132640-0000/0 on hostPort node3-scads06:7078 with 8 cores, 512.0 MB
RAM
15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added:
app-20150723132640-0000/1 on worker-20150723132513-node2-scads05-7078
(node2-scads05:7078) with 8 cores
15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20150723132640-0000/1 on hostPort node2-scads05:7078 with 8 cores, 512.0 MB
RAM
15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
app-20150723132640-0000/0 is now RUNNING
15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
app-20150723132640-0000/1 is now RUNNING
15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
app-20150723132640-0000/0 is now LOADING
15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
app-20150723132640-0000/1 is now LOADING
15/07/23 13:26:40 INFO NettyBlockTransferService: Server created on 45786
15/07/23 13:26:40 INFO BlockManagerMaster: Trying to register BlockManager
15/07/23 13:26:40 INFO BlockManagerMasterActor: Registering block manager
node1-scads02:45786 with 1916.2 MB RAM, BlockManagerId(<driver>, node1-scads02,
45786)
15/07/23 13:26:40 INFO BlockManagerMaster: Registered BlockManager
15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
app-20150723132640-0000/0 is now FAILED (java.io.IOException: No space left on
device)
15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Executor
app-20150723132640-0000/0 removed: java.io.IOException: No space left on device
15/07/23 13:26:40 ERROR SparkDeploySchedulerBackend: Asked to remove
non-existent executor 0
15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added:
app-20150723132640-0000/2 on worker-20150723132524-node3-scads06-7078
(node3-scads06:7078) with 8 cores
15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20150723132640-0000/2 on hostPort node3-scads06:7078 with 8 cores, 512.0 MB
RAM
15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
app-20150723132640-0000/1 is now FAILED (java.io.IOException: No space left on
device)
15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Executor
app-20150723132640-0000/1 removed: java.io.IOException: No space left on device
15/07/23 13:26:40 ERROR SparkDeploySchedulerBackend: Asked to remove
non-existent executor 1
15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added:
app-20150723132640-0000/3 on worker-20150723132513-node2-scads05-7078
(node2-scads05:7078) with 8 cores
15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20150723132640-0000/3 on hostPort node2-scads05:7078 with 8 cores, 512.0 MB
RAM
15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
app-20150723132640-0000/2 is now LOADING
15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
app-20150723132640-0000/3 is now LOADING
15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
app-20150723132640-0000/2 is now RUNNING
15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready
for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
app-20150723132640-0000/3 is now RUNNING
15/07/23 13:26:41 INFO AppClient$ClientActor: Executor updated:
app-20150723132640-0000/2 is now FAILED (java.io.IOException: No space left on
device)
15/07/23 13:26:41 INFO SparkDeploySchedulerBackend: Executor
app-20150723132640-0000/2 removed: java.io.IOException: No space left on device
15/07/23 13:26:41 ERROR SparkDeploySchedulerBackend: Asked to remove
non-existent executor 2...
