Hi Yana,
As per my custom split code, only three splits submit to the system. So
three executors are sufficient for that. but it had run 8 executors. First
three executors logs show the exact output what I want(i did put some syso
in console to debug the code), but next five are have some other and random
exceptions.
I think it is due to first three executor didn't exist properly thatswy
driver run more executors on top it, which create so many processes hitting
the same application and overall result it fails.
from Log i can see first three executors return with exit status 1. and
logs are below :
15/01/23 15:51:39 INFO executor.CoarseGrainedExecutorBackend: Registered
signal handlers for [TERM, HUP, INT]
15/01/23 15:51:39 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/01/23 15:51:40 INFO spark.SecurityManager: Changing view acls to:
sparkAdmin
15/01/23 15:51:40 INFO spark.SecurityManager: Changing modify acls to:
sparkAdmin
15/01/23 15:51:40 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(sparkAdmin); users with modify permissions: Set(sparkAdmin)
15/01/23 15:51:40 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/01/23 15:51:40 INFO Remoting: Starting remoting
15/01/23 15:51:41 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://driverPropsFetcher@VM219:40166]
15/01/23 15:51:41 INFO util.Utils: Successfully started service
'driverPropsFetcher' on port 40166.
15/01/23 15:51:41 INFO spark.SecurityManager: Changing view acls to:
sparkAdmin
15/01/23 15:51:41 INFO spark.SecurityManager: Changing modify acls to:
sparkAdmin
15/01/23 15:51:41 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(sparkAdmin); users with modify permissions: Set(sparkAdmin)
15/01/23 15:51:41 INFO remote.RemoteActorRefProvider$RemotingTerminator:
Shutting down remote daemon.
15/01/23 15:51:41 INFO remote.RemoteActorRefProvider$RemotingTerminator:
Remote daemon shut down; proceeding with flushing remote transports.
15/01/23 15:51:41 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/01/23 15:51:41 INFO Remoting: Starting remoting
15/01/23 15:51:41 INFO remote.RemoteActorRefProvider$RemotingTerminator:
Remoting shut down.
15/01/23 15:51:41 INFO util.Utils: Successfully started service
'sparkExecutor' on port 57695.
15/01/23 15:51:41 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkExecutor@VM219:57695]
15/01/23 15:51:41 INFO executor.CoarseGrainedExecutorBackend: Connecting to
driver: akka.tcp://sparkDriver@VM220:53484/user/CoarseGrainedScheduler
15/01/23 15:51:41 INFO worker.WorkerWatcher: Connecting to worker
akka.tcp://sparkWorker@VM219:44826/user/Worker
15/01/23 15:51:41 INFO worker.WorkerWatcher: Successfully connected to
akka.tcp://sparkWorker@VM219:44826/user/Worker
15/01/23 15:51:41 INFO executor.CoarseGrainedExecutorBackend: Successfully
registered with driver
15/01/23 15:51:41 INFO spark.SecurityManager: Changing view acls to:
sparkAdmin
15/01/23 15:51:41 INFO spark.SecurityManager: Changing modify acls to:
sparkAdmin
15/01/23 15:51:41 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(sparkAdmin); users with modify permissions: Set(sparkAdmin)
15/01/23 15:51:41 INFO util.AkkaUtils: Connecting to MapOutputTracker:
akka.tcp://sparkDriver@VM220:53484/user/MapOutputTracker
15/01/23 15:51:41 INFO util.AkkaUtils: Connecting to BlockManagerMaster:
akka.tcp://sparkDriver@VM220:53484/user/BlockManagerMaster
15/01/23 15:51:41 INFO storage.DiskBlockManager: Created local directory at
/tmp/spark-local-20150123155141-b237
15/01/23 15:51:41 INFO storage.MemoryStore: MemoryStore started with
capacity 529.9 MB
15/01/23 15:51:41 INFO netty.NettyBlockTransferService: Server created on
54273
15/01/23 15:51:41 INFO storage.BlockManagerMaster: Trying to register
BlockManager
15/01/23 15:51:41 INFO storage.BlockManagerMaster: Registered BlockManager
15/01/23 15:51:41 INFO util.AkkaUtils: Connecting to HeartbeatReceiver:
akka.tcp://sparkDriver@VM220:53484/user/HeartbeatReceiver
15/01/23 15:51:47 ERROR executor.CoarseGrainedExecutorBackend: Driver
Disassociated [akka.tcp://sparkExecutor@VM219:57695] -
[akka.tcp://sparkDriver@VM220:53484] disassociated! Shutting down.
15/01/23 15:51:47 WARN remote.ReliableDeliverySupervisor: Association with
remote system [akka.tcp://sparkDriver@VM220:53484] has failed, address is
now gated for [5000] ms. Reason is: [Disassociated].
On 24 January 2015 at 06:37, Yana Kadiyska yana.kadiy...@gmail.com wrote:
It looks to me like your executor actually crashed and didn't just finish
properly.
Can you check the executor log?
It is available in the UI, or on the worker machine, under
$SPARK_HOME/work/ app-20150123155114-/6/stderr (unless you manually
changed the work
