Hi,
When ever I enable DEBUG level logs for my spark cluster, on running a job
all the executors die with the below exception. On disabling the DEBUG logs
my jobs move to the next step.
I am on spark-1.1.0
Is this a known issue with spark?
Thanks
Ankur
2015-01-29 22:27:42,467 [main] INFO org.apache.spark.SecurityManager -
SecurityManager: authentication disabled; ui acls disabled; users with view
permissions: Set(ubuntu); users with modify permissions: Set(ubuntu)
2015-01-29 22:27:42,478 [main] DEBUG org.apache.spark.util.AkkaUtils - In
createActorSystem, requireCookie is: off
2015-01-29 22:27:42,871
[driverPropsFetcher-akka.actor.default-dispatcher-4] INFO
akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2015-01-29 22:27:42,912
[driverPropsFetcher-akka.actor.default-dispatcher-4] INFO Remoting -
Starting remoting
2015-01-29 22:27:43,057
[driverPropsFetcher-akka.actor.default-dispatcher-4] INFO Remoting -
Remoting started; listening on addresses :[akka.tcp://
driverPropsFetcher@10.77.9.155:36035]
2015-01-29 22:27:43,060
[driverPropsFetcher-akka.actor.default-dispatcher-4] INFO Remoting -
Remoting now listens on addresses: [akka.tcp://
driverPropsFetcher@10.77.9.155:36035]
2015-01-29 22:27:43,067 [main] INFO org.apache.spark.util.Utils -
Successfully started service 'driverPropsFetcher' on port 36035.
2015-01-29 22:28:13,077 [main] ERROR
org.apache.hadoop.security.UserGroupInformation -
PriviledgedActionException as:ubuntu
cause:java.util.concurrent.TimeoutException: Futures timed out after [30
seconds]
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException:
Unknown exception in doAs
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1134)
at
org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:52)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:113)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:156)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: java.security.PrivilegedActionException:
java.util.concurrent.TimeoutException: Futures timed out after [30 seconds]
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
... 4 more
Caused by: java.util.concurrent.TimeoutException: Futures timed out after
[30 seconds]
at
scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at
scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
at
scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:107)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:125)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:53)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:52)
... 7 more
