Hi,
I am trying to submit a job to a spark cluster running on a single machine
(1 master + 1 worker) with hadoop 1.0.4. I submit it in the code:
«val sparkConf = new
SparkConf().setMaster("spark://myserver:7077").setAppName("MyApp").setJars(Array("target/my-app-1.0-SNAPSHOT.jar"))».
When I run this application on the same machine as the cluster everything
works fine.
But when I run it from a remote machine I get the following error:
Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 14/11/04 00:15:38 INFO CoarseGrainedExecutorBackend: Registered signal
> handlers for [TERM, HUP, INT]
> 14/11/04 00:15:38 INFO SecurityManager: Changing view acls to:
> myuser,Myuser
> 14/11/04 00:15:38 INFO SecurityManager: Changing modify acls to:
> myuser,Myuser
> 14/11/04 00:15:38 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(myuser,
> Myuser); users with modify permissions: Set(myuser, Myuser)
> 14/11/04 00:15:38 INFO Slf4jLogger: Slf4jLogger started
> 14/11/04 00:15:38 INFO Remoting: Starting remoting
> 14/11/04 00:15:38 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://driverPropsFetcher@myserver:49190]
> 14/11/04 00:15:38 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://driverPropsFetcher@myserver:49190]
> 14/11/04 00:15:38 INFO Utils: Successfully started service
> 'driverPropsFetcher' on port 49190.
> 14/11/04 00:15:38 WARN Remoting: Tried to associate with unreachable
> remote address [akka.tcp://sparkDriver@mylaptop:57418]. Address is now
> gated for 60000 ms, all messages to this address will be delivered to dead
> letters.
> 14/11/04 00:16:08 ERROR UserGroupInformation: PriviledgedActionException
> as:Myuser cause:java.util.concurrent.TimeoutException: Futures timed out
> after [30 seconds]
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException:
> Unknown exception in doAs
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1134)
> at
> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:52)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:113)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:156)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
> Caused by: java.security.PrivilegedActionException:
> java.util.concurrent.TimeoutException: Futures timed out after [30 seconds]
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> ... 4 more
> Caused by: java.util.concurrent.TimeoutException: Futures timed out after
> [30 seconds]
> at
> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
> at
> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.result(package.scala:107)
> at
> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:125)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:53)
> at
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:52)
> ... 7 more
>
I know this has something to do with hadoop permissions. I have checked and
all necessary hadoop ports in the server are open and accessible from
outside.
How can I configure the right permissions?
Thanks.
