Guys,
I'm trying to switch my zeppelin instance (0.6 snapshot) from the spark
instance installed on very same machine onto ec2 created spark. Both
versions of spark are 1.5.2.
I've just created a test instance in EC2, I can submit jobs or use spark
shell. I have revised logs each and every worker up and running, master
alive. So far so good.
The next step is to switch the zeppelin on the newly created spark. I'm
changing two places: zeppelin-env.sh and URL in the interpreter settings.
I'm SURE these settings are pointing the new instance.
Next step. I'm stopping the spark instance installed together with zeppelin.
Final step, zeppelin is restarted, settings are checked. Somehow any
paragraph that requires computations can't be completed. Master logs are
ok, in the slave log however I can find a following error:
16/03/25 12:42:25 INFO Remoting: Starting remoting
16/03/25 12:42:25 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://driverPropsFetcher@172.31.40.27:36098]
16/03/25 12:42:25 INFO util.Utils: Successfully started service
'driverPropsFetcher' on port 36098.
16/03/25 12:43:28 WARN Remoting: Tried to associate with unreachable remote
address [akka.tcp://sparkDriver@172.31.41.186:46358]. Address is now gated
for 5000 ms, all messages to this address will be delivered to dead
letters.$
Exception in thread "main" akka.actor.ActorNotFound: Actor not found for:
ActorSelection[Anchor(akka.tcp://sparkDriver@172.31.41.186:46358/),
Path(/user/CoarseGrainedScheduler)]
at
akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
at
akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
at
scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at
akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
at
akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
at
akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
at
akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
at
scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
at
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
at
akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)
at akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
at akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
at
akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
at
akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
at akka.actor.ActorCell.terminate(ActorCell.scala:369)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
172.31.41.186 -> that's the address where zeppelin is running and previous
spark WAS running, In the zeppelin configuration there is no trace of this
IP. Please note again, spark shell and submit on the new master node are
executing their jobs.
Is there yet another place I need to change? Or generally, is there any
problem with running zeppelin with remote/external spark?
Regards,
Marcin
