Re: Spark + Zeppelin on EC2

2016-03-26 Thread Marcin Pilarczyk 
Thanks Moon,

you were right. I've adjusted inbound rules of the machine where zeppelin
is installed to accept all incoming traffic from the spark security group
and now it works fine.

Regards,
Marcin

2016-03-26 5:52 GMT+01:00 moon soo Lee :

> Hi,
>
> Spark workers will open connections to access Spark driver (SparkContext),
> which is running on Zeppelin instance. So make sure your network
> configuration (firewall, routing table, etc) allows workers connect to
> Zeppelin instance.
>
> Could you verify spark shell command, not on the master node, but the same
> node that Zeppelin running? If that works, Zeppelin should works, too.
>
> Thanks,
> moon
>
> On Fri, Mar 25, 2016 at 3:31 PM Chris Miller 
> wrote:
>
>> Curious about this too... I'll be moving Zeppelin off to its own box in
>> the near future. If you figure this out, post your resolution here.
>>
>> --
>> Chris Miller
>>
>> On Sat, Mar 26, 2016 at 12:54 AM, Marcin Pilarczyk <
>> marcin.pilarc...@interia.pl> wrote:
>>
>>> Guys,
>>>
>>> I'm trying to switch my zeppelin instance (0.6 snapshot) from the spark
>>> instance installed on very same machine onto ec2 created spark. Both
>>> versions of spark are 1.5.2.
>>>
>>> I've just created a test instance in EC2, I can submit jobs or use spark
>>> shell. I have revised logs each and every worker up and running, master
>>> alive. So far so good.
>>>
>>> The next step is to switch the zeppelin on the newly created spark. I'm
>>> changing two places: zeppelin-env.sh and URL in the interpreter settings.
>>> I'm SURE these settings are pointing the new instance.
>>>
>>> Next step. I'm stopping the spark instance installed together with
>>> zeppelin.
>>>
>>> Final step, zeppelin is restarted, settings are checked. Somehow any
>>> paragraph that requires computations can't be completed. Master logs are
>>> ok, in the slave log however I can find a following error:
>>>
>>> 16/03/25 12:42:25 INFO Remoting: Starting remoting
>>> 16/03/25 12:42:25 INFO Remoting: Remoting started; listening on
>>> addresses :[akka.tcp://driverPropsFetcher@172.31.40.27:36098]
>>> 16/03/25 12:42:25 INFO util.Utils: Successfully started service
>>> 'driverPropsFetcher' on port 36098.
>>> 16/03/25 12:43:28 WARN Remoting: Tried to associate with unreachable
>>> remote address [akka.tcp://sparkDriver@172.31.41.186:46358]. Address is
>>> now gated for 5000 ms, all messages to this address will be delivered to
>>> dead letters.$
>>> Exception in thread "main" akka.actor.ActorNotFound: Actor not found
>>> for: ActorSelection[Anchor(akka.tcp://sparkDriver@172.31.41.186:46358/),
>>> Path(/user/CoarseGrainedScheduler)]
>>> at
>>> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
>>> at
>>> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
>>> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
>>> at
>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
>>> at
>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
>>> at
>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>>> at
>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>>> at
>>> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
>>> at
>>> akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
>>> at
>>> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
>>> at
>>> akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
>>> at
>>> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
>>> at
>>> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
>>> at
>>> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
>>> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
>>> at
>>> akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
>>> at
>>> akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
>>> at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
>>> at
>>> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)
>>> at akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
>>> at akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
>>> at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
>>> at
>>> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
>>> at
>>> akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
>>> at akka.actor.ActorCell.terminate(ActorCell.scala:369)
>>>

Re: Spark + Zeppelin on EC2

2016-03-25 Thread moon soo Lee 
Hi,

Spark workers will open connections to access Spark driver (SparkContext),
which is running on Zeppelin instance. So make sure your network
configuration (firewall, routing table, etc) allows workers connect to
Zeppelin instance.

Could you verify spark shell command, not on the master node, but the same
node that Zeppelin running? If that works, Zeppelin should works, too.

Thanks,
moon

On Fri, Mar 25, 2016 at 3:31 PM Chris Miller  wrote:

> Curious about this too... I'll be moving Zeppelin off to its own box in
> the near future. If you figure this out, post your resolution here.
>
> --
> Chris Miller
>
> On Sat, Mar 26, 2016 at 12:54 AM, Marcin Pilarczyk <
> marcin.pilarc...@interia.pl> wrote:
>
>> Guys,
>>
>> I'm trying to switch my zeppelin instance (0.6 snapshot) from the spark
>> instance installed on very same machine onto ec2 created spark. Both
>> versions of spark are 1.5.2.
>>
>> I've just created a test instance in EC2, I can submit jobs or use spark
>> shell. I have revised logs each and every worker up and running, master
>> alive. So far so good.
>>
>> The next step is to switch the zeppelin on the newly created spark. I'm
>> changing two places: zeppelin-env.sh and URL in the interpreter settings.
>> I'm SURE these settings are pointing the new instance.
>>
>> Next step. I'm stopping the spark instance installed together with
>> zeppelin.
>>
>> Final step, zeppelin is restarted, settings are checked. Somehow any
>> paragraph that requires computations can't be completed. Master logs are
>> ok, in the slave log however I can find a following error:
>>
>> 16/03/25 12:42:25 INFO Remoting: Starting remoting
>> 16/03/25 12:42:25 INFO Remoting: Remoting started; listening on addresses
>> :[akka.tcp://driverPropsFetcher@172.31.40.27:36098]
>> 16/03/25 12:42:25 INFO util.Utils: Successfully started service
>> 'driverPropsFetcher' on port 36098.
>> 16/03/25 12:43:28 WARN Remoting: Tried to associate with unreachable
>> remote address [akka.tcp://sparkDriver@172.31.41.186:46358]. Address is
>> now gated for 5000 ms, all messages to this address will be delivered to
>> dead letters.$
>> Exception in thread "main" akka.actor.ActorNotFound: Actor not found for:
>> ActorSelection[Anchor(akka.tcp://sparkDriver@172.31.41.186:46358/),
>> Path(/user/CoarseGrainedScheduler)]
>> at
>> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
>> at
>> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
>> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
>> at
>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
>> at
>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
>> at
>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>> at
>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>> at
>> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
>> at
>> akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
>> at
>> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
>> at
>> akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
>> at
>> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
>> at
>> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
>> at
>> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
>> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
>> at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
>> at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
>> at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
>> at
>> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)
>> at akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
>> at akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
>> at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
>> at
>> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
>> at
>> akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
>> at akka.actor.ActorCell.terminate(ActorCell.scala:369)
>> at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
>> at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
>> at
>> akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
>> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
>> at
>>

Re: Spark + Zeppelin on EC2

2016-03-25 Thread Chris Miller 
Curious about this too... I'll be moving Zeppelin off to its own box in the
near future. If you figure this out, post your resolution here.

--
Chris Miller

On Sat, Mar 26, 2016 at 12:54 AM, Marcin Pilarczyk <
marcin.pilarc...@interia.pl> wrote:

> Guys,
>
> I'm trying to switch my zeppelin instance (0.6 snapshot) from the spark
> instance installed on very same machine onto ec2 created spark. Both
> versions of spark are 1.5.2.
>
> I've just created a test instance in EC2, I can submit jobs or use spark
> shell. I have revised logs each and every worker up and running, master
> alive. So far so good.
>
> The next step is to switch the zeppelin on the newly created spark. I'm
> changing two places: zeppelin-env.sh and URL in the interpreter settings.
> I'm SURE these settings are pointing the new instance.
>
> Next step. I'm stopping the spark instance installed together with
> zeppelin.
>
> Final step, zeppelin is restarted, settings are checked. Somehow any
> paragraph that requires computations can't be completed. Master logs are
> ok, in the slave log however I can find a following error:
>
> 16/03/25 12:42:25 INFO Remoting: Starting remoting
> 16/03/25 12:42:25 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://driverPropsFetcher@172.31.40.27:36098]
> 16/03/25 12:42:25 INFO util.Utils: Successfully started service
> 'driverPropsFetcher' on port 36098.
> 16/03/25 12:43:28 WARN Remoting: Tried to associate with unreachable
> remote address [akka.tcp://sparkDriver@172.31.41.186:46358]. Address is
> now gated for 5000 ms, all messages to this address will be delivered to
> dead letters.$
> Exception in thread "main" akka.actor.ActorNotFound: Actor not found for:
> ActorSelection[Anchor(akka.tcp://sparkDriver@172.31.41.186:46358/),
> Path(/user/CoarseGrainedScheduler)]
> at
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
> at
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> at
> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
> at
> akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
> at
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
> at
> akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
> at
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
> at
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
> at
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
> at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
> at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
> at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
> at
> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)
> at akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
> at akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
> at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
> at
> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
> at
> akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
> at akka.actor.ActorCell.terminate(ActorCell.scala:369)
> at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
> at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
> at
> akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
> at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> 172.31.41.186 -> that's the address where zeppelin is running and previous
> spark WAS running, In the zeppelin configuration there is no trace of this
> IP. Please note again, spark shell and submit