Re: Zeppelin on EMR doesn't start?
Indeed, one of the notebooks JSON files was corrupt... not sure how it
happened, but it wasn't an important one so I just deleted it.
Thanks!!
--
Chris Miller
On Mon, Apr 11, 2016 at 9:06 PM, Hyung Sung Shim <hss...@nflabs.com> wrote:
> Hello.
> It seems like one of your notebook json file has problem.
> Could you check your notebook files are valid? and What zeppelin and EMR
> version are you using?
>
> 2016-04-11 20:41 GMT+09:00 Chris Miller <cmiller11...@gmail.com>:
>
>> Hi. I'm running Zeppelin on the latest EMR distribution. Recently the
>> Zeppelin service will not start. I'm not sure what's going on... here is
>> what I see in the logs:
>>
>>
>> WARN [2016-04-11 11:36:29,512] ({main}
>> ZeppelinConfiguration.java[create]:95) - Failed to load configuration,
>> proceeding with a default
>> INFO [2016-04-11 11:36:29,865] ({main} ZeppelinServer.java[main]:101) -
>> Start zeppelin server
>> INFO [2016-04-11 11:36:29,870] ({main} Server.java[doStart]:272) -
>> jetty-8.1.14.v20131031
>> INFO [2016-04-11 11:36:30,073] ({main}
>> InterpreterFactory.java[init]:113) - Reading
>> /usr/lib/zeppelin/interpreter/spark
>> INFO [2016-04-11 11:36:30,139] ({main}
>> InterpreterFactory.java[init]:130) - Interpreter spark.spark found.
>> class=org.apache.zeppelin.spark.SparkInterpreter
>> INFO [2016-04-11 11:36:30,144] ({main}
>> InterpreterFactory.java[init]:130) - Interpreter spark.pyspark found.
>> class=org.apache.zeppelin.spark.PySparkInterpreter
>> INFO [2016-04-11 11:36:30,146] ({main}
>> InterpreterFactory.java[init]:130) - Interpreter spark.sql found.
>> class=org.apache.zeppelin.spark.SparkSqlInterpreter
>> INFO [2016-04-11 11:36:30,149] ({main}
>> InterpreterFactory.java[init]:130) - Interpreter spark.dep found.
>> class=org.apache.zeppelin.spark.DepInterpreter
>> INFO [2016-04-11 11:36:30,171] ({main}
>> InterpreterFactory.java[init]:113) - Reading
>> /usr/lib/zeppelin/interpreter/sh
>> INFO [2016-04-11 11:36:30,177] ({main}
>> InterpreterFactory.java[init]:130) - Interpreter sh.sh found.
>> class=org.apache.zeppelin.shell.ShellInterpreter
>> INFO [2016-04-11 11:36:30,182] ({main}
>> InterpreterFactory.java[init]:113) - Reading
>> /usr/lib/zeppelin/interpreter/lens
>> INFO [2016-04-11 11:36:30,253] ({main}
>> InterpreterFactory.java[init]:130) - Interpreter lens.lens found.
>> class=org.apache.zeppelin.lens.LensInterpreter
>> INFO [2016-04-11 11:36:30,256] ({main}
>> InterpreterFactory.java[init]:113) - Reading
>> /usr/lib/zeppelin/interpreter/psql
>> INFO [2016-04-11 11:36:30,270] ({main}
>> InterpreterFactory.java[init]:130) - Interpreter psql.sql found.
>> class=org.apache.zeppelin.postgresql.PostgreSqlInterpreter
>> INFO [2016-04-11 11:36:30,270] ({main}
>> InterpreterFactory.java[init]:113) - Reading
>> /usr/lib/zeppelin/interpreter/hive
>> INFO [2016-04-11 11:36:30,322] ({main}
>> InterpreterFactory.java[init]:130) - Interpreter hive.hql found.
>> class=org.apache.zeppelin.hive.HiveInterpreter
>> INFO [2016-04-11 11:36:30,329] ({main}
>> InterpreterFactory.java[init]:113) - Reading
>> /usr/lib/zeppelin/interpreter/kylin
>> INFO [2016-04-11 11:36:30,342] ({main}
>> InterpreterFactory.java[init]:130) - Interpreter kylin.kylin found.
>> class=org.apache.zeppelin.kylin.KylinInterpreter
>> INFO [2016-04-11 11:36:30,343] ({main}
>> InterpreterFactory.java[init]:113) - Reading
>> /usr/lib/zeppelin/interpreter/cassandra
>> INFO [2016-04-11 11:36:30,361] ({main}
>> CassandraInterpreter.java[]:154) - Bootstrapping Cassandra
>> Interpreter
>> INFO [2016-04-11 11:36:30,362] ({main}
>> InterpreterFactory.java[init]:130) - Interpreter cassandra.cassandra found.
>> class=org.apache.zeppelin.cassandra.CassandraInterpreter
>> INFO [2016-04-11 11:36:30,363] ({main}
>> InterpreterFactory.java[init]:113) - Reading
>> /usr/lib/zeppelin/interpreter/tajo
>> INFO [2016-04-11 11:36:30,376] ({main}
>> InterpreterFactory.java[init]:130) - Interpreter tajo.tql found.
>> class=org.apache.zeppelin.tajo.TajoInterpreter
>> INFO [2016-04-11 11:36:30,380] ({main}
>> InterpreterFactory.java[init]:113) - Reading
>> /usr/lib/zeppelin/interpreter/md
>> INFO [2016-04-11 11:36:30,385] ({main}
>> InterpreterFactory.java[init]:130) - Interpreter md.md found.
>> class=org.apache.zeppelin.markdown.Markdown
>> INFO [2016-04-11 11:36:30,392] ({main}
>> InterpreterFactory.java[init]:113) - Reading
>> /usr/lib/zeppelin/interpreter/flink
>> INFO [2016-04-11 11:36:30,443] ({main}
&g
Re: Multi-User Zeppelin Deployment?
Thank you for your detailed reply! -- Chris Miller On Thu, Mar 10, 2016 at 1:40 AM, moon soo Lee <m...@apache.org> wrote: > Hi Chris Miller, > > >- If one user is running a job with an interpreter, can another user >simultaneously run a job (such as, in another notebook) with the same >interpreter? > > Short answer is yes, but it depends. > Long answer is, first, it depends on which scheduler interpreter > implementation uses among FIFO and Parallel. Interpreter uses Parallel > scheduler (e.g. spark sql interpreter, shell interpreter, etc) will able to > run simultaneously. Interpreter uses FIFO scheduler will not (e.g. spark > interpreter, etc) > > Recently, http://issues.apache.org/jira/browse/ZEPPELIN-513 is resolved. > Which allows Interpreter use FIFO scheduler run simultaneously, by creating > interpreter instance per notebook. > > >- Does Zeppelin have any kind of user authentication capabilities? > > > Master branch has authentication capabilities based on Apache Shiro. > https://issues.apache.org/jira/browse/ZEPPELIN-548 > > > >- Can I give users either read-only or no access to particular >notebooks but also let users create their own notebooks that only they can >see? > > Recently https://github.com/apache/incubator-zeppelin/pull/681 has been > merged to master branch. It lets users adjust read/write/execute permission > for each notebook. > > > >- Can I have jobs run as the logged-in user rather than a generic >"zeppelin" user so things like HDFS permissions will apply? > > > User impersonation is on the roadmap > https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Roadmap, > but not working in progress, yet. > > Hope this helps. > > Thanks, > moon > > > On Wed, Mar 9, 2016 at 1:08 AM Chris Miller <cmiller11...@gmail.com> > wrote: > >> Hi, >> >> I want to deploy Zeppelin so that multiple users in our organization can >> use it concurrently. I have a few questions: >> >>- If one user is running a job with an interpreter, can another user >>simultaneously run a job (such as, in another notebook) with the same >>interpreter? >>- Does Zeppelin have any kind of user authentication capabilities? >>- Can I give users either read-only or no access to particular >>notebooks but also let users create their own notebooks that only they can >>see? >>- Can I have jobs run as the logged-in user rather than a generic >>"zeppelin" user so things like HDFS permissions will apply? >> >> For those of you using Zeppelin in production, any other deployment or >> configuration tips? >> -- >> Chris Miller >> >
Re: Spark + Zeppelin on EC2
Curious about this too... I'll be moving Zeppelin off to its own box in the near future. If you figure this out, post your resolution here. -- Chris Miller On Sat, Mar 26, 2016 at 12:54 AM, Marcin Pilarczyk < marcin.pilarc...@interia.pl> wrote: > Guys, > > I'm trying to switch my zeppelin instance (0.6 snapshot) from the spark > instance installed on very same machine onto ec2 created spark. Both > versions of spark are 1.5.2. > > I've just created a test instance in EC2, I can submit jobs or use spark > shell. I have revised logs each and every worker up and running, master > alive. So far so good. > > The next step is to switch the zeppelin on the newly created spark. I'm > changing two places: zeppelin-env.sh and URL in the interpreter settings. > I'm SURE these settings are pointing the new instance. > > Next step. I'm stopping the spark instance installed together with > zeppelin. > > Final step, zeppelin is restarted, settings are checked. Somehow any > paragraph that requires computations can't be completed. Master logs are > ok, in the slave log however I can find a following error: > > 16/03/25 12:42:25 INFO Remoting: Starting remoting > 16/03/25 12:42:25 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://driverPropsFetcher@172.31.40.27:36098] > 16/03/25 12:42:25 INFO util.Utils: Successfully started service > 'driverPropsFetcher' on port 36098. > 16/03/25 12:43:28 WARN Remoting: Tried to associate with unreachable > remote address [akka.tcp://sparkDriver@172.31.41.186:46358]. Address is > now gated for 5000 ms, all messages to this address will be delivered to > dead letters.$ > Exception in thread "main" akka.actor.ActorNotFound: Actor not found for: > ActorSelection[Anchor(akka.tcp://sparkDriver@172.31.41.186:46358/), > Path(/user/CoarseGrainedScheduler)] > at > akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65) > at > akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) > at > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) > at > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) > at > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) > at > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) > at > scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) > at > akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) > at > akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74) > at > akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110) > at > akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73) > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) > at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) > at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267) > at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508) > at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541) > at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531) > at > akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87) > at akka.remote.EndpointWriter.postStop(Endpoint.scala:561) > at akka.actor.Actor$class.aroundPostStop(Actor.scala:475) > at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415) > at > akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210) > at > akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172) > at akka.actor.ActorCell.terminate(ActorCell.scala:369) > at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462) > at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) > at > akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) > at akka.dispatch.Mailbox.run(Mailbox.scala:219) > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) > at > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > sc
Re: Interpreter dependency not loading?
Yeah, that isn't very clear. I'll improve it.
--
Chris Miller
On Wed, Mar 9, 2016 at 5:12 PM, mina lee <mina...@apache.org> wrote:
> Glad to hear that it works!
> Actually, there is document
> https://zeppelin.incubator.apache.org/docs/0.5.5-incubating/interpreter/spark.html
> in
> dependency management section, but obviously it seems hard to find for new
> users. So feel free to improve it.
>
>
> On Wed, Mar 9, 2016 at 6:05 PM Chris Miller <cmiller11...@gmail.com>
> wrote:
>
>> Oh, I see. Yeah, that's not documented... no wonder it's confusing. I'll
>> open a PR with some improvements to the documentation for this case when I
>> have a moment.
>>
>> Changing spark-default.conf as you suggested indeed worked. Thanks!
>>
>>
>> --
>> Chris Miller
>>
>> On Wed, Mar 9, 2016 at 10:04 AM, mina lee <mina...@apache.org> wrote:
>>
>>> Hi Chris,
>>>
>>> there are several ways to load dependencies to Zeppelin 0.5.5.
>>> Using %dep is one of them.
>>> If you want do it by setting spark.jars.packages property, proper way of
>>> doing it is editing your SPARK_HOME/conf/spark-default.conf
>>> and adding below line.(I assume that you set SPARK_HOME in
>>> ZEPPELIN_HOME/conf/zeppelin-env.sh)
>>>
>>> spark.jars.packages org.apache.avro:avro:1.8.0,org.
>>> joda:joda-convert:1.8.1
>>>
>>> The reason you can import avro dependency is that spark assembly already
>>> includes avro dependencies, not because you added it in Zeppelin
>>> interpreter setting.
>>>
>>> You can add dependencies via GUI with the latest master
>>> branch(0.6.0-incubating-SNAPSHOT) which is experimental at the moment.
>>> Please let me know it answers your question.
>>>
>>> Regards,
>>> Mina
>>>
>>> On Wed, Mar 9, 2016 at 1:41 AM Chris Miller <cmiller11...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have a strange situation going on. I'm running Zeppelin 0.5.5 and
>>>> Spark 1.6.0 (on Amazon EMR). I added this property to the interpreter
>>>> settings (and restarted it):
>>>>
>>>>
>>>> spark.jars.packages: org.apache.avro:avro:1.8.0,org.joda:joda-convert:1.8.1
>>>>
>>>> The avro dependency loads fine and I'm able to import and use it.
>>>> However, if I try to import something in the joda-convert package (such as,
>>>> org.joda.convert.FromString), I get an error that "error: object convert is
>>>> not a member of package org.joda".
>>>>
>>>> If I run the spark-shell from the CLI and include the same string above
>>>> in the --package parameter, I'm able to import joda-convert just fine.
>>>> Also, if I restart the interpreter and manually import the dependency with
>>>> z.load(), it also works fine:
>>>>
>>>> %dep
>>>> z.load("org.joda:joda-convert:1.8.1")
>>>>
>>>> So, what's going on here?
>>>>
>>>> --
>>>> Chris Miller
>>>>
>>>
>>
Multi-User Zeppelin Deployment?
Hi, I want to deploy Zeppelin so that multiple users in our organization can use it concurrently. I have a few questions: - If one user is running a job with an interpreter, can another user simultaneously run a job (such as, in another notebook) with the same interpreter? - Does Zeppelin have any kind of user authentication capabilities? - Can I give users either read-only or no access to particular notebooks but also let users create their own notebooks that only they can see? - Can I have jobs run as the logged-in user rather than a generic "zeppelin" user so things like HDFS permissions will apply? For those of you using Zeppelin in production, any other deployment or configuration tips? -- Chris Miller
Re: Interpreter dependency not loading?
Oh, I see. Yeah, that's not documented... no wonder it's confusing. I'll
open a PR with some improvements to the documentation for this case when I
have a moment.
Changing spark-default.conf as you suggested indeed worked. Thanks!
--
Chris Miller
On Wed, Mar 9, 2016 at 10:04 AM, mina lee <mina...@apache.org> wrote:
> Hi Chris,
>
> there are several ways to load dependencies to Zeppelin 0.5.5.
> Using %dep is one of them.
> If you want do it by setting spark.jars.packages property, proper way of
> doing it is editing your SPARK_HOME/conf/spark-default.conf
> and adding below line.(I assume that you set SPARK_HOME in
> ZEPPELIN_HOME/conf/zeppelin-env.sh)
>
> spark.jars.packages org.apache.avro:avro:1.8.0,org.
> joda:joda-convert:1.8.1
>
> The reason you can import avro dependency is that spark assembly already
> includes avro dependencies, not because you added it in Zeppelin
> interpreter setting.
>
> You can add dependencies via GUI with the latest master
> branch(0.6.0-incubating-SNAPSHOT) which is experimental at the moment.
> Please let me know it answers your question.
>
> Regards,
> Mina
>
> On Wed, Mar 9, 2016 at 1:41 AM Chris Miller <cmiller11...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I have a strange situation going on. I'm running Zeppelin 0.5.5 and Spark
>> 1.6.0 (on Amazon EMR). I added this property to the interpreter settings
>> (and restarted it):
>>
>>
>> spark.jars.packages: org.apache.avro:avro:1.8.0,org.joda:joda-convert:1.8.1
>>
>> The avro dependency loads fine and I'm able to import and use it.
>> However, if I try to import something in the joda-convert package (such as,
>> org.joda.convert.FromString), I get an error that "error: object convert is
>> not a member of package org.joda".
>>
>> If I run the spark-shell from the CLI and include the same string above
>> in the --package parameter, I'm able to import joda-convert just fine.
>> Also, if I restart the interpreter and manually import the dependency with
>> z.load(), it also works fine:
>>
>> %dep
>> z.load("org.joda:joda-convert:1.8.1")
>>
>> So, what's going on here?
>>
>> --
>> Chris Miller
>>
>
Interpreter dependency not loading?
Hi,
I have a strange situation going on. I'm running Zeppelin 0.5.5 and Spark
1.6.0 (on Amazon EMR). I added this property to the interpreter settings
(and restarted it):
spark.jars.packages: org.apache.avro:avro:1.8.0,org.joda:joda-convert:1.8.1
The avro dependency loads fine and I'm able to import and use it. However,
if I try to import something in the joda-convert package (such as,
org.joda.convert.FromString), I get an error that "error: object convert is
not a member of package org.joda".
If I run the spark-shell from the CLI and include the same string above in
the --package parameter, I'm able to import joda-convert just fine. Also,
if I restart the interpreter and manually import the dependency with
z.load(), it also works fine:
%dep
z.load("org.joda:joda-convert:1.8.1")
So, what's going on here?
--
Chris Miller
Notebook Editor 2-space Indentation?
Hi, Is there a way to configure the notebook text editor to use 2-space indentation when writing Scala (i.e., %spark)? -- Chris Miller
