Re: configure Zeppelin to use existing external hive metastore

[Jeff Zhang]

I suppose the %sql means %spark.sql, in that case you need to modify the
hive-site.xml under SPARK_CONF_DIR


Richard Xin 于2017年8月3日周四 上午9:21写道：

> on AWS EMR
> I am trying to bind zeppelin to an existing hive metastore, I modified
> hive-site.xml under /etc/hive/conf to pointing to existing metastore, and I
> tested with commandline hive, it works as expected.
> but under zeppelin %sql is still connected to previous hive metastore,
> I added line
> export HIVE_CONF_DIR=/etc/hive/conf
> inside zeppelin-env.sh
>
>
> And /etc/spark/conf's spark-env.sh has following line by default:
> export HIVE_CONF_DIR=${HIVE_CONF_DIR:-/etc/hive/conf}
>
> What did I miss?
>
> Thanks
>

configure Zeppelin to use existing external hive metastore

[Richard Xin]

on AWS EMRI am trying to bind zeppelin to an existing hive metastore, I 
modified hive-site.xml under /etc/hive/conf to pointing to existing metastore, 
and I tested with commandline hive, it works as expected.but under zeppelin 
%sql is still connected to previous hive metastore, I added line export 
HIVE_CONF_DIR=/etc/hive/conf
inside zeppelin-env.sh

And /etc/spark/conf's spark-env.sh has following line by default:export 
HIVE_CONF_DIR=${HIVE_CONF_DIR:-/etc/hive/conf}

What did I miss?

Thanks

Re: Geo Map Charting

[Jeff Zhang]

Zeppelin support bokeh which support Geo charting.  Here's some links which
might be useful for you
https://community.hortonworks.com/articles/109837/use-bokeh-in-apache-zeppelin.html
http://bokeh.pydata.org/en/latest/docs/user_guide/geo.html



Benjamin Kim 于2017年8月3日周四 上午6:23写道：

> Anyone every try to chart density clusters or heat maps onto a geo map of
> the earth in Zeppelin? Can it be done?
>
> Cheers,
> Ben
>

Re: Cloudera Spark 2.2

[Ruslan Dautkhanov]

It was built. I think binaries are only available for official releases?



-- 
Ruslan Dautkhanov

On Wed, Aug 2, 2017 at 4:41 PM, Benjamin Kim  wrote:

> Did you build Zeppelin or download the binary?
>
> On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov 
> wrote:
>
>> We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1.
>>
>> Yes, we have that spark home in zeppelin-env.sh
>>
>>
>>
>> --
>> Ruslan Dautkhanov
>>
>> On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim  wrote:
>>
>>> Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME
>>> in zeppelin-env.sh to what you have below.
>>>
>>> On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov 
>>> wrote:
>>>
 You don't have to use spark2-shell and spark2-submit to use Spark 2.
 That can be controled by setting SPARK_HOME using regular
 spark-submit/spark-shell.

 $ which spark-submit
 /usr/bin/spark-submit
 $ which spark-shell
 /usr/bin/spark-shell

 $ spark-shell
 Welcome to
     __
  / __/__  ___ _/ /__
 _\ \/ _ \/ _ `/ __/  '_/
/___/ .__/\_,_/_/ /_/\_\   version 1.6.0
   /_/



 $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2

 $ spark-shell
 Welcome to
     __
  / __/__  ___ _/ /__
 _\ \/ _ \/ _ `/ __/  '_/
/___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1
   /_/


 spark-submit and spark-shell are just shell script wrappers.



 --
 Ruslan Dautkhanov

 On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim 
 wrote:

> According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark
> 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera.
> For some reason, Cloudera defaults to Spark 1.6 and so does the calls to
> spark-shell and spark-submit. To force the use of Spark 2.x, the calls 
> need
> to be spark2-shell and spark2-submit. I wonder if this is causing the
> problem. By the way, we are using Java8 corporate wide, and there seems to
> be no problems using Zeppelin.
>
> Cheers,
> Ben
>
> On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov 
> wrote:
>
>> Might need to recompile Zeppelin with Scala 2.11?
>> Also Spark 2.2 now requires JDK8 I believe.
>>
>>
>>
>> --
>> Ruslan Dautkhanov
>>
>> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim 
>> wrote:
>>
>>> Here is more.
>>>
>>> org.apache.zeppelin.interpreter.InterpreterException: WARNING:
>>> User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-
>>> 2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides
>>> detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
>>> WARNING: Running spark-class from user-defined location.
>>> Exception in thread "main" java.lang.NoSuchMethodError:
>>> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
>>> at org.apache.spark.util.Utils$.getDefaultPropertiesFile(
>>> Utils.scala:2103)
>>> at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$
>>> mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
>>> at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$
>>> mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
>>> at scala.Option.getOrElse(Option.scala:120)
>>> at org.apache.spark.deploy.SparkSubmitArguments.
>>> mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
>>> at org.apache.spark.deploy.SparkSubmitArguments.(
>>> SparkSubmitArguments.scala:110)
>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>
>>> Cheers,
>>> Ben
>>>
>>>
>>> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang  wrote:
>>>

 Then it is due to some classpath issue. I am not sure familiar with
 CDH, please check whether spark of CDH include hadoop jar with it.


 Benjamin Kim 于2017年8月2日周三 上午8:22写道：

> Here is the error that was sent to me.
>
> org.apache.zeppelin.interpreter.InterpreterException: Exception
> in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/fs/FSDataInputStream
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.
> FSDataInputStream
>
> Cheers,
> Ben
>
>
> On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang 
> wrote:
>
>>
>> By default, 0.7.1 doesn't support spark 2.2. But you can set
>> zeppelin.spark.enableSupportedVersionCheck 

Re: Cloudera Spark 2.2

[Benjamin Kim]

Did you build Zeppelin or download the binary?

On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov 
wrote:

> We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1.
>
> Yes, we have that spark home in zeppelin-env.sh
>
>
>
> --
> Ruslan Dautkhanov
>
> On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim  wrote:
>
>> Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME
>> in zeppelin-env.sh to what you have below.
>>
>> On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov 
>> wrote:
>>
>>> You don't have to use spark2-shell and spark2-submit to use Spark 2.
>>> That can be controled by setting SPARK_HOME using regular
>>> spark-submit/spark-shell.
>>>
>>> $ which spark-submit
>>> /usr/bin/spark-submit
>>> $ which spark-shell
>>> /usr/bin/spark-shell
>>>
>>> $ spark-shell
>>> Welcome to
>>>     __
>>>  / __/__  ___ _/ /__
>>> _\ \/ _ \/ _ `/ __/  '_/
>>>/___/ .__/\_,_/_/ /_/\_\   version 1.6.0
>>>   /_/
>>>
>>>
>>>
>>> $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2
>>>
>>> $ spark-shell
>>> Welcome to
>>>     __
>>>  / __/__  ___ _/ /__
>>> _\ \/ _ \/ _ `/ __/  '_/
>>>/___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1
>>>   /_/
>>>
>>>
>>> spark-submit and spark-shell are just shell script wrappers.
>>>
>>>
>>>
>>> --
>>> Ruslan Dautkhanov
>>>
>>> On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim 
>>> wrote:
>>>
 According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark
 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera.
 For some reason, Cloudera defaults to Spark 1.6 and so does the calls to
 spark-shell and spark-submit. To force the use of Spark 2.x, the calls need
 to be spark2-shell and spark2-submit. I wonder if this is causing the
 problem. By the way, we are using Java8 corporate wide, and there seems to
 be no problems using Zeppelin.

 Cheers,
 Ben

 On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov 
 wrote:

> Might need to recompile Zeppelin with Scala 2.11?
> Also Spark 2.2 now requires JDK8 I believe.
>
>
>
> --
> Ruslan Dautkhanov
>
> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim 
> wrote:
>
>> Here is more.
>>
>> org.apache.zeppelin.interpreter.InterpreterException: WARNING:
>> User-defined SPARK_HOME
>> (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2)
>> overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
>> WARNING: Running spark-class from user-defined location.
>> Exception in thread "main" java.lang.NoSuchMethodError:
>> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
>> at
>> org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
>> at
>> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
>> at
>> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
>> at scala.Option.getOrElse(Option.scala:120)
>> at
>> org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
>> at
>> org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>> Cheers,
>> Ben
>>
>>
>> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang  wrote:
>>
>>>
>>> Then it is due to some classpath issue. I am not sure familiar with
>>> CDH, please check whether spark of CDH include hadoop jar with it.
>>>
>>>
>>> Benjamin Kim 于2017年8月2日周三 上午8:22写道：
>>>
 Here is the error that was sent to me.

 org.apache.zeppelin.interpreter.InterpreterException: Exception in
 thread "main" java.lang.NoClassDefFoundError:
 org/apache/hadoop/fs/FSDataInputStream
 Caused by: java.lang.ClassNotFoundException:
 org.apache.hadoop.fs.FSDataInputStream

 Cheers,
 Ben


 On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang  wrote:

>
> By default, 0.7.1 doesn't support spark 2.2. But you can set 
> zeppelin.spark.enableSupportedVersionCheck
> in interpreter setting to disable the supported version check.
>
>
> Jeff Zhang 于2017年8月2日周三 上午8:18写道：
>
>>
>> What's the error you see in log ?
>>
>>
>> Benjamin Kim 于2017年8月2日周三 上午8:18写道：
>>
>>> Has 

Re: Cloudera Spark 2.2

[Benjamin Kim]

Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME in
zeppelin-env.sh to what you have below.

On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov 
wrote:

> You don't have to use spark2-shell and spark2-submit to use Spark 2.
> That can be controled by setting SPARK_HOME using regular
> spark-submit/spark-shell.
>
> $ which spark-submit
> /usr/bin/spark-submit
> $ which spark-shell
> /usr/bin/spark-shell
>
> $ spark-shell
> Welcome to
>     __
>  / __/__  ___ _/ /__
> _\ \/ _ \/ _ `/ __/  '_/
>/___/ .__/\_,_/_/ /_/\_\   version 1.6.0
>   /_/
>
>
>
> $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2
>
> $ spark-shell
> Welcome to
>     __
>  / __/__  ___ _/ /__
> _\ \/ _ \/ _ `/ __/  '_/
>/___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1
>   /_/
>
>
> spark-submit and spark-shell are just shell script wrappers.
>
>
>
> --
> Ruslan Dautkhanov
>
> On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim  wrote:
>
>> According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark
>> 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera.
>> For some reason, Cloudera defaults to Spark 1.6 and so does the calls to
>> spark-shell and spark-submit. To force the use of Spark 2.x, the calls need
>> to be spark2-shell and spark2-submit. I wonder if this is causing the
>> problem. By the way, we are using Java8 corporate wide, and there seems to
>> be no problems using Zeppelin.
>>
>> Cheers,
>> Ben
>>
>> On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov 
>> wrote:
>>
>>> Might need to recompile Zeppelin with Scala 2.11?
>>> Also Spark 2.2 now requires JDK8 I believe.
>>>
>>>
>>>
>>> --
>>> Ruslan Dautkhanov
>>>
>>> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim  wrote:
>>>
 Here is more.

 org.apache.zeppelin.interpreter.InterpreterException: WARNING:
 User-defined SPARK_HOME
 (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2)
 overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
 WARNING: Running spark-class from user-defined location.
 Exception in thread "main" java.lang.NoSuchMethodError:
 scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
 at
 org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
 at
 org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
 at
 org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
 at scala.Option.getOrElse(Option.scala:120)
 at
 org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
 at
 org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110)
 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

 Cheers,
 Ben


 On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang  wrote:

>
> Then it is due to some classpath issue. I am not sure familiar with
> CDH, please check whether spark of CDH include hadoop jar with it.
>
>
> Benjamin Kim 于2017年8月2日周三 上午8:22写道：
>
>> Here is the error that was sent to me.
>>
>> org.apache.zeppelin.interpreter.InterpreterException: Exception in
>> thread "main" java.lang.NoClassDefFoundError:
>> org/apache/hadoop/fs/FSDataInputStream
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.fs.FSDataInputStream
>>
>> Cheers,
>> Ben
>>
>>
>> On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang  wrote:
>>
>>>
>>> By default, 0.7.1 doesn't support spark 2.2. But you can set 
>>> zeppelin.spark.enableSupportedVersionCheck
>>> in interpreter setting to disable the supported version check.
>>>
>>>
>>> Jeff Zhang 于2017年8月2日周三 上午8:18写道：
>>>

 What's the error you see in log ?


 Benjamin Kim 于2017年8月2日周三 上午8:18写道：

> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of
> Spark 2.2? I can't get it to work. I downloaded the binary and set
> SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be 
> missing
> something.
>
> Cheers,
> Ben
>

>>>
>

Geo Map Charting

[Benjamin Kim]

Anyone every try to chart density clusters or heat maps onto a geo map of
the earth in Zeppelin? Can it be done?

Cheers,
Ben

unsubscribe

[Brian Liao]

Re: Cloudera Spark 2.2

[Benjamin Kim]

According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark 2.1.
But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. For
some reason, Cloudera defaults to Spark 1.6 and so does the calls to
spark-shell and spark-submit. To force the use of Spark 2.x, the calls need
to be spark2-shell and spark2-submit. I wonder if this is causing the
problem. By the way, we are using Java8 corporate wide, and there seems to
be no problems using Zeppelin.

Cheers,
Ben

On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov 
wrote:

> Might need to recompile Zeppelin with Scala 2.11?
> Also Spark 2.2 now requires JDK8 I believe.
>
>
>
> --
> Ruslan Dautkhanov
>
> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim  wrote:
>
>> Here is more.
>>
>> org.apache.zeppelin.interpreter.InterpreterException: WARNING:
>> User-defined SPARK_HOME
>> (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2)
>> overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
>> WARNING: Running spark-class from user-defined location.
>> Exception in thread "main" java.lang.NoSuchMethodError:
>> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
>> at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
>> at
>> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
>> at
>> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
>> at scala.Option.getOrElse(Option.scala:120)
>> at
>> org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
>> at
>> org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>> Cheers,
>> Ben
>>
>>
>> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang  wrote:
>>
>>>
>>> Then it is due to some classpath issue. I am not sure familiar with CDH,
>>> please check whether spark of CDH include hadoop jar with it.
>>>
>>>
>>> Benjamin Kim 于2017年8月2日周三 上午8:22写道：
>>>
 Here is the error that was sent to me.

 org.apache.zeppelin.interpreter.InterpreterException: Exception in
 thread "main" java.lang.NoClassDefFoundError:
 org/apache/hadoop/fs/FSDataInputStream
 Caused by: java.lang.ClassNotFoundException:
 org.apache.hadoop.fs.FSDataInputStream

 Cheers,
 Ben


 On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang  wrote:

>
> By default, 0.7.1 doesn't support spark 2.2. But you can set 
> zeppelin.spark.enableSupportedVersionCheck
> in interpreter setting to disable the supported version check.
>
>
> Jeff Zhang 于2017年8月2日周三 上午8:18写道：
>
>>
>> What's the error you see in log ?
>>
>>
>> Benjamin Kim 于2017年8月2日周三 上午8:18写道：
>>
>>> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark
>>> 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME 
>>> to
>>> /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.
>>>
>>> Cheers,
>>> Ben
>>>
>>
>