Re: Cloudera Spark 2.2

[Ruslan Dautkhanov]

This should do:


> export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2
> export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive
> export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop
> export HADOOP_CONF_DIR=/etc/hadoop/conf
> export HIVE_CONF_DIR=/etc/hive/conf



>
> mvn clean package -DskipTests -Pspark-2.1 -Dhadoop.version=2.6.0-cdh5.10.1
> -Phadoop-2.6 -Pvendor-repo -Pscala-2.10 -Psparkr -pl
> '!alluxio,!flink,!ignite,!lens,!cassandra,!bigquery,!scio' -e


You may needs additional steps depending which interpreters you use (like R
etc).


-- 
Ruslan Dautkhanov

On Fri, Aug 4, 2017 at 8:31 AM, Benjamin Kim  wrote:

> Hi Ruslan,
>
> Can you send me the steps you used to build it, especially the Maven
> command with the arguments? I will try to build it also.
>
> I do believe that the binaries are for official releases.
>
> Cheers,
> Ben
>
>
> On Wed, Aug 2, 2017 at 3:44 PM Ruslan Dautkhanov 
> wrote:
>
>> It was built. I think binaries are only available for official releases?
>>
>>
>>
>> --
>> Ruslan Dautkhanov
>>
>> On Wed, Aug 2, 2017 at 4:41 PM, Benjamin Kim  wrote:
>>
>>> Did you build Zeppelin or download the binary?
>>>
>>> On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov 
>>> wrote:
>>>
 We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1.

 Yes, we have that spark home in zeppelin-env.sh



 --
 Ruslan Dautkhanov

 On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim 
 wrote:

> Does this work with Zeppelin 0.7.1? We an error when setting
> SPARK_HOME in zeppelin-env.sh to what you have below.
>
> On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov 
> wrote:
>
>> You don't have to use spark2-shell and spark2-submit to use Spark 2.
>> That can be controled by setting SPARK_HOME using regular
>> spark-submit/spark-shell.
>>
>> $ which spark-submit
>> /usr/bin/spark-submit
>> $ which spark-shell
>> /usr/bin/spark-shell
>>
>> $ spark-shell
>> Welcome to
>>     __
>>  / __/__  ___ _/ /__
>> _\ \/ _ \/ _ `/ __/  '_/
>>/___/ .__/\_,_/_/ /_/\_\   version 1.6.0
>>   /_/
>>
>>
>>
>> $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2
>>
>> $ spark-shell
>> Welcome to
>>     __
>>  / __/__  ___ _/ /__
>> _\ \/ _ \/ _ `/ __/  '_/
>>/___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1
>>   /_/
>>
>>
>> spark-submit and spark-shell are just shell script wrappers.
>>
>>
>>
>> --
>> Ruslan Dautkhanov
>>
>> On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim 
>> wrote:
>>
>>> According to the Zeppelin documentation, Zeppelin 0.7.1 supports
>>> Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from
>>> Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does 
>>> the
>>> calls to spark-shell and spark-submit. To force the use of Spark 2.x, 
>>> the
>>> calls need to be spark2-shell and spark2-submit. I wonder if this is
>>> causing the problem. By the way, we are using Java8 corporate wide, and
>>> there seems to be no problems using Zeppelin.
>>>
>>> Cheers,
>>> Ben
>>>
>>> On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov <
>>> dautkha...@gmail.com> wrote:
>>>
 Might need to recompile Zeppelin with Scala 2.11?
 Also Spark 2.2 now requires JDK8 I believe.



 --
 Ruslan Dautkhanov

 On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim 
 wrote:

> Here is more.
>
> org.apache.zeppelin.interpreter.InterpreterException: WARNING:
> User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-
> 2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides
> detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
> WARNING: Running spark-class from user-defined location.
> Exception in thread "main" java.lang.NoSuchMethodError:
> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
> at org.apache.spark.util.Utils$.getDefaultPropertiesFile(
> Utils.scala:2103)
> at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$
> mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.
> scala:124)
> at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$
> mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.
> scala:124)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.deploy.SparkSubmitArguments.
> mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
> at 

Re: Cloudera Spark 2.2

[Benjamin Kim]

Hi Ruslan,

Can you send me the steps you used to build it, especially the Maven
command with the arguments? I will try to build it also.

I do believe that the binaries are for official releases.

Cheers,
Ben


On Wed, Aug 2, 2017 at 3:44 PM Ruslan Dautkhanov 
wrote:

> It was built. I think binaries are only available for official releases?
>
>
>
> --
> Ruslan Dautkhanov
>
> On Wed, Aug 2, 2017 at 4:41 PM, Benjamin Kim  wrote:
>
>> Did you build Zeppelin or download the binary?
>>
>> On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov 
>> wrote:
>>
>>> We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1.
>>>
>>> Yes, we have that spark home in zeppelin-env.sh
>>>
>>>
>>>
>>> --
>>> Ruslan Dautkhanov
>>>
>>> On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim  wrote:
>>>
 Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME
 in zeppelin-env.sh to what you have below.

 On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov 
 wrote:

> You don't have to use spark2-shell and spark2-submit to use Spark 2.
> That can be controled by setting SPARK_HOME using regular
> spark-submit/spark-shell.
>
> $ which spark-submit
> /usr/bin/spark-submit
> $ which spark-shell
> /usr/bin/spark-shell
>
> $ spark-shell
> Welcome to
>     __
>  / __/__  ___ _/ /__
> _\ \/ _ \/ _ `/ __/  '_/
>/___/ .__/\_,_/_/ /_/\_\   version 1.6.0
>   /_/
>
>
>
> $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2
>
> $ spark-shell
> Welcome to
>     __
>  / __/__  ___ _/ /__
> _\ \/ _ \/ _ `/ __/  '_/
>/___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1
>   /_/
>
>
> spark-submit and spark-shell are just shell script wrappers.
>
>
>
> --
> Ruslan Dautkhanov
>
> On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim 
> wrote:
>
>> According to the Zeppelin documentation, Zeppelin 0.7.1 supports
>> Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from
>> Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does the
>> calls to spark-shell and spark-submit. To force the use of Spark 2.x, the
>> calls need to be spark2-shell and spark2-submit. I wonder if this is
>> causing the problem. By the way, we are using Java8 corporate wide, and
>> there seems to be no problems using Zeppelin.
>>
>> Cheers,
>> Ben
>>
>> On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov <
>> dautkha...@gmail.com> wrote:
>>
>>> Might need to recompile Zeppelin with Scala 2.11?
>>> Also Spark 2.2 now requires JDK8 I believe.
>>>
>>>
>>>
>>> --
>>> Ruslan Dautkhanov
>>>
>>> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim 
>>> wrote:
>>>
 Here is more.

 org.apache.zeppelin.interpreter.InterpreterException: WARNING:
 User-defined SPARK_HOME
 (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2)
 overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
 WARNING: Running spark-class from user-defined location.
 Exception in thread "main" java.lang.NoSuchMethodError:
 scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
 at
 org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
 at
 org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
 at
 org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
 at scala.Option.getOrElse(Option.scala:120)
 at
 org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
 at
 org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110)
 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

 Cheers,
 Ben


 On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang  wrote:

>
> Then it is due to some classpath issue. I am not sure familiar
> with CDH, please check whether spark of CDH include hadoop jar with 
> it.
>
>
> Benjamin Kim 于2017年8月2日周三 上午8:22写道：
>
>> Here is the error that was sent to me.
>>
>> org.apache.zeppelin.interpreter.InterpreterException: Exception
>> in thread "main" java.lang.NoClassDefFoundError:
>> 

Re: Cloudera Spark 2.2

[Ruslan Dautkhanov]

It was built. I think binaries are only available for official releases?



-- 
Ruslan Dautkhanov

On Wed, Aug 2, 2017 at 4:41 PM, Benjamin Kim  wrote:

> Did you build Zeppelin or download the binary?
>
> On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov 
> wrote:
>
>> We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1.
>>
>> Yes, we have that spark home in zeppelin-env.sh
>>
>>
>>
>> --
>> Ruslan Dautkhanov
>>
>> On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim  wrote:
>>
>>> Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME
>>> in zeppelin-env.sh to what you have below.
>>>
>>> On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov 
>>> wrote:
>>>
 You don't have to use spark2-shell and spark2-submit to use Spark 2.
 That can be controled by setting SPARK_HOME using regular
 spark-submit/spark-shell.

 $ which spark-submit
 /usr/bin/spark-submit
 $ which spark-shell
 /usr/bin/spark-shell

 $ spark-shell
 Welcome to
     __
  / __/__  ___ _/ /__
 _\ \/ _ \/ _ `/ __/  '_/
/___/ .__/\_,_/_/ /_/\_\   version 1.6.0
   /_/



 $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2

 $ spark-shell
 Welcome to
     __
  / __/__  ___ _/ /__
 _\ \/ _ \/ _ `/ __/  '_/
/___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1
   /_/


 spark-submit and spark-shell are just shell script wrappers.



 --
 Ruslan Dautkhanov

 On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim 
 wrote:

> According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark
> 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera.
> For some reason, Cloudera defaults to Spark 1.6 and so does the calls to
> spark-shell and spark-submit. To force the use of Spark 2.x, the calls 
> need
> to be spark2-shell and spark2-submit. I wonder if this is causing the
> problem. By the way, we are using Java8 corporate wide, and there seems to
> be no problems using Zeppelin.
>
> Cheers,
> Ben
>
> On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov 
> wrote:
>
>> Might need to recompile Zeppelin with Scala 2.11?
>> Also Spark 2.2 now requires JDK8 I believe.
>>
>>
>>
>> --
>> Ruslan Dautkhanov
>>
>> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim 
>> wrote:
>>
>>> Here is more.
>>>
>>> org.apache.zeppelin.interpreter.InterpreterException: WARNING:
>>> User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-
>>> 2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides
>>> detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
>>> WARNING: Running spark-class from user-defined location.
>>> Exception in thread "main" java.lang.NoSuchMethodError:
>>> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
>>> at org.apache.spark.util.Utils$.getDefaultPropertiesFile(
>>> Utils.scala:2103)
>>> at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$
>>> mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
>>> at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$
>>> mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
>>> at scala.Option.getOrElse(Option.scala:120)
>>> at org.apache.spark.deploy.SparkSubmitArguments.
>>> mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
>>> at org.apache.spark.deploy.SparkSubmitArguments.(
>>> SparkSubmitArguments.scala:110)
>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>
>>> Cheers,
>>> Ben
>>>
>>>
>>> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang  wrote:
>>>

 Then it is due to some classpath issue. I am not sure familiar with
 CDH, please check whether spark of CDH include hadoop jar with it.


 Benjamin Kim 于2017年8月2日周三 上午8:22写道：

> Here is the error that was sent to me.
>
> org.apache.zeppelin.interpreter.InterpreterException: Exception
> in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/fs/FSDataInputStream
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.
> FSDataInputStream
>
> Cheers,
> Ben
>
>
> On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang 
> wrote:
>
>>
>> By default, 0.7.1 doesn't support spark 2.2. But you can set
>> zeppelin.spark.enableSupportedVersionCheck 

Re: Cloudera Spark 2.2

[Benjamin Kim]

Did you build Zeppelin or download the binary?

On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov 
wrote:

> We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1.
>
> Yes, we have that spark home in zeppelin-env.sh
>
>
>
> --
> Ruslan Dautkhanov
>
> On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim  wrote:
>
>> Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME
>> in zeppelin-env.sh to what you have below.
>>
>> On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov 
>> wrote:
>>
>>> You don't have to use spark2-shell and spark2-submit to use Spark 2.
>>> That can be controled by setting SPARK_HOME using regular
>>> spark-submit/spark-shell.
>>>
>>> $ which spark-submit
>>> /usr/bin/spark-submit
>>> $ which spark-shell
>>> /usr/bin/spark-shell
>>>
>>> $ spark-shell
>>> Welcome to
>>>     __
>>>  / __/__  ___ _/ /__
>>> _\ \/ _ \/ _ `/ __/  '_/
>>>/___/ .__/\_,_/_/ /_/\_\   version 1.6.0
>>>   /_/
>>>
>>>
>>>
>>> $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2
>>>
>>> $ spark-shell
>>> Welcome to
>>>     __
>>>  / __/__  ___ _/ /__
>>> _\ \/ _ \/ _ `/ __/  '_/
>>>/___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1
>>>   /_/
>>>
>>>
>>> spark-submit and spark-shell are just shell script wrappers.
>>>
>>>
>>>
>>> --
>>> Ruslan Dautkhanov
>>>
>>> On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim 
>>> wrote:
>>>
 According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark
 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera.
 For some reason, Cloudera defaults to Spark 1.6 and so does the calls to
 spark-shell and spark-submit. To force the use of Spark 2.x, the calls need
 to be spark2-shell and spark2-submit. I wonder if this is causing the
 problem. By the way, we are using Java8 corporate wide, and there seems to
 be no problems using Zeppelin.

 Cheers,
 Ben

 On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov 
 wrote:

> Might need to recompile Zeppelin with Scala 2.11?
> Also Spark 2.2 now requires JDK8 I believe.
>
>
>
> --
> Ruslan Dautkhanov
>
> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim 
> wrote:
>
>> Here is more.
>>
>> org.apache.zeppelin.interpreter.InterpreterException: WARNING:
>> User-defined SPARK_HOME
>> (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2)
>> overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
>> WARNING: Running spark-class from user-defined location.
>> Exception in thread "main" java.lang.NoSuchMethodError:
>> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
>> at
>> org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
>> at
>> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
>> at
>> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
>> at scala.Option.getOrElse(Option.scala:120)
>> at
>> org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
>> at
>> org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>> Cheers,
>> Ben
>>
>>
>> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang  wrote:
>>
>>>
>>> Then it is due to some classpath issue. I am not sure familiar with
>>> CDH, please check whether spark of CDH include hadoop jar with it.
>>>
>>>
>>> Benjamin Kim 于2017年8月2日周三 上午8:22写道：
>>>
 Here is the error that was sent to me.

 org.apache.zeppelin.interpreter.InterpreterException: Exception in
 thread "main" java.lang.NoClassDefFoundError:
 org/apache/hadoop/fs/FSDataInputStream
 Caused by: java.lang.ClassNotFoundException:
 org.apache.hadoop.fs.FSDataInputStream

 Cheers,
 Ben


 On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang  wrote:

>
> By default, 0.7.1 doesn't support spark 2.2. But you can set 
> zeppelin.spark.enableSupportedVersionCheck
> in interpreter setting to disable the supported version check.
>
>
> Jeff Zhang 于2017年8月2日周三 上午8:18写道：
>
>>
>> What's the error you see in log ?
>>
>>
>> Benjamin Kim 于2017年8月2日周三 上午8:18写道：
>>
>>> Has 

Re: Cloudera Spark 2.2

[Benjamin Kim]

Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME in
zeppelin-env.sh to what you have below.

On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov 
wrote:

> You don't have to use spark2-shell and spark2-submit to use Spark 2.
> That can be controled by setting SPARK_HOME using regular
> spark-submit/spark-shell.
>
> $ which spark-submit
> /usr/bin/spark-submit
> $ which spark-shell
> /usr/bin/spark-shell
>
> $ spark-shell
> Welcome to
>     __
>  / __/__  ___ _/ /__
> _\ \/ _ \/ _ `/ __/  '_/
>/___/ .__/\_,_/_/ /_/\_\   version 1.6.0
>   /_/
>
>
>
> $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2
>
> $ spark-shell
> Welcome to
>     __
>  / __/__  ___ _/ /__
> _\ \/ _ \/ _ `/ __/  '_/
>/___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1
>   /_/
>
>
> spark-submit and spark-shell are just shell script wrappers.
>
>
>
> --
> Ruslan Dautkhanov
>
> On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim  wrote:
>
>> According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark
>> 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera.
>> For some reason, Cloudera defaults to Spark 1.6 and so does the calls to
>> spark-shell and spark-submit. To force the use of Spark 2.x, the calls need
>> to be spark2-shell and spark2-submit. I wonder if this is causing the
>> problem. By the way, we are using Java8 corporate wide, and there seems to
>> be no problems using Zeppelin.
>>
>> Cheers,
>> Ben
>>
>> On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov 
>> wrote:
>>
>>> Might need to recompile Zeppelin with Scala 2.11?
>>> Also Spark 2.2 now requires JDK8 I believe.
>>>
>>>
>>>
>>> --
>>> Ruslan Dautkhanov
>>>
>>> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim  wrote:
>>>
 Here is more.

 org.apache.zeppelin.interpreter.InterpreterException: WARNING:
 User-defined SPARK_HOME
 (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2)
 overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
 WARNING: Running spark-class from user-defined location.
 Exception in thread "main" java.lang.NoSuchMethodError:
 scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
 at
 org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
 at
 org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
 at
 org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
 at scala.Option.getOrElse(Option.scala:120)
 at
 org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
 at
 org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110)
 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

 Cheers,
 Ben


 On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang  wrote:

>
> Then it is due to some classpath issue. I am not sure familiar with
> CDH, please check whether spark of CDH include hadoop jar with it.
>
>
> Benjamin Kim 于2017年8月2日周三 上午8:22写道：
>
>> Here is the error that was sent to me.
>>
>> org.apache.zeppelin.interpreter.InterpreterException: Exception in
>> thread "main" java.lang.NoClassDefFoundError:
>> org/apache/hadoop/fs/FSDataInputStream
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.fs.FSDataInputStream
>>
>> Cheers,
>> Ben
>>
>>
>> On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang  wrote:
>>
>>>
>>> By default, 0.7.1 doesn't support spark 2.2. But you can set 
>>> zeppelin.spark.enableSupportedVersionCheck
>>> in interpreter setting to disable the supported version check.
>>>
>>>
>>> Jeff Zhang 于2017年8月2日周三 上午8:18写道：
>>>

 What's the error you see in log ?


 Benjamin Kim 于2017年8月2日周三 上午8:18写道：

> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of
> Spark 2.2? I can't get it to work. I downloaded the binary and set
> SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be 
> missing
> something.
>
> Cheers,
> Ben
>

>>>
>

Re: Cloudera Spark 2.2

[Benjamin Kim]

According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark 2.1.
But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. For
some reason, Cloudera defaults to Spark 1.6 and so does the calls to
spark-shell and spark-submit. To force the use of Spark 2.x, the calls need
to be spark2-shell and spark2-submit. I wonder if this is causing the
problem. By the way, we are using Java8 corporate wide, and there seems to
be no problems using Zeppelin.

Cheers,
Ben

On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov 
wrote:

> Might need to recompile Zeppelin with Scala 2.11?
> Also Spark 2.2 now requires JDK8 I believe.
>
>
>
> --
> Ruslan Dautkhanov
>
> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim  wrote:
>
>> Here is more.
>>
>> org.apache.zeppelin.interpreter.InterpreterException: WARNING:
>> User-defined SPARK_HOME
>> (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2)
>> overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
>> WARNING: Running spark-class from user-defined location.
>> Exception in thread "main" java.lang.NoSuchMethodError:
>> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
>> at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
>> at
>> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
>> at
>> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
>> at scala.Option.getOrElse(Option.scala:120)
>> at
>> org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
>> at
>> org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>> Cheers,
>> Ben
>>
>>
>> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang  wrote:
>>
>>>
>>> Then it is due to some classpath issue. I am not sure familiar with CDH,
>>> please check whether spark of CDH include hadoop jar with it.
>>>
>>>
>>> Benjamin Kim 于2017年8月2日周三 上午8:22写道：
>>>
 Here is the error that was sent to me.

 org.apache.zeppelin.interpreter.InterpreterException: Exception in
 thread "main" java.lang.NoClassDefFoundError:
 org/apache/hadoop/fs/FSDataInputStream
 Caused by: java.lang.ClassNotFoundException:
 org.apache.hadoop.fs.FSDataInputStream

 Cheers,
 Ben


 On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang  wrote:

>
> By default, 0.7.1 doesn't support spark 2.2. But you can set 
> zeppelin.spark.enableSupportedVersionCheck
> in interpreter setting to disable the supported version check.
>
>
> Jeff Zhang 于2017年8月2日周三 上午8:18写道：
>
>>
>> What's the error you see in log ?
>>
>>
>> Benjamin Kim 于2017年8月2日周三 上午8:18写道：
>>
>>> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark
>>> 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME 
>>> to
>>> /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.
>>>
>>> Cheers,
>>> Ben
>>>
>>
>

Re: Cloudera Spark 2.2

[Ruslan Dautkhanov]

Might need to recompile Zeppelin with Scala 2.11?
Also Spark 2.2 now requires JDK8 I believe.



-- 
Ruslan Dautkhanov

On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim  wrote:

> Here is more.
>
> org.apache.zeppelin.interpreter.InterpreterException: WARNING:
> User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-
> 2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected
> (/opt/cloudera/parcels/SPARK2/lib/spark2).
> WARNING: Running spark-class from user-defined location.
> Exception in thread "main" java.lang.NoSuchMethodError:
> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
> at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
> at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$
> mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
> at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$
> mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.deploy.SparkSubmitArguments.
> mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
> at org.apache.spark.deploy.SparkSubmitArguments.(
> SparkSubmitArguments.scala:110)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> Cheers,
> Ben
>
>
> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang  wrote:
>
>>
>> Then it is due to some classpath issue. I am not sure familiar with CDH,
>> please check whether spark of CDH include hadoop jar with it.
>>
>>
>> Benjamin Kim 于2017年8月2日周三 上午8:22写道：
>>
>>> Here is the error that was sent to me.
>>>
>>> org.apache.zeppelin.interpreter.InterpreterException: Exception in
>>> thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/
>>> FSDataInputStream
>>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.
>>> FSDataInputStream
>>>
>>> Cheers,
>>> Ben
>>>
>>>
>>> On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang  wrote:
>>>

 By default, 0.7.1 doesn't support spark 2.2. But you can set
 zeppelin.spark.enableSupportedVersionCheck in interpreter setting to
 disable the supported version check.


 Jeff Zhang 于2017年8月2日周三 上午8:18写道：

>
> What's the error you see in log ?
>
>
> Benjamin Kim 于2017年8月2日周三 上午8:18写道：
>
>> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark
>> 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME 
>> to
>> /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.
>>
>> Cheers,
>> Ben
>>
>

Re: Cloudera Spark 2.2

[Benjamin Kim]

Here is more.

org.apache.zeppelin.interpreter.InterpreterException: WARNING: User-defined
SPARK_HOME
(/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2)
overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
WARNING: Running spark-class from user-defined location.
Exception in thread "main" java.lang.NoSuchMethodError:
scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
at
org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at
org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at scala.Option.getOrElse(Option.scala:120)
at
org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
at
org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang  wrote:

>
> Then it is due to some classpath issue. I am not sure familiar with CDH,
> please check whether spark of CDH include hadoop jar with it.
>
>
> Benjamin Kim 于2017年8月2日周三 上午8:22写道：
>
>> Here is the error that was sent to me.
>>
>> org.apache.zeppelin.interpreter.InterpreterException: Exception in thread
>> "main" java.lang.NoClassDefFoundError:
>> org/apache/hadoop/fs/FSDataInputStream
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.fs.FSDataInputStream
>>
>> Cheers,
>> Ben
>>
>>
>> On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang  wrote:
>>
>>>
>>> By default, 0.7.1 doesn't support spark 2.2. But you can set 
>>> zeppelin.spark.enableSupportedVersionCheck
>>> in interpreter setting to disable the supported version check.
>>>
>>>
>>> Jeff Zhang 于2017年8月2日周三 上午8:18写道：
>>>

 What's the error you see in log ?


 Benjamin Kim 于2017年8月2日周三 上午8:18写道：

> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark
> 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to
> /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.
>
> Cheers,
> Ben
>

Re: Cloudera Spark 2.2

[Benjamin Kim]

Here is the error that was sent to me.

org.apache.zeppelin.interpreter.InterpreterException: Exception in thread
"main" java.lang.NoClassDefFoundError:
org/apache/hadoop/fs/FSDataInputStream
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.fs.FSDataInputStream

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang  wrote:

>
> By default, 0.7.1 doesn't support spark 2.2. But you can set 
> zeppelin.spark.enableSupportedVersionCheck
> in interpreter setting to disable the supported version check.
>
>
> Jeff Zhang 于2017年8月2日周三 上午8:18写道：
>
>>
>> What's the error you see in log ?
>>
>>
>> Benjamin Kim 于2017年8月2日周三 上午8:18写道：
>>
>>> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark
>>> 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to
>>> /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.
>>>
>>> Cheers,
>>> Ben
>>>
>>

Re: Cloudera Spark 2.2

[Jeff Zhang]

What's the error you see in log ?


Benjamin Kim 于2017年8月2日周三 上午8:18写道：

> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2?
> I can't get it to work. I downloaded the binary and set SPARK_HOME to
> /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.
>
> Cheers,
> Ben
>