You don't have to use spark2-shell and spark2-submit to use Spark 2.
That can be controled by setting SPARK_HOME using regular
spark-submit/spark-shell.
$ which spark-submit
/usr/bin/spark-submit
$ which spark-shell
/usr/bin/spark-shell
$ spark-shell
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.6.0
/_/
$ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2
$ spark-shell
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.1.0.cloudera1
/_/
spark-submit and spark-shell are just shell script wrappers.
--
Ruslan Dautkhanov
On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim <bbuil...@gmail.com> wrote:
> According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark
> 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera.
> For some reason, Cloudera defaults to Spark 1.6 and so does the calls to
> spark-shell and spark-submit. To force the use of Spark 2.x, the calls need
> to be spark2-shell and spark2-submit. I wonder if this is causing the
> problem. By the way, we are using Java8 corporate wide, and there seems to
> be no problems using Zeppelin.
>
> Cheers,
> Ben
>
> On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov <dautkha...@gmail.com>
> wrote:
>
>> Might need to recompile Zeppelin with Scala 2.11?
>> Also Spark 2.2 now requires JDK8 I believe.
>>
>>
>>
>> --
>> Ruslan Dautkhanov
>>
>> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim <bbuil...@gmail.com> wrote:
>>
>>> Here is more.
>>>
>>> org.apache.zeppelin.interpreter.InterpreterException: WARNING:
>>> User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-
>>> 2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected
>>> (/opt/cloudera/parcels/SPARK2/lib/spark2).
>>> WARNING: Running spark-class from user-defined location.
>>> Exception in thread "main" java.lang.NoSuchMethodError:
>>> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
>>> at org.apache.spark.util.Utils$.getDefaultPropertiesFile(
>>> Utils.scala:2103)
>>> at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$
>>> mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
>>> at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$
>>> mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
>>> at scala.Option.getOrElse(Option.scala:120)
>>> at org.apache.spark.deploy.SparkSubmitArguments.
>>> mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
>>> at org.apache.spark.deploy.SparkSubmitArguments.<init>(
>>> SparkSubmitArguments.scala:110)
>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>
>>> Cheers,
>>> Ben
>>>
>>>
>>> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang <zjf...@gmail.com> wrote:
>>>
>>>>
>>>> Then it is due to some classpath issue. I am not sure familiar with
>>>> CDH, please check whether spark of CDH include hadoop jar with it.
>>>>
>>>>
>>>> Benjamin Kim <bbuil...@gmail.com>于2017年8月2日周三 上午8:22写道：
>>>>
>>>>> Here is the error that was sent to me.
>>>>>
>>>>> org.apache.zeppelin.interpreter.InterpreterException: Exception in
>>>>> thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/
>>>>> FSDataInputStream
>>>>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.
>>>>> FSDataInputStream
>>>>>
>>>>> Cheers,
>>>>> Ben
>>>>>
>>>>>
>>>>> On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <zjf...@gmail.com> wrote:
>>>>>
>>>>>>
>>>>>> By default, 0.7.1 doesn't support spark 2.2. But you can set
>>>>>> zeppelin.spark.enableSupportedVersionCheck in interpreter setting to
>>>>>> disable the supported version check.
>>>>>>
>>>>>>
>>>>>> Jeff Zhang <zjf...@gmail.com>于2017年8月2日周三 上午8:18写道：
>>>>>>
>>>>>>>
>>>>>>> What's the error you see in log ?
>>>>>>>
>>>>>>>
>>>>>>> Benjamin Kim <bbuil...@gmail.com>于2017年8月2日周三 上午8:18写道：
>>>>>>>
>>>>>>>> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of
>>>>>>>> Spark 2.2? I can't get it to work. I downloaded the binary and set
>>>>>>>> SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be
>>>>>>>> missing something.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Ben
>>>>>>>>
>>>>>>>
>>
