Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME in zeppelin-env.sh to what you have below.
On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov <dautkha...@gmail.com> wrote: > You don't have to use spark2-shell and spark2-submit to use Spark 2. > That can be controled by setting SPARK_HOME using regular > spark-submit/spark-shell. > > $ which spark-submit > /usr/bin/spark-submit > $ which spark-shell > /usr/bin/spark-shell > > $ spark-shell > Welcome to > ____ __ > / __/__ ___ _____/ /__ > _\ \/ _ \/ _ `/ __/ '_/ > /___/ .__/\_,_/_/ /_/\_\ version 1.6.0 > /_/ > > > > $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 > > $ spark-shell > Welcome to > ____ __ > / __/__ ___ _____/ /__ > _\ \/ _ \/ _ `/ __/ '_/ > /___/ .__/\_,_/_/ /_/\_\ version 2.1.0.cloudera1 > /_/ > > > spark-submit and spark-shell are just shell script wrappers. > > > > -- > Ruslan Dautkhanov > > On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim <bbuil...@gmail.com> wrote: > >> According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark >> 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. >> For some reason, Cloudera defaults to Spark 1.6 and so does the calls to >> spark-shell and spark-submit. To force the use of Spark 2.x, the calls need >> to be spark2-shell and spark2-submit. I wonder if this is causing the >> problem. By the way, we are using Java8 corporate wide, and there seems to >> be no problems using Zeppelin. >> >> Cheers, >> Ben >> >> On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov <dautkha...@gmail.com> >> wrote: >> >>> Might need to recompile Zeppelin with Scala 2.11? >>> Also Spark 2.2 now requires JDK8 I believe. >>> >>> >>> >>> -- >>> Ruslan Dautkhanov >>> >>> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim <bbuil...@gmail.com> wrote: >>> >>>> Here is more. >>>> >>>> org.apache.zeppelin.interpreter.InterpreterException: WARNING: >>>> User-defined SPARK_HOME >>>> (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) >>>> overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2). >>>> WARNING: Running spark-class from user-defined location. >>>> Exception in thread "main" java.lang.NoSuchMethodError: >>>> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; >>>> at >>>> org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103) >>>> at >>>> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >>>> at >>>> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >>>> at scala.Option.getOrElse(Option.scala:120) >>>> at >>>> org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124) >>>> at >>>> org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:110) >>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) >>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>>> >>>> Cheers, >>>> Ben >>>> >>>> >>>> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang <zjf...@gmail.com> wrote: >>>> >>>>> >>>>> Then it is due to some classpath issue. I am not sure familiar with >>>>> CDH, please check whether spark of CDH include hadoop jar with it. >>>>> >>>>> >>>>> Benjamin Kim <bbuil...@gmail.com>于2017年8月2日周三 上午8:22写道： >>>>> >>>>>> Here is the error that was sent to me. >>>>>> >>>>>> org.apache.zeppelin.interpreter.InterpreterException: Exception in >>>>>> thread "main" java.lang.NoClassDefFoundError: >>>>>> org/apache/hadoop/fs/FSDataInputStream >>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>> org.apache.hadoop.fs.FSDataInputStream >>>>>> >>>>>> Cheers, >>>>>> Ben >>>>>> >>>>>> >>>>>> On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <zjf...@gmail.com> wrote: >>>>>> >>>>>>> >>>>>>> By default, 0.7.1 doesn't support spark 2.2. But you can set >>>>>>> zeppelin.spark.enableSupportedVersionCheck >>>>>>> in interpreter setting to disable the supported version check. >>>>>>> >>>>>>> >>>>>>> Jeff Zhang <zjf...@gmail.com>于2017年8月2日周三 上午8:18写道： >>>>>>> >>>>>>>> >>>>>>>> What's the error you see in log ? >>>>>>>> >>>>>>>> >>>>>>>> Benjamin Kim <bbuil...@gmail.com>于2017年8月2日周三 上午8:18写道： >>>>>>>> >>>>>>>>> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of >>>>>>>>> Spark 2.2? I can't get it to work. I downloaded the binary and set >>>>>>>>> SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be >>>>>>>>> missing >>>>>>>>> something. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Ben >>>>>>>>> >>>>>>>> >>> >
