We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1. Yes, we have that spark home in zeppelin-env.sh
-- Ruslan Dautkhanov On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim <bbuil...@gmail.com> wrote: > Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME in > zeppelin-env.sh to what you have below. > > On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov <dautkha...@gmail.com> > wrote: > >> You don't have to use spark2-shell and spark2-submit to use Spark 2. >> That can be controled by setting SPARK_HOME using regular >> spark-submit/spark-shell. >> >> $ which spark-submit >> /usr/bin/spark-submit >> $ which spark-shell >> /usr/bin/spark-shell >> >> $ spark-shell >> Welcome to >> ____ __ >> / __/__ ___ _____/ /__ >> _\ \/ _ \/ _ `/ __/ '_/ >> /___/ .__/\_,_/_/ /_/\_\ version 1.6.0 >> /_/ >> >> >> >> $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 >> >> $ spark-shell >> Welcome to >> ____ __ >> / __/__ ___ _____/ /__ >> _\ \/ _ \/ _ `/ __/ '_/ >> /___/ .__/\_,_/_/ /_/\_\ version 2.1.0.cloudera1 >> /_/ >> >> >> spark-submit and spark-shell are just shell script wrappers. >> >> >> >> -- >> Ruslan Dautkhanov >> >> On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim <bbuil...@gmail.com> wrote: >> >>> According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark >>> 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. >>> For some reason, Cloudera defaults to Spark 1.6 and so does the calls to >>> spark-shell and spark-submit. To force the use of Spark 2.x, the calls need >>> to be spark2-shell and spark2-submit. I wonder if this is causing the >>> problem. By the way, we are using Java8 corporate wide, and there seems to >>> be no problems using Zeppelin. >>> >>> Cheers, >>> Ben >>> >>> On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov <dautkha...@gmail.com> >>> wrote: >>> >>>> Might need to recompile Zeppelin with Scala 2.11? >>>> Also Spark 2.2 now requires JDK8 I believe. >>>> >>>> >>>> >>>> -- >>>> Ruslan Dautkhanov >>>> >>>> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim <bbuil...@gmail.com> >>>> wrote: >>>> >>>>> Here is more. >>>>> >>>>> org.apache.zeppelin.interpreter.InterpreterException: WARNING: >>>>> User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2- >>>>> 2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected >>>>> (/opt/cloudera/parcels/SPARK2/lib/spark2). >>>>> WARNING: Running spark-class from user-defined location. >>>>> Exception in thread "main" java.lang.NoSuchMethodError: >>>>> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; >>>>> at org.apache.spark.util.Utils$.getDefaultPropertiesFile( >>>>> Utils.scala:2103) >>>>> at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$ >>>>> mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >>>>> at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$ >>>>> mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >>>>> at scala.Option.getOrElse(Option.scala:120) >>>>> at org.apache.spark.deploy.SparkSubmitArguments. >>>>> mergeDefaultSparkProperties(SparkSubmitArguments.scala:124) >>>>> at org.apache.spark.deploy.SparkSubmitArguments.<init>( >>>>> SparkSubmitArguments.scala:110) >>>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) >>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>>>> >>>>> Cheers, >>>>> Ben >>>>> >>>>> >>>>> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang <zjf...@gmail.com> wrote: >>>>> >>>>>> >>>>>> Then it is due to some classpath issue. I am not sure familiar with >>>>>> CDH, please check whether spark of CDH include hadoop jar with it. >>>>>> >>>>>> >>>>>> Benjamin Kim <bbuil...@gmail.com>于2017年8月2日周三 上午8:22写道： >>>>>> >>>>>>> Here is the error that was sent to me. >>>>>>> >>>>>>> org.apache.zeppelin.interpreter.InterpreterException: Exception in >>>>>>> thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/ >>>>>>> FSDataInputStream >>>>>>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs. >>>>>>> FSDataInputStream >>>>>>> >>>>>>> Cheers, >>>>>>> Ben >>>>>>> >>>>>>> >>>>>>> On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <zjf...@gmail.com> wrote: >>>>>>> >>>>>>>> >>>>>>>> By default, 0.7.1 doesn't support spark 2.2. But you can set >>>>>>>> zeppelin.spark.enableSupportedVersionCheck in interpreter setting >>>>>>>> to disable the supported version check. >>>>>>>> >>>>>>>> >>>>>>>> Jeff Zhang <zjf...@gmail.com>于2017年8月2日周三 上午8:18写道： >>>>>>>> >>>>>>>>> >>>>>>>>> What's the error you see in log ? >>>>>>>>> >>>>>>>>> >>>>>>>>> Benjamin Kim <bbuil...@gmail.com>于2017年8月2日周三 上午8:18写道： >>>>>>>>> >>>>>>>>>> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of >>>>>>>>>> Spark 2.2? I can't get it to work. I downloaded the binary and set >>>>>>>>>> SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be >>>>>>>>>> missing something. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Ben >>>>>>>>>> >>>>>>>>> >>>> >>
