This should do:
> export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 > export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive > export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop > export HADOOP_CONF_DIR=/etc/hadoop/conf > export HIVE_CONF_DIR=/etc/hive/conf > > mvn clean package -DskipTests -Pspark-2.1 -Dhadoop.version=2.6.0-cdh5.10.1 > -Phadoop-2.6 -Pvendor-repo -Pscala-2.10 -Psparkr -pl > '!alluxio,!flink,!ignite,!lens,!cassandra,!bigquery,!scio' -e You may needs additional steps depending which interpreters you use (like R etc). -- Ruslan Dautkhanov On Fri, Aug 4, 2017 at 8:31 AM, Benjamin Kim <bbuil...@gmail.com> wrote: > Hi Ruslan, > > Can you send me the steps you used to build it, especially the Maven > command with the arguments? I will try to build it also. > > I do believe that the binaries are for official releases. > > Cheers, > Ben > > > On Wed, Aug 2, 2017 at 3:44 PM Ruslan Dautkhanov <dautkha...@gmail.com> > wrote: > >> It was built. I think binaries are only available for official releases? >> >> >> >> -- >> Ruslan Dautkhanov >> >> On Wed, Aug 2, 2017 at 4:41 PM, Benjamin Kim <bbuil...@gmail.com> wrote: >> >>> Did you build Zeppelin or download the binary? >>> >>> On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov <dautkha...@gmail.com> >>> wrote: >>> >>>> We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1. >>>> >>>> Yes, we have that spark home in zeppelin-env.sh >>>> >>>> >>>> >>>> -- >>>> Ruslan Dautkhanov >>>> >>>> On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim <bbuil...@gmail.com> >>>> wrote: >>>> >>>>> Does this work with Zeppelin 0.7.1? We an error when setting >>>>> SPARK_HOME in zeppelin-env.sh to what you have below. >>>>> >>>>> On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov <dautkha...@gmail.com> >>>>> wrote: >>>>> >>>>>> You don't have to use spark2-shell and spark2-submit to use Spark 2. >>>>>> That can be controled by setting SPARK_HOME using regular >>>>>> spark-submit/spark-shell. >>>>>> >>>>>> $ which spark-submit >>>>>> /usr/bin/spark-submit >>>>>> $ which spark-shell >>>>>> /usr/bin/spark-shell >>>>>> >>>>>> $ spark-shell >>>>>> Welcome to >>>>>> ____ __ >>>>>> / __/__ ___ _____/ /__ >>>>>> _\ \/ _ \/ _ `/ __/ '_/ >>>>>> /___/ .__/\_,_/_/ /_/\_\ version 1.6.0 >>>>>> /_/ >>>>>> >>>>>> >>>>>> >>>>>> $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 >>>>>> >>>>>> $ spark-shell >>>>>> Welcome to >>>>>> ____ __ >>>>>> / __/__ ___ _____/ /__ >>>>>> _\ \/ _ \/ _ `/ __/ '_/ >>>>>> /___/ .__/\_,_/_/ /_/\_\ version 2.1.0.cloudera1 >>>>>> /_/ >>>>>> >>>>>> >>>>>> spark-submit and spark-shell are just shell script wrappers. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Ruslan Dautkhanov >>>>>> >>>>>> On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim <bbuil...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> According to the Zeppelin documentation, Zeppelin 0.7.1 supports >>>>>>> Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from >>>>>>> Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does >>>>>>> the >>>>>>> calls to spark-shell and spark-submit. To force the use of Spark 2.x, >>>>>>> the >>>>>>> calls need to be spark2-shell and spark2-submit. I wonder if this is >>>>>>> causing the problem. By the way, we are using Java8 corporate wide, and >>>>>>> there seems to be no problems using Zeppelin. >>>>>>> >>>>>>> Cheers, >>>>>>> Ben >>>>>>> >>>>>>> On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov < >>>>>>> dautkha...@gmail.com> wrote: >>>>>>> >>>>>>>> Might need to recompile Zeppelin with Scala 2.11? >>>>>>>> Also Spark 2.2 now requires JDK8 I believe. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Ruslan Dautkhanov >>>>>>>> >>>>>>>> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim <bbuil...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Here is more. >>>>>>>>> >>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException: WARNING: >>>>>>>>> User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2- >>>>>>>>> 2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides >>>>>>>>> detected (/opt/cloudera/parcels/SPARK2/lib/spark2). >>>>>>>>> WARNING: Running spark-class from user-defined location. >>>>>>>>> Exception in thread "main" java.lang.NoSuchMethodError: >>>>>>>>> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; >>>>>>>>> at org.apache.spark.util.Utils$.getDefaultPropertiesFile( >>>>>>>>> Utils.scala:2103) >>>>>>>>> at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$ >>>>>>>>> mergeDefaultSparkProperties$1.apply(SparkSubmitArguments. >>>>>>>>> scala:124) >>>>>>>>> at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$ >>>>>>>>> mergeDefaultSparkProperties$1.apply(SparkSubmitArguments. >>>>>>>>> scala:124) >>>>>>>>> at scala.Option.getOrElse(Option.scala:120) >>>>>>>>> at org.apache.spark.deploy.SparkSubmitArguments. >>>>>>>>> mergeDefaultSparkProperties(SparkSubmitArguments.scala:124) >>>>>>>>> at org.apache.spark.deploy.SparkSubmitArguments.<init>( >>>>>>>>> SparkSubmitArguments.scala:110) >>>>>>>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit. >>>>>>>>> scala:112) >>>>>>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Ben >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang <zjf...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Then it is due to some classpath issue. I am not sure familiar >>>>>>>>>> with CDH, please check whether spark of CDH include hadoop jar with >>>>>>>>>> it. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Benjamin Kim <bbuil...@gmail.com>于2017年8月2日周三 上午8:22写道： >>>>>>>>>> >>>>>>>>>>> Here is the error that was sent to me. >>>>>>>>>>> >>>>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException: Exception >>>>>>>>>>> in thread "main" java.lang.NoClassDefFoundError: >>>>>>>>>>> org/apache/hadoop/fs/FSDataInputStream >>>>>>>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>>>>>>> org.apache.hadoop.fs.FSDataInputStream >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Ben >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <zjf...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> By default, 0.7.1 doesn't support spark 2.2. But you can set >>>>>>>>>>>> zeppelin.spark.enableSupportedVersionCheck in interpreter >>>>>>>>>>>> setting to disable the supported version check. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Jeff Zhang <zjf...@gmail.com>于2017年8月2日周三 上午8:18写道： >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> What's the error you see in log ? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Benjamin Kim <bbuil...@gmail.com>于2017年8月2日周三 上午8:18写道： >>>>>>>>>>>>> >>>>>>>>>>>>>> Has anyone configured Zeppelin 0.7.1 for Cloudera's release >>>>>>>>>>>>>> of Spark 2.2? I can't get it to work. I downloaded the binary >>>>>>>>>>>>>> and set >>>>>>>>>>>>>> SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I >>>>>>>>>>>>>> must be missing something. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>> Ben >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>> >>>> >>
