Ruslan, Quick observation. I see that you put Scala 2.10. How come you didn't use Scala 2.11?
Just curious. Cheers, Ben On Fri, Aug 4, 2017 at 8:36 AM Ruslan Dautkhanov <dautkha...@gmail.com> wrote: > This should do: > > >> export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 >> export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive >> export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop >> export HADOOP_CONF_DIR=/etc/hadoop/conf >> export HIVE_CONF_DIR=/etc/hive/conf > > > >> >> mvn clean package -DskipTests -Pspark-2.1 >> -Dhadoop.version=2.6.0-cdh5.10.1 -Phadoop-2.6 -Pvendor-repo -Pscala-2.10 >> -Psparkr -pl '!alluxio,!flink,!ignite,!lens,!cassandra,!bigquery,!scio' -e > > > You may needs additional steps depending which interpreters you use (like > R etc). > > > -- > Ruslan Dautkhanov > > On Fri, Aug 4, 2017 at 8:31 AM, Benjamin Kim <bbuil...@gmail.com> wrote: > >> Hi Ruslan, >> >> Can you send me the steps you used to build it, especially the Maven >> command with the arguments? I will try to build it also. >> >> I do believe that the binaries are for official releases. >> >> Cheers, >> Ben >> >> >> On Wed, Aug 2, 2017 at 3:44 PM Ruslan Dautkhanov <dautkha...@gmail.com> >> wrote: >> >>> It was built. I think binaries are only available for official releases? >>> >>> >>> >>> -- >>> Ruslan Dautkhanov >>> >>> On Wed, Aug 2, 2017 at 4:41 PM, Benjamin Kim <bbuil...@gmail.com> wrote: >>> >>>> Did you build Zeppelin or download the binary? >>>> >>>> On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov <dautkha...@gmail.com> >>>> wrote: >>>> >>>>> We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1. >>>>> >>>>> Yes, we have that spark home in zeppelin-env.sh >>>>> >>>>> >>>>> >>>>> -- >>>>> Ruslan Dautkhanov >>>>> >>>>> On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim <bbuil...@gmail.com> >>>>> wrote: >>>>> >>>>>> Does this work with Zeppelin 0.7.1? We an error when setting >>>>>> SPARK_HOME in zeppelin-env.sh to what you have below. >>>>>> >>>>>> On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov < >>>>>> dautkha...@gmail.com> wrote: >>>>>> >>>>>>> You don't have to use spark2-shell and spark2-submit to use Spark 2. >>>>>>> That can be controled by setting SPARK_HOME using regular >>>>>>> spark-submit/spark-shell. >>>>>>> >>>>>>> $ which spark-submit >>>>>>> /usr/bin/spark-submit >>>>>>> $ which spark-shell >>>>>>> /usr/bin/spark-shell >>>>>>> >>>>>>> $ spark-shell >>>>>>> Welcome to >>>>>>> ____ __ >>>>>>> / __/__ ___ _____/ /__ >>>>>>> _\ \/ _ \/ _ `/ __/ '_/ >>>>>>> /___/ .__/\_,_/_/ /_/\_\ version 1.6.0 >>>>>>> /_/ >>>>>>> >>>>>>> >>>>>>> >>>>>>> $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 >>>>>>> >>>>>>> $ spark-shell >>>>>>> Welcome to >>>>>>> ____ __ >>>>>>> / __/__ ___ _____/ /__ >>>>>>> _\ \/ _ \/ _ `/ __/ '_/ >>>>>>> /___/ .__/\_,_/_/ /_/\_\ version 2.1.0.cloudera1 >>>>>>> /_/ >>>>>>> >>>>>>> >>>>>>> spark-submit and spark-shell are just shell script wrappers. >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Ruslan Dautkhanov >>>>>>> >>>>>>> On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim <bbuil...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> According to the Zeppelin documentation, Zeppelin 0.7.1 supports >>>>>>>> Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from >>>>>>>> Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does >>>>>>>> the >>>>>>>> calls to spark-shell and spark-submit. To force the use of Spark 2.x, >>>>>>>> the >>>>>>>> calls need to be spark2-shell and spark2-submit. I wonder if this is >>>>>>>> causing the problem. By the way, we are using Java8 corporate wide, and >>>>>>>> there seems to be no problems using Zeppelin. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Ben >>>>>>>> >>>>>>>> On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov < >>>>>>>> dautkha...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Might need to recompile Zeppelin with Scala 2.11? >>>>>>>>> Also Spark 2.2 now requires JDK8 I believe. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Ruslan Dautkhanov >>>>>>>>> >>>>>>>>> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim <bbuil...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Here is more. >>>>>>>>>> >>>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException: WARNING: >>>>>>>>>> User-defined SPARK_HOME >>>>>>>>>> (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) >>>>>>>>>> overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2). >>>>>>>>>> WARNING: Running spark-class from user-defined location. >>>>>>>>>> Exception in thread "main" java.lang.NoSuchMethodError: >>>>>>>>>> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; >>>>>>>>>> at >>>>>>>>>> org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103) >>>>>>>>>> at >>>>>>>>>> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >>>>>>>>>> at >>>>>>>>>> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >>>>>>>>>> at scala.Option.getOrElse(Option.scala:120) >>>>>>>>>> at >>>>>>>>>> org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124) >>>>>>>>>> at >>>>>>>>>> org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:110) >>>>>>>>>> at >>>>>>>>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) >>>>>>>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Ben >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang <zjf...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Then it is due to some classpath issue. I am not sure familiar >>>>>>>>>>> with CDH, please check whether spark of CDH include hadoop jar with >>>>>>>>>>> it. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Benjamin Kim <bbuil...@gmail.com>于2017年8月2日周三 上午8:22写道： >>>>>>>>>>> >>>>>>>>>>>> Here is the error that was sent to me. >>>>>>>>>>>> >>>>>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException: Exception >>>>>>>>>>>> in thread "main" java.lang.NoClassDefFoundError: >>>>>>>>>>>> org/apache/hadoop/fs/FSDataInputStream >>>>>>>>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>>>>>>>> org.apache.hadoop.fs.FSDataInputStream >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> Ben >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <zjf...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> By default, 0.7.1 doesn't support spark 2.2. But you can set >>>>>>>>>>>>> zeppelin.spark.enableSupportedVersionCheck >>>>>>>>>>>>> in interpreter setting to disable the supported version check. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Jeff Zhang <zjf...@gmail.com>于2017年8月2日周三 上午8:18写道： >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> What's the error you see in log ? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Benjamin Kim <bbuil...@gmail.com>于2017年8月2日周三 上午8:18写道： >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Has anyone configured Zeppelin 0.7.1 for Cloudera's release >>>>>>>>>>>>>>> of Spark 2.2? I can't get it to work. I downloaded the binary >>>>>>>>>>>>>>> and set >>>>>>>>>>>>>>> SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must >>>>>>>>>>>>>>> be missing >>>>>>>>>>>>>>> something. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>> Ben >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> >
