I can't reproduce this in %spark, nor %sql
It seems to be %pyspark-specific.
Also seems it runs fine first time I start Zeppelin, then it shows this
error
You must build Spark with Hive. Export 'SPARK_HIVE=true' and run build/sbt
assembly
sqlc = HiveContext(sc)
sqlc.sql("select count(*) from hivedb.someTable")
It runs fine only one time, then
You must build Spark with Hive. Export 'SPARK_HIVE=true' and run build/sbt
> assembly
> Traceback (most recent call last):
> File "/tmp/zeppelin_pyspark-8000586427786928449.py", line 267, in <module>
> raise Exception(traceback.format_exc())
> Exception: Traceback (most recent call last):
> File "/tmp/zeppelin_pyspark-8000586427786928449.py", line 265, in <module>
> exec(code)
> File "<stdin>", line 2, in <module>
> File "/opt/cloudera/parcels/CDH/lib/spark/python/pyspark/sql/context.py",
> line 580, in sql
> return DataFrame(self._ssql_ctx.sql(sqlQuery), self)
> File "/opt/cloudera/parcels/CDH/lib/spark/python/pyspark/sql/context.py",
> line 683, in _ssql_ctx
> self._scala_HiveContext = self._get_hive_ctx()
> File "/opt/cloudera/parcels/CDH/lib/spark/python/pyspark/sql/context.py",
> line 692, in _get_hive_ctx
> return self._jvm.HiveContext(self._jsc.sc())
> File
> "/opt/cloudera/parcels/CDH/lib/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py",
> line 1064, in __call__
> answer, self._gateway_client, None, self._fqn)
> File "/opt/cloudera/parcels/CDH/lib/spark/python/pyspark/sql/utils.py",
> line 45, in deco
> return f(*a, **kw)
I don't see more details in logs than above error stack.
--
Ruslan Dautkhanov
On Wed, Nov 23, 2016 at 7:02 AM, Felix Cheung <felixcheun...@hotmail.com>
wrote:
> Hmm, SPARK_HOME is set it should pick up the right Spark.
>
> Does this work with the Scala Spark interpreter instead of pyspark? If it
> doesn't, is there more info in the log?
>
>
> ------------------------------
> *From:* Ruslan Dautkhanov <dautkha...@gmail.com>
> *Sent:* Monday, November 21, 2016 1:52:36 PM
> *To:* users@zeppelin.apache.org
> *Subject:* "You must build Spark with Hive. Export 'SPARK_HIVE=true'"
>
> Getting
> You must *build Spark with Hive*. Export 'SPARK_HIVE=true'
> See full stack [2] below.
>
> I'm using Spark 1.6 that comes with CDH 5.8.3.
> So it's definitely compiled with Hive.
> We use Jupyter notebooks without problems in the same environment.
>
> Using Zeppelin 0.6.2, downloaded as zeppelin-0.6.2-bin-all.tgz from from
> apache.org
>
> Is Zeppelin compiled with Hive too? I guess so.
> Not sure what else is missing.
>
> Tried to play with ZEPPELIN_SPARK_USEHIVECONTEXT but it does not make
> difference.
>
>
> [1]
> $ cat zeppelin-env.sh
> export JAVA_HOME=/usr/java/java7
> export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark
> export SPARK_SUBMIT_OPTIONS="--principal xxxx --keytab yyy --conf
> spark.driver.memory=7g --conf spark.executor.cores=2 --conf
> spark.executor.memory=8g"
> export SPARK_APP_NAME="Zeppelin notebook"
> export HADOOP_CONF_DIR=/etc/hadoop/conf
> export HIVE_CONF_DIR=/etc/hive/conf
> export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive
> export PYSPARK_PYTHON="/opt/cloudera/parcels/Anaconda/bin/python2"
> export PYTHONPATH="/opt/cloudera/parcels/CDH/lib/spark/python:/
> opt/cloudera/parcels/CDH/lib/spark/python/lib/py4j-0.9-src.zip"
> export MASTER="yarn-client"
> export ZEPPELIN_SPARK_USEHIVECONTEXT=true
>
>
>
>
> [2]
>
> You must build Spark with Hive. Export 'SPARK_HIVE=true' and run build/sbt
> assembly
> Traceback (most recent call last):
> File "/tmp/zeppelin_pyspark-9143637669637506477.py", line 267, in <module>
> raise Exception(traceback.format_exc())
> Exception: Traceback (most recent call last):
> File "/tmp/zeppelin_pyspark-9143637669637506477.py", line 265, in <module>
> exec(code)
> File "<stdin>", line 9, in <module>
> File "/opt/cloudera/parcels/CDH/lib/spark/python/pyspark/sql/context.py",
> line 580, in sql
>
> [3]
> Also have correct symlinks in zeppelin_home/conf for
> - hive-site.xml
> - hdfs-site.xml
> - core-site.xml
> - yarn-site.xml
>
>
>
> Thank you,
> Ruslan Dautkhanov
>
