Hi there
We have a CDH 5.9 Cluster where zeppelin 0.7.1is running fine with spark 1.6,
impala, hive, hbase and apache phoenix. As we have both, Spark 1.6 and Spark 2
distributed and activated on the cluster, we now also want to use a spark2
interpreter in zeppelin.
I created a new interpreter %spark2 in zeppelin belonging to the spark
interpreter group.
Also I added an interpreter property “SPARK_HOME” with value
/opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2
(I have also tried to set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2
with the same result.)
When testing the interpreter:
%spark2
sc.version
spark.version
we get
res0: String = 1.6.0
<console>:28: error: not found: value spark
spark.version
Why is the interpreter still using Spark 1.6? I thought the interpreter
property would override the SPARK_HOME setting in zeppelin-env.sh
Any ideas?
btw: we also use impersonation and run the interpreter instantiated per user in
isolated mode, but that as "just in case” info.
Best regards
Alex
