Spark error when loading phoenix-spark dependency
Hi all,
I am getting the following error when loading the
org.apache.phoenix:phoenix-spark:4.4.0-HBase-1.1 dependency from spark
interpreter. I am using Zeppelin Version 0.6.2-SNAPSHOT with spark 1.6.1 and
hdp 2.7.1.
The packages that I am inporting is:
import org.apache.phoenix.spark._
import org.apache.phoenix.spark.PhoenixRDD._
import java.sql.{ Date, Timestamp}
My build command is
mvn clean package -DskipTests -Drat.ignoreErrors=true
-Dcheckstyle.skip=true -Pspark-1.6 -Dspark.version=1.6.1 -Phadoop-2.6 -Pyarn
java.lang.NoSuchMethodError:
org.apache.spark.util.Utils$.resolveURIs(Ljava/lang/String;)Ljava/lang/String;
at
org.apache.spark.repl.SparkILoop$.getAddedJars(SparkILoop.scala:1079)
at
org.apache.spark.repl.SparkILoop.createInterpreter(SparkILoop.scala:210)
at
org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:698)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341)
at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
at
org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Thanks and Regard,
Vikash Kumar
Re: Zeppelin Kerberos error
Jeff- Thanks!! I figured the issue, I didn't need to copy the hive-site.xml
to spark_home/conf.
All I needed to do was set the SPARK_HOME environment variable in
"zeppelin-env.sh". That has made the local mode to work as well.
export SPARK_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/spark
In 0.5.6, even though I didn't set the SPARK_HOME in "zeppelin-env.sh", the
embedded spark was able to bootstrap itself and talk to my hive metastore.
Thanks & Regards,
Pradeep
On Wed, Aug 31, 2016 at 9:09 PM, Jeff Zhang wrote:
> AFAIK, kerberos should not related here. ZEPPELIN-1175 just remove
> ZEPPELIN_HOME/conf from the classpath of interpreter process. I guess you
> put hive-site.xml under ZEPPELIN_HOME/conf, can you try to put it
> under SPARK_HOME/conf ?
>
> On Thu, Sep 1, 2016 at 10:01 AM, Pradeep Reddy <
> pradeepreddy.a...@gmail.com> wrote:
>
>> I just tried enabling kerberos on 0.6.1 and its now able to talk to my
>> hive metastore, I see all my databases and tables. However the moment I
>> takeout kerberos and run zeppelin 0.6.1 in local mode by resetting the
>> "master" & removing spark variables, i see just one default database.
>>
>> I can live with this, as enabling kerberos integration is able to make
>> zeppelin talk to my hive metastore.
>>
>> Thanks,
>> Pradeep
>>
>> On Wed, Aug 31, 2016 at 8:36 PM, Pradeep Reddy <
>> pradeepreddy.a...@gmail.com> wrote:
>>
>>> No luck, even after copying the hive-site.xml in
>>> interpreter/spark/depI also tried, downloading 0.6.1 src vs 0.5.6. for
>>> the same steps I followed, 0.5.6 is able to talk to my hive metastore,
>>> where as the other recent builds are not, they just show one "default"
>>> database.
>>>
>>> Thanks,
>>> Pradeep
>>>
>>> On Wed, Aug 31, 2016 at 12:48 PM, Jongyoul Lee
>>> wrote:
>>>
I think it's related to https://issues.apache.org/j
ira/browse/ZEPPELIN-1175 which remove some class path when Zeppelin
launches interpreter. Could you please check your hive-site.xml is included
in your interpreter process? It looks like a configuration issue because
you can see the default database. If it doesn't exists, you should copy
your xml into interpreter/spark/dep/
Regards,
JL
On Wed, Aug 31, 2016 at 9:52 PM, Pradeep Reddy <
pradeepreddy.a...@gmail.com> wrote:
> Hi Jongyoul- I followed the exact same steps for compiling and setting
> up the new build from source as 0.5.6 (only difference is, I acquired the
> source for latest build using "git clone")
>
> hive-site.xml was copied to conf directory. But, the spark interpreter
> is not talking to the hive metastore. Both the 0.5.6 & the latest builds
> are running in the same machine. In 0.5.6 when i run the below command, I
> see 116 databases listed, as per my expectations and I'm able to run my
> notebooks built on those databases.
>
> [image: Inline image 1]
>
> Thanks,
> Pradeep
>
>
> On Wed, Aug 31, 2016 at 2:52 AM, Jongyoul Lee
> wrote:
>
>> Hello,
>>
>> Do you copy your hive-site.xml in a proper position?
>>
>> On Wed, Aug 31, 2016 at 3:52 PM, Pradeep Reddy <
>> pradeepreddy.a...@gmail.com> wrote:
>>
>>> nothing obvious. I will stick to 0.5.6 build, until the latest
>>> builds stabilize.
>>>
>>> On Wed, Aug 31, 2016 at 1:39 AM, Jeff Zhang
>>> wrote:
>>>
Then I guess maybe you are connecting to different database. Why
not using 'z.show(sql("databases"))' to display the databases ?
Then you will get a hint what's going on.
On Wed, Aug 31, 2016 at 2:36 PM, Pradeep Reddy <
pradeepreddy.a...@gmail.com> wrote:
> Yes...I didn't wish to show the names of the databases that we
> have in our data lake on that screen shot. so thats why I chose to
> display
> the count. The latest zeppelin build just shows 1 count which is
> "default"
> database.
>
> Thanks,
> Pradeep
>
> On Wed, Aug 31, 2016 at 1:33 AM, Jeff Zhang
> wrote:
>
>> 116 is the databases count number. Do you expect a list of
>> database ? then you need to use 'z.show(sql("databases"))'
>>
>> On Wed, Aug 31, 2016 at 2:26 PM, Pradeep Reddy <
>> pradeepreddy.a...@gmail.com> wrote:
>>
>>> Here it is Jeff
>>>
>>> [image: Inline image 1]
>>>
>>> On Wed, Aug 31, 2016 at 1:24 AM, Jeff Zhang
>>> wrote:
>>>
Hi Pradeep,
I don't see the databases on your screenshot (second one for
0.5.6). I think the output is correct.
On Wed, Aug 31, 2016 at 12:55 PM, Pradeep Reddy <
pradeepreddy.a...@gmail.com> wrote:
> Hi Jeff- I was able to make
Re: Zeppelin Kerberos error
AFAIK, kerberos should not related here. ZEPPELIN-1175 just remove
ZEPPELIN_HOME/conf from the classpath of interpreter process. I guess you
put hive-site.xml under ZEPPELIN_HOME/conf, can you try to put it
under SPARK_HOME/conf ?
On Thu, Sep 1, 2016 at 10:01 AM, Pradeep Reddy
wrote:
> I just tried enabling kerberos on 0.6.1 and its now able to talk to my
> hive metastore, I see all my databases and tables. However the moment I
> takeout kerberos and run zeppelin 0.6.1 in local mode by resetting the
> "master" & removing spark variables, i see just one default database.
>
> I can live with this, as enabling kerberos integration is able to make
> zeppelin talk to my hive metastore.
>
> Thanks,
> Pradeep
>
> On Wed, Aug 31, 2016 at 8:36 PM, Pradeep Reddy <
> pradeepreddy.a...@gmail.com> wrote:
>
>> No luck, even after copying the hive-site.xml in
>> interpreter/spark/depI also tried, downloading 0.6.1 src vs 0.5.6. for
>> the same steps I followed, 0.5.6 is able to talk to my hive metastore,
>> where as the other recent builds are not, they just show one "default"
>> database.
>>
>> Thanks,
>> Pradeep
>>
>> On Wed, Aug 31, 2016 at 12:48 PM, Jongyoul Lee
>> wrote:
>>
>>> I think it's related to https://issues.apache.org/j
>>> ira/browse/ZEPPELIN-1175 which remove some class path when Zeppelin
>>> launches interpreter. Could you please check your hive-site.xml is included
>>> in your interpreter process? It looks like a configuration issue because
>>> you can see the default database. If it doesn't exists, you should copy
>>> your xml into interpreter/spark/dep/
>>>
>>> Regards,
>>> JL
>>>
>>> On Wed, Aug 31, 2016 at 9:52 PM, Pradeep Reddy <
>>> pradeepreddy.a...@gmail.com> wrote:
>>>
Hi Jongyoul- I followed the exact same steps for compiling and setting
up the new build from source as 0.5.6 (only difference is, I acquired the
source for latest build using "git clone")
hive-site.xml was copied to conf directory. But, the spark interpreter
is not talking to the hive metastore. Both the 0.5.6 & the latest builds
are running in the same machine. In 0.5.6 when i run the below command, I
see 116 databases listed, as per my expectations and I'm able to run my
notebooks built on those databases.
[image: Inline image 1]
Thanks,
Pradeep
On Wed, Aug 31, 2016 at 2:52 AM, Jongyoul Lee
wrote:
> Hello,
>
> Do you copy your hive-site.xml in a proper position?
>
> On Wed, Aug 31, 2016 at 3:52 PM, Pradeep Reddy <
> pradeepreddy.a...@gmail.com> wrote:
>
>> nothing obvious. I will stick to 0.5.6 build, until the latest builds
>> stabilize.
>>
>> On Wed, Aug 31, 2016 at 1:39 AM, Jeff Zhang wrote:
>>
>>> Then I guess maybe you are connecting to different database. Why not
>>> using 'z.show(sql("databases"))' to display the databases ? Then
>>> you will get a hint what's going on.
>>>
>>> On Wed, Aug 31, 2016 at 2:36 PM, Pradeep Reddy <
>>> pradeepreddy.a...@gmail.com> wrote:
>>>
Yes...I didn't wish to show the names of the databases that we have
in our data lake on that screen shot. so thats why I chose to display
the
count. The latest zeppelin build just shows 1 count which is "default"
database.
Thanks,
Pradeep
On Wed, Aug 31, 2016 at 1:33 AM, Jeff Zhang
wrote:
> 116 is the databases count number. Do you expect a list of
> database ? then you need to use 'z.show(sql("databases"))'
>
> On Wed, Aug 31, 2016 at 2:26 PM, Pradeep Reddy <
> pradeepreddy.a...@gmail.com> wrote:
>
>> Here it is Jeff
>>
>> [image: Inline image 1]
>>
>> On Wed, Aug 31, 2016 at 1:24 AM, Jeff Zhang
>> wrote:
>>
>>> Hi Pradeep,
>>>
>>> I don't see the databases on your screenshot (second one for
>>> 0.5.6). I think the output is correct.
>>>
>>>
>>>
>>> On Wed, Aug 31, 2016 at 12:55 PM, Pradeep Reddy <
>>> pradeepreddy.a...@gmail.com> wrote:
>>>
Hi Jeff- I was able to make Kerberos work in 0.5.6 zeppelin
build. It seems like Kerberos not working & spark not able to talk
to the
shared hive meta store are defects in the current build.
On Tue, Aug 30, 2016 at 11:09 PM, Pradeep Reddy <
pradeepreddy.a...@gmail.com> wrote:
> Hi Jeff-
>
> I switched to local mode now, I'm able to summon the implicit
> objects like sc, sqlContext etc., but it doesn't show my
> databases &
> tables, just shows 1 database "default".
>
> Zeppelin Latest Build
>
> [image: Inline image 3]
>
Re: Zeppelin Kerberos error
I just tried enabling kerberos on 0.6.1 and its now able to talk to my hive
metastore, I see all my databases and tables. However the moment I takeout
kerberos and run zeppelin 0.6.1 in local mode by resetting the "master" &
removing spark variables, i see just one default database.
I can live with this, as enabling kerberos integration is able to make
zeppelin talk to my hive metastore.
Thanks,
Pradeep
On Wed, Aug 31, 2016 at 8:36 PM, Pradeep Reddy
wrote:
> No luck, even after copying the hive-site.xml in
> interpreter/spark/depI also tried, downloading 0.6.1 src vs 0.5.6. for
> the same steps I followed, 0.5.6 is able to talk to my hive metastore,
> where as the other recent builds are not, they just show one "default"
> database.
>
> Thanks,
> Pradeep
>
> On Wed, Aug 31, 2016 at 12:48 PM, Jongyoul Lee wrote:
>
>> I think it's related to https://issues.apache.org/j
>> ira/browse/ZEPPELIN-1175 which remove some class path when Zeppelin
>> launches interpreter. Could you please check your hive-site.xml is included
>> in your interpreter process? It looks like a configuration issue because
>> you can see the default database. If it doesn't exists, you should copy
>> your xml into interpreter/spark/dep/
>>
>> Regards,
>> JL
>>
>> On Wed, Aug 31, 2016 at 9:52 PM, Pradeep Reddy <
>> pradeepreddy.a...@gmail.com> wrote:
>>
>>> Hi Jongyoul- I followed the exact same steps for compiling and setting
>>> up the new build from source as 0.5.6 (only difference is, I acquired the
>>> source for latest build using "git clone")
>>>
>>> hive-site.xml was copied to conf directory. But, the spark interpreter
>>> is not talking to the hive metastore. Both the 0.5.6 & the latest builds
>>> are running in the same machine. In 0.5.6 when i run the below command, I
>>> see 116 databases listed, as per my expectations and I'm able to run my
>>> notebooks built on those databases.
>>>
>>> [image: Inline image 1]
>>>
>>> Thanks,
>>> Pradeep
>>>
>>>
>>> On Wed, Aug 31, 2016 at 2:52 AM, Jongyoul Lee
>>> wrote:
>>>
Hello,
Do you copy your hive-site.xml in a proper position?
On Wed, Aug 31, 2016 at 3:52 PM, Pradeep Reddy <
pradeepreddy.a...@gmail.com> wrote:
> nothing obvious. I will stick to 0.5.6 build, until the latest builds
> stabilize.
>
> On Wed, Aug 31, 2016 at 1:39 AM, Jeff Zhang wrote:
>
>> Then I guess maybe you are connecting to different database. Why not
>> using 'z.show(sql("databases"))' to display the databases ? Then
>> you will get a hint what's going on.
>>
>> On Wed, Aug 31, 2016 at 2:36 PM, Pradeep Reddy <
>> pradeepreddy.a...@gmail.com> wrote:
>>
>>> Yes...I didn't wish to show the names of the databases that we have
>>> in our data lake on that screen shot. so thats why I chose to display
>>> the
>>> count. The latest zeppelin build just shows 1 count which is "default"
>>> database.
>>>
>>> Thanks,
>>> Pradeep
>>>
>>> On Wed, Aug 31, 2016 at 1:33 AM, Jeff Zhang
>>> wrote:
>>>
116 is the databases count number. Do you expect a list of database
? then you need to use 'z.show(sql("databases"))'
On Wed, Aug 31, 2016 at 2:26 PM, Pradeep Reddy <
pradeepreddy.a...@gmail.com> wrote:
> Here it is Jeff
>
> [image: Inline image 1]
>
> On Wed, Aug 31, 2016 at 1:24 AM, Jeff Zhang
> wrote:
>
>> Hi Pradeep,
>>
>> I don't see the databases on your screenshot (second one for
>> 0.5.6). I think the output is correct.
>>
>>
>>
>> On Wed, Aug 31, 2016 at 12:55 PM, Pradeep Reddy <
>> pradeepreddy.a...@gmail.com> wrote:
>>
>>> Hi Jeff- I was able to make Kerberos work in 0.5.6 zeppelin
>>> build. It seems like Kerberos not working & spark not able to talk
>>> to the
>>> shared hive meta store are defects in the current build.
>>>
>>> On Tue, Aug 30, 2016 at 11:09 PM, Pradeep Reddy <
>>> pradeepreddy.a...@gmail.com> wrote:
>>>
Hi Jeff-
I switched to local mode now, I'm able to summon the implicit
objects like sc, sqlContext etc., but it doesn't show my databases
&
tables, just shows 1 database "default".
Zeppelin Latest Build
[image: Inline image 3]
Zeppelin 0.5.6, running on the same machine, is able to show my
databases and tables.
[image: Inline image 4]
On Tue, Aug 30, 2016 at 8:20 PM, Jeff Zhang
wrote:
> > the spark interpreter is not showing my tables & databases,
> may be its running in an isolated mode... I'm just getting empty
> list, so I
>
Re: Zeppelin Kerberos error
No luck, even after copying the hive-site.xml in interpreter/spark/depI
also tried, downloading 0.6.1 src vs 0.5.6. for the same steps I followed,
0.5.6 is able to talk to my hive metastore, where as the other recent
builds are not, they just show one "default" database.
Thanks,
Pradeep
On Wed, Aug 31, 2016 at 12:48 PM, Jongyoul Lee wrote:
> I think it's related to https://issues.apache.org/
> jira/browse/ZEPPELIN-1175 which remove some class path when Zeppelin
> launches interpreter. Could you please check your hive-site.xml is included
> in your interpreter process? It looks like a configuration issue because
> you can see the default database. If it doesn't exists, you should copy
> your xml into interpreter/spark/dep/
>
> Regards,
> JL
>
> On Wed, Aug 31, 2016 at 9:52 PM, Pradeep Reddy <
> pradeepreddy.a...@gmail.com> wrote:
>
>> Hi Jongyoul- I followed the exact same steps for compiling and setting up
>> the new build from source as 0.5.6 (only difference is, I acquired the
>> source for latest build using "git clone")
>>
>> hive-site.xml was copied to conf directory. But, the spark interpreter is
>> not talking to the hive metastore. Both the 0.5.6 & the latest builds are
>> running in the same machine. In 0.5.6 when i run the below command, I see
>> 116 databases listed, as per my expectations and I'm able to run my
>> notebooks built on those databases.
>>
>> [image: Inline image 1]
>>
>> Thanks,
>> Pradeep
>>
>>
>> On Wed, Aug 31, 2016 at 2:52 AM, Jongyoul Lee wrote:
>>
>>> Hello,
>>>
>>> Do you copy your hive-site.xml in a proper position?
>>>
>>> On Wed, Aug 31, 2016 at 3:52 PM, Pradeep Reddy <
>>> pradeepreddy.a...@gmail.com> wrote:
>>>
nothing obvious. I will stick to 0.5.6 build, until the latest builds
stabilize.
On Wed, Aug 31, 2016 at 1:39 AM, Jeff Zhang wrote:
> Then I guess maybe you are connecting to different database. Why not
> using 'z.show(sql("databases"))' to display the databases ? Then you
> will get a hint what's going on.
>
> On Wed, Aug 31, 2016 at 2:36 PM, Pradeep Reddy <
> pradeepreddy.a...@gmail.com> wrote:
>
>> Yes...I didn't wish to show the names of the databases that we have
>> in our data lake on that screen shot. so thats why I chose to display the
>> count. The latest zeppelin build just shows 1 count which is "default"
>> database.
>>
>> Thanks,
>> Pradeep
>>
>> On Wed, Aug 31, 2016 at 1:33 AM, Jeff Zhang wrote:
>>
>>> 116 is the databases count number. Do you expect a list of database
>>> ? then you need to use 'z.show(sql("databases"))'
>>>
>>> On Wed, Aug 31, 2016 at 2:26 PM, Pradeep Reddy <
>>> pradeepreddy.a...@gmail.com> wrote:
>>>
Here it is Jeff
[image: Inline image 1]
On Wed, Aug 31, 2016 at 1:24 AM, Jeff Zhang
wrote:
> Hi Pradeep,
>
> I don't see the databases on your screenshot (second one for
> 0.5.6). I think the output is correct.
>
>
>
> On Wed, Aug 31, 2016 at 12:55 PM, Pradeep Reddy <
> pradeepreddy.a...@gmail.com> wrote:
>
>> Hi Jeff- I was able to make Kerberos work in 0.5.6 zeppelin
>> build. It seems like Kerberos not working & spark not able to talk
>> to the
>> shared hive meta store are defects in the current build.
>>
>> On Tue, Aug 30, 2016 at 11:09 PM, Pradeep Reddy <
>> pradeepreddy.a...@gmail.com> wrote:
>>
>>> Hi Jeff-
>>>
>>> I switched to local mode now, I'm able to summon the implicit
>>> objects like sc, sqlContext etc., but it doesn't show my databases &
>>> tables, just shows 1 database "default".
>>>
>>> Zeppelin Latest Build
>>>
>>> [image: Inline image 3]
>>>
>>> Zeppelin 0.5.6, running on the same machine, is able to show my
>>> databases and tables.
>>>
>>> [image: Inline image 4]
>>>
>>> On Tue, Aug 30, 2016 at 8:20 PM, Jeff Zhang
>>> wrote:
>>>
> the spark interpreter is not showing my tables & databases,
may be its running in an isolated mode... I'm just getting empty
list, so I
attempted to do kerberos authentication to workaround that issue,
and
bumped into this road block.
kerberos would not help here, actually I think it would make
the problem more complicated. You need to first check the log why
you get
empty list.
On Wed, Aug 31, 2016 at 8:56 AM, Pradeep Reddy <
pradeepreddy.a...@gmail.com> wrote:
> Jeff- I was also successfully able to run spark shell, after
> running kdestroy, with the below command and was able to get t
Re: Zeppelin Kerberos error
I think it's related to https://issues.apache.org/jira/browse/ZEPPELIN-1175
which remove some class path when Zeppelin launches interpreter. Could you
please check your hive-site.xml is included in your interpreter process? It
looks like a configuration issue because you can see the default database.
If it doesn't exists, you should copy your xml into interpreter/spark/dep/
Regards,
JL
On Wed, Aug 31, 2016 at 9:52 PM, Pradeep Reddy
wrote:
> Hi Jongyoul- I followed the exact same steps for compiling and setting up
> the new build from source as 0.5.6 (only difference is, I acquired the
> source for latest build using "git clone")
>
> hive-site.xml was copied to conf directory. But, the spark interpreter is
> not talking to the hive metastore. Both the 0.5.6 & the latest builds are
> running in the same machine. In 0.5.6 when i run the below command, I see
> 116 databases listed, as per my expectations and I'm able to run my
> notebooks built on those databases.
>
> [image: Inline image 1]
>
> Thanks,
> Pradeep
>
>
> On Wed, Aug 31, 2016 at 2:52 AM, Jongyoul Lee wrote:
>
>> Hello,
>>
>> Do you copy your hive-site.xml in a proper position?
>>
>> On Wed, Aug 31, 2016 at 3:52 PM, Pradeep Reddy <
>> pradeepreddy.a...@gmail.com> wrote:
>>
>>> nothing obvious. I will stick to 0.5.6 build, until the latest builds
>>> stabilize.
>>>
>>> On Wed, Aug 31, 2016 at 1:39 AM, Jeff Zhang wrote:
>>>
Then I guess maybe you are connecting to different database. Why not
using 'z.show(sql("databases"))' to display the databases ? Then you
will get a hint what's going on.
On Wed, Aug 31, 2016 at 2:36 PM, Pradeep Reddy <
pradeepreddy.a...@gmail.com> wrote:
> Yes...I didn't wish to show the names of the databases that we have in
> our data lake on that screen shot. so thats why I chose to display the
> count. The latest zeppelin build just shows 1 count which is "default"
> database.
>
> Thanks,
> Pradeep
>
> On Wed, Aug 31, 2016 at 1:33 AM, Jeff Zhang wrote:
>
>> 116 is the databases count number. Do you expect a list of database ?
>> then you need to use 'z.show(sql("databases"))'
>>
>> On Wed, Aug 31, 2016 at 2:26 PM, Pradeep Reddy <
>> pradeepreddy.a...@gmail.com> wrote:
>>
>>> Here it is Jeff
>>>
>>> [image: Inline image 1]
>>>
>>> On Wed, Aug 31, 2016 at 1:24 AM, Jeff Zhang
>>> wrote:
>>>
Hi Pradeep,
I don't see the databases on your screenshot (second one for
0.5.6). I think the output is correct.
On Wed, Aug 31, 2016 at 12:55 PM, Pradeep Reddy <
pradeepreddy.a...@gmail.com> wrote:
> Hi Jeff- I was able to make Kerberos work in 0.5.6 zeppelin build.
> It seems like Kerberos not working & spark not able to talk to the
> shared
> hive meta store are defects in the current build.
>
> On Tue, Aug 30, 2016 at 11:09 PM, Pradeep Reddy <
> pradeepreddy.a...@gmail.com> wrote:
>
>> Hi Jeff-
>>
>> I switched to local mode now, I'm able to summon the implicit
>> objects like sc, sqlContext etc., but it doesn't show my databases &
>> tables, just shows 1 database "default".
>>
>> Zeppelin Latest Build
>>
>> [image: Inline image 3]
>>
>> Zeppelin 0.5.6, running on the same machine, is able to show my
>> databases and tables.
>>
>> [image: Inline image 4]
>>
>> On Tue, Aug 30, 2016 at 8:20 PM, Jeff Zhang
>> wrote:
>>
>>> > the spark interpreter is not showing my tables & databases,
>>> may be its running in an isolated mode... I'm just getting empty
>>> list, so I
>>> attempted to do kerberos authentication to workaround that issue,
>>> and
>>> bumped into this road block.
>>>
>>> kerberos would not help here, actually I think it would make the
>>> problem more complicated. You need to first check the log why you
>>> get
>>> empty list.
>>>
>>> On Wed, Aug 31, 2016 at 8:56 AM, Pradeep Reddy <
>>> pradeepreddy.a...@gmail.com> wrote:
>>>
Jeff- I was also successfully able to run spark shell, after
running kdestroy, with the below command and was able to get to my
hive
tables.
spark-shell --conf spark.yarn.keytab=$HOME/pradeep.x.alla.keytab
--conf spark.yarn.principal=pradeep.x.alla --deploy-mode
client --master yarn --queue
On Tue, Aug 30, 2016 at 7:34 PM, Pradeep Reddy <
pradeepreddy.a...@gmail.com> wrote:
> Thanks Jeff..I have always used zeppelin in local mode, but
> when I migrated from 0.5.6 to this version,
Re: Configure Zeppelin 0.61 for Hive
Using this URL made it work: jdbc:hive2://myhost.example.com:21050/;auth=noSasl On Wed, Aug 31, 2016 at 11:13 AM, Abhi Basu <9000r...@gmail.com> wrote: > Except spark-sql is geared more towards developers and our users are > looking for a SQL engine like hive (except faster). :) > > > > On Wed, Aug 31, 2016 at 11:11 AM, Pradeep Reddy < > pradeepreddy.a...@gmail.com> wrote: > >> You could use the JDBC interpreter to setup a new interpreter for Impala. >> That said, Impala is just using your hive metastore and enabling >> proprietary caching for high performance on your hive database tables >> rather than doing map reduce translation of hive queries. Running Spark SQL >> on spark interpreter can be considered as a reasonable alternative to >> running impala queries through jdbc. >> >> http://www.cloudera.com/documentation/archive/impala/2-x/2- >> 1-x/topics/impala_jdbc.html >> >> Thanks, >> Pradeep >> >> >> On Wed, Aug 31, 2016 at 10:45 AM, Abhi Basu <9000r...@gmail.com> wrote: >> >>> How do I setup a connection to impala? Do I need to point to >>> impala-jdbc jar in dependencies? >>> >>> Thanks, >>> >>> Abhi >>> >>> On Wed, Aug 31, 2016 at 10:36 AM, Abhi Basu <9000r...@gmail.com> wrote: >>> OK, got it. Added the hadoop jar to dependencies and it started working. Thanks. On Wed, Aug 31, 2016 at 10:24 AM, Abhi Basu <9000r...@gmail.com> wrote: > So, path to the jars like /usr/lib/hive/* ? > > On Wed, Aug 31, 2016 at 9:53 AM, Jeff Zhang wrote: > >> You don't need to copy these jars manually, just specify them in the >> interpreter setting page. >> >> On Wed, Aug 31, 2016 at 9:52 PM, Abhi Basu <9000r...@gmail.com> >> wrote: >> >>> Where do these jars have to be placed? >>> >>> I thought copying the hive-site.xml and pointing to hadoop conf >>> folder in zeppelin conf should be enough (like before). >>> >>> Thanks, >>> >>> Abhi >>> >>> On Tue, Aug 30, 2016 at 6:59 PM, Jeff Zhang >>> wrote: >>> You need add the following 2 dependencies in the interpreter setting page. https://zeppelin.apache.org/docs/0.6.1/interpreter/hive.html #dependencies org.apache.hive:hive-jdbc:0.14.0 org.apache.hadoop:hadoop-common:2.6.0 On Wed, Aug 31, 2016 at 2:39 AM, Abhi Basu <9000r...@gmail.com> wrote: > Folks: > > Seems like a config issue. > > 1. Copied hive-site.xml into /ZEPP_HOME/conf folder > 2. Added following to config file: > > export JAVA_HOME=/./...export HADOOP_CONF_DIR=/etc/hadoop/conf > > > I am using Zeppelin after a while, and looks like Hive interpreter > is part of JDBC interpreter now. > Interpreter properties seem to be set correctly: > PropertyValue > hive.driver org.apache.hive.jdbc.HiveDriver > hive.url jdbc:hive2://localhost:1 > hive.user hiveUser > hive.password hivePassword > > When I run %hive from Zeppelin, I get a hive jdbc driver not found > error. How do I fix this? Also, how do I configure for Impala within > the > JDBC section of interpreters. > > Thanks, > > Abhi > > -- > Abhi Basu > -- Best Regards Jeff Zhang >>> >>> >>> >>> -- >>> Abhi Basu >>> >> >> >> >> -- >> Best Regards >> >> Jeff Zhang >> > > > > -- > Abhi Basu > -- Abhi Basu >>> >>> >>> >>> -- >>> Abhi Basu >>> >> >> > > > -- > Abhi Basu > -- Abhi Basu
Re: Configure Zeppelin 0.61 for Hive
Except spark-sql is geared more towards developers and our users are looking for a SQL engine like hive (except faster). :) On Wed, Aug 31, 2016 at 11:11 AM, Pradeep Reddy wrote: > You could use the JDBC interpreter to setup a new interpreter for Impala. > That said, Impala is just using your hive metastore and enabling > proprietary caching for high performance on your hive database tables > rather than doing map reduce translation of hive queries. Running Spark SQL > on spark interpreter can be considered as a reasonable alternative to > running impala queries through jdbc. > > http://www.cloudera.com/documentation/archive/impala/ > 2-x/2-1-x/topics/impala_jdbc.html > > Thanks, > Pradeep > > > On Wed, Aug 31, 2016 at 10:45 AM, Abhi Basu <9000r...@gmail.com> wrote: > >> How do I setup a connection to impala? Do I need to point to impala-jdbc >> jar in dependencies? >> >> Thanks, >> >> Abhi >> >> On Wed, Aug 31, 2016 at 10:36 AM, Abhi Basu <9000r...@gmail.com> wrote: >> >>> OK, got it. Added the hadoop jar to dependencies and it started working. >>> >>> Thanks. >>> >>> On Wed, Aug 31, 2016 at 10:24 AM, Abhi Basu <9000r...@gmail.com> wrote: >>> So, path to the jars like /usr/lib/hive/* ? On Wed, Aug 31, 2016 at 9:53 AM, Jeff Zhang wrote: > You don't need to copy these jars manually, just specify them in the > interpreter setting page. > > On Wed, Aug 31, 2016 at 9:52 PM, Abhi Basu <9000r...@gmail.com> wrote: > >> Where do these jars have to be placed? >> >> I thought copying the hive-site.xml and pointing to hadoop conf >> folder in zeppelin conf should be enough (like before). >> >> Thanks, >> >> Abhi >> >> On Tue, Aug 30, 2016 at 6:59 PM, Jeff Zhang wrote: >> >>> You need add the following 2 dependencies in the interpreter setting >>> page. >>> >>> https://zeppelin.apache.org/docs/0.6.1/interpreter/hive.html >>> #dependencies >>> >>> org.apache.hive:hive-jdbc:0.14.0 >>> org.apache.hadoop:hadoop-common:2.6.0 >>> >>> >>> On Wed, Aug 31, 2016 at 2:39 AM, Abhi Basu <9000r...@gmail.com> >>> wrote: >>> Folks: Seems like a config issue. 1. Copied hive-site.xml into /ZEPP_HOME/conf folder 2. Added following to config file: export JAVA_HOME=/./...export HADOOP_CONF_DIR=/etc/hadoop/conf I am using Zeppelin after a while, and looks like Hive interpreter is part of JDBC interpreter now. Interpreter properties seem to be set correctly: PropertyValue hive.driver org.apache.hive.jdbc.HiveDriver hive.url jdbc:hive2://localhost:1 hive.user hiveUser hive.password hivePassword When I run %hive from Zeppelin, I get a hive jdbc driver not found error. How do I fix this? Also, how do I configure for Impala within the JDBC section of interpreters. Thanks, Abhi -- Abhi Basu >>> >>> >>> >>> -- >>> Best Regards >>> >>> Jeff Zhang >>> >> >> >> >> -- >> Abhi Basu >> > > > > -- > Best Regards > > Jeff Zhang > -- Abhi Basu >>> >>> >>> >>> -- >>> Abhi Basu >>> >> >> >> >> -- >> Abhi Basu >> > > -- Abhi Basu
Re: Configure Zeppelin 0.61 for Hive
You could use the JDBC interpreter to setup a new interpreter for Impala. That said, Impala is just using your hive metastore and enabling proprietary caching for high performance on your hive database tables rather than doing map reduce translation of hive queries. Running Spark SQL on spark interpreter can be considered as a reasonable alternative to running impala queries through jdbc. http://www.cloudera.com/documentation/archive/impala/2-x/2-1-x/topics/impala_jdbc.html Thanks, Pradeep On Wed, Aug 31, 2016 at 10:45 AM, Abhi Basu <9000r...@gmail.com> wrote: > How do I setup a connection to impala? Do I need to point to impala-jdbc > jar in dependencies? > > Thanks, > > Abhi > > On Wed, Aug 31, 2016 at 10:36 AM, Abhi Basu <9000r...@gmail.com> wrote: > >> OK, got it. Added the hadoop jar to dependencies and it started working. >> >> Thanks. >> >> On Wed, Aug 31, 2016 at 10:24 AM, Abhi Basu <9000r...@gmail.com> wrote: >> >>> So, path to the jars like /usr/lib/hive/* ? >>> >>> On Wed, Aug 31, 2016 at 9:53 AM, Jeff Zhang wrote: >>> You don't need to copy these jars manually, just specify them in the interpreter setting page. On Wed, Aug 31, 2016 at 9:52 PM, Abhi Basu <9000r...@gmail.com> wrote: > Where do these jars have to be placed? > > I thought copying the hive-site.xml and pointing to hadoop conf folder > in zeppelin conf should be enough (like before). > > Thanks, > > Abhi > > On Tue, Aug 30, 2016 at 6:59 PM, Jeff Zhang wrote: > >> You need add the following 2 dependencies in the interpreter setting >> page. >> >> https://zeppelin.apache.org/docs/0.6.1/interpreter/hive.html >> #dependencies >> >> org.apache.hive:hive-jdbc:0.14.0 >> org.apache.hadoop:hadoop-common:2.6.0 >> >> >> On Wed, Aug 31, 2016 at 2:39 AM, Abhi Basu <9000r...@gmail.com> >> wrote: >> >>> Folks: >>> >>> Seems like a config issue. >>> >>> 1. Copied hive-site.xml into /ZEPP_HOME/conf folder >>> 2. Added following to config file: >>> >>> export JAVA_HOME=/./...export HADOOP_CONF_DIR=/etc/hadoop/conf >>> >>> >>> I am using Zeppelin after a while, and looks like Hive interpreter >>> is part of JDBC interpreter now. >>> Interpreter properties seem to be set correctly: >>> PropertyValue >>> hive.driver org.apache.hive.jdbc.HiveDriver >>> hive.url jdbc:hive2://localhost:1 >>> hive.user hiveUser >>> hive.password hivePassword >>> >>> When I run %hive from Zeppelin, I get a hive jdbc driver not found >>> error. How do I fix this? Also, how do I configure for Impala within the >>> JDBC section of interpreters. >>> >>> Thanks, >>> >>> Abhi >>> >>> -- >>> Abhi Basu >>> >> >> >> >> -- >> Best Regards >> >> Jeff Zhang >> > > > > -- > Abhi Basu > -- Best Regards Jeff Zhang >>> >>> >>> >>> -- >>> Abhi Basu >>> >> >> >> >> -- >> Abhi Basu >> > > > > -- > Abhi Basu >
Re: Configure Zeppelin 0.61 for Hive
How do I setup a connection to impala? Do I need to point to impala-jdbc jar in dependencies? Thanks, Abhi On Wed, Aug 31, 2016 at 10:36 AM, Abhi Basu <9000r...@gmail.com> wrote: > OK, got it. Added the hadoop jar to dependencies and it started working. > > Thanks. > > On Wed, Aug 31, 2016 at 10:24 AM, Abhi Basu <9000r...@gmail.com> wrote: > >> So, path to the jars like /usr/lib/hive/* ? >> >> On Wed, Aug 31, 2016 at 9:53 AM, Jeff Zhang wrote: >> >>> You don't need to copy these jars manually, just specify them in the >>> interpreter setting page. >>> >>> On Wed, Aug 31, 2016 at 9:52 PM, Abhi Basu <9000r...@gmail.com> wrote: >>> Where do these jars have to be placed? I thought copying the hive-site.xml and pointing to hadoop conf folder in zeppelin conf should be enough (like before). Thanks, Abhi On Tue, Aug 30, 2016 at 6:59 PM, Jeff Zhang wrote: > You need add the following 2 dependencies in the interpreter setting > page. > > https://zeppelin.apache.org/docs/0.6.1/interpreter/hive.html > #dependencies > > org.apache.hive:hive-jdbc:0.14.0 > org.apache.hadoop:hadoop-common:2.6.0 > > > On Wed, Aug 31, 2016 at 2:39 AM, Abhi Basu <9000r...@gmail.com> wrote: > >> Folks: >> >> Seems like a config issue. >> >> 1. Copied hive-site.xml into /ZEPP_HOME/conf folder >> 2. Added following to config file: >> >> export JAVA_HOME=/./...export HADOOP_CONF_DIR=/etc/hadoop/conf >> >> >> I am using Zeppelin after a while, and looks like Hive interpreter is >> part of JDBC interpreter now. >> Interpreter properties seem to be set correctly: >> PropertyValue >> hive.driver org.apache.hive.jdbc.HiveDriver >> hive.url jdbc:hive2://localhost:1 >> hive.user hiveUser >> hive.password hivePassword >> >> When I run %hive from Zeppelin, I get a hive jdbc driver not found >> error. How do I fix this? Also, how do I configure for Impala within the >> JDBC section of interpreters. >> >> Thanks, >> >> Abhi >> >> -- >> Abhi Basu >> > > > > -- > Best Regards > > Jeff Zhang > -- Abhi Basu >>> >>> >>> >>> -- >>> Best Regards >>> >>> Jeff Zhang >>> >> >> >> >> -- >> Abhi Basu >> > > > > -- > Abhi Basu > -- Abhi Basu
Re: Configure Zeppelin 0.61 for Hive
OK, got it. Added the hadoop jar to dependencies and it started working. Thanks. On Wed, Aug 31, 2016 at 10:24 AM, Abhi Basu <9000r...@gmail.com> wrote: > So, path to the jars like /usr/lib/hive/* ? > > On Wed, Aug 31, 2016 at 9:53 AM, Jeff Zhang wrote: > >> You don't need to copy these jars manually, just specify them in the >> interpreter setting page. >> >> On Wed, Aug 31, 2016 at 9:52 PM, Abhi Basu <9000r...@gmail.com> wrote: >> >>> Where do these jars have to be placed? >>> >>> I thought copying the hive-site.xml and pointing to hadoop conf folder >>> in zeppelin conf should be enough (like before). >>> >>> Thanks, >>> >>> Abhi >>> >>> On Tue, Aug 30, 2016 at 6:59 PM, Jeff Zhang wrote: >>> You need add the following 2 dependencies in the interpreter setting page. https://zeppelin.apache.org/docs/0.6.1/interpreter/hive.html #dependencies org.apache.hive:hive-jdbc:0.14.0 org.apache.hadoop:hadoop-common:2.6.0 On Wed, Aug 31, 2016 at 2:39 AM, Abhi Basu <9000r...@gmail.com> wrote: > Folks: > > Seems like a config issue. > > 1. Copied hive-site.xml into /ZEPP_HOME/conf folder > 2. Added following to config file: > > export JAVA_HOME=/./...export HADOOP_CONF_DIR=/etc/hadoop/conf > > > I am using Zeppelin after a while, and looks like Hive interpreter is > part of JDBC interpreter now. > Interpreter properties seem to be set correctly: > PropertyValue > hive.driver org.apache.hive.jdbc.HiveDriver > hive.url jdbc:hive2://localhost:1 > hive.user hiveUser > hive.password hivePassword > > When I run %hive from Zeppelin, I get a hive jdbc driver not found > error. How do I fix this? Also, how do I configure for Impala within the > JDBC section of interpreters. > > Thanks, > > Abhi > > -- > Abhi Basu > -- Best Regards Jeff Zhang >>> >>> >>> >>> -- >>> Abhi Basu >>> >> >> >> >> -- >> Best Regards >> >> Jeff Zhang >> > > > > -- > Abhi Basu > -- Abhi Basu
Re: Configure Zeppelin 0.61 for Hive
So, path to the jars like /usr/lib/hive/* ? On Wed, Aug 31, 2016 at 9:53 AM, Jeff Zhang wrote: > You don't need to copy these jars manually, just specify them in the > interpreter setting page. > > On Wed, Aug 31, 2016 at 9:52 PM, Abhi Basu <9000r...@gmail.com> wrote: > >> Where do these jars have to be placed? >> >> I thought copying the hive-site.xml and pointing to hadoop conf folder in >> zeppelin conf should be enough (like before). >> >> Thanks, >> >> Abhi >> >> On Tue, Aug 30, 2016 at 6:59 PM, Jeff Zhang wrote: >> >>> You need add the following 2 dependencies in the interpreter setting >>> page. >>> >>> https://zeppelin.apache.org/docs/0.6.1/interpreter/hive.html >>> #dependencies >>> >>> org.apache.hive:hive-jdbc:0.14.0 >>> org.apache.hadoop:hadoop-common:2.6.0 >>> >>> >>> On Wed, Aug 31, 2016 at 2:39 AM, Abhi Basu <9000r...@gmail.com> wrote: >>> Folks: Seems like a config issue. 1. Copied hive-site.xml into /ZEPP_HOME/conf folder 2. Added following to config file: export JAVA_HOME=/./...export HADOOP_CONF_DIR=/etc/hadoop/conf I am using Zeppelin after a while, and looks like Hive interpreter is part of JDBC interpreter now. Interpreter properties seem to be set correctly: PropertyValue hive.driver org.apache.hive.jdbc.HiveDriver hive.url jdbc:hive2://localhost:1 hive.user hiveUser hive.password hivePassword When I run %hive from Zeppelin, I get a hive jdbc driver not found error. How do I fix this? Also, how do I configure for Impala within the JDBC section of interpreters. Thanks, Abhi -- Abhi Basu >>> >>> >>> >>> -- >>> Best Regards >>> >>> Jeff Zhang >>> >> >> >> >> -- >> Abhi Basu >> > > > > -- > Best Regards > > Jeff Zhang > -- Abhi Basu
Re: Configure Zeppelin 0.61 for Hive
You don't need to copy these jars manually, just specify them in the interpreter setting page. On Wed, Aug 31, 2016 at 9:52 PM, Abhi Basu <9000r...@gmail.com> wrote: > Where do these jars have to be placed? > > I thought copying the hive-site.xml and pointing to hadoop conf folder in > zeppelin conf should be enough (like before). > > Thanks, > > Abhi > > On Tue, Aug 30, 2016 at 6:59 PM, Jeff Zhang wrote: > >> You need add the following 2 dependencies in the interpreter setting >> page. >> >> https://zeppelin.apache.org/docs/0.6.1/interpreter/hive.html#dependencies >> >> org.apache.hive:hive-jdbc:0.14.0 >> org.apache.hadoop:hadoop-common:2.6.0 >> >> >> On Wed, Aug 31, 2016 at 2:39 AM, Abhi Basu <9000r...@gmail.com> wrote: >> >>> Folks: >>> >>> Seems like a config issue. >>> >>> 1. Copied hive-site.xml into /ZEPP_HOME/conf folder >>> 2. Added following to config file: >>> >>> export JAVA_HOME=/./...export HADOOP_CONF_DIR=/etc/hadoop/conf >>> >>> >>> I am using Zeppelin after a while, and looks like Hive interpreter is >>> part of JDBC interpreter now. >>> Interpreter properties seem to be set correctly: >>> PropertyValue >>> hive.driver org.apache.hive.jdbc.HiveDriver >>> hive.url jdbc:hive2://localhost:1 >>> hive.user hiveUser >>> hive.password hivePassword >>> >>> When I run %hive from Zeppelin, I get a hive jdbc driver not found >>> error. How do I fix this? Also, how do I configure for Impala within the >>> JDBC section of interpreters. >>> >>> Thanks, >>> >>> Abhi >>> >>> -- >>> Abhi Basu >>> >> >> >> >> -- >> Best Regards >> >> Jeff Zhang >> > > > > -- > Abhi Basu > -- Best Regards Jeff Zhang
Re: Configure Zeppelin 0.61 for Hive
Where do these jars have to be placed? I thought copying the hive-site.xml and pointing to hadoop conf folder in zeppelin conf should be enough (like before). Thanks, Abhi On Tue, Aug 30, 2016 at 6:59 PM, Jeff Zhang wrote: > You need add the following 2 dependencies in the interpreter setting page. > > https://zeppelin.apache.org/docs/0.6.1/interpreter/hive.html#dependencies > > org.apache.hive:hive-jdbc:0.14.0 > org.apache.hadoop:hadoop-common:2.6.0 > > > On Wed, Aug 31, 2016 at 2:39 AM, Abhi Basu <9000r...@gmail.com> wrote: > >> Folks: >> >> Seems like a config issue. >> >> 1. Copied hive-site.xml into /ZEPP_HOME/conf folder >> 2. Added following to config file: >> >> export JAVA_HOME=/./...export HADOOP_CONF_DIR=/etc/hadoop/conf >> >> >> I am using Zeppelin after a while, and looks like Hive interpreter is >> part of JDBC interpreter now. >> Interpreter properties seem to be set correctly: >> PropertyValue >> hive.driver org.apache.hive.jdbc.HiveDriver >> hive.url jdbc:hive2://localhost:1 >> hive.user hiveUser >> hive.password hivePassword >> >> When I run %hive from Zeppelin, I get a hive jdbc driver not found error. >> How do I fix this? Also, how do I configure for Impala within the JDBC >> section of interpreters. >> >> Thanks, >> >> Abhi >> >> -- >> Abhi Basu >> > > > > -- > Best Regards > > Jeff Zhang > -- Abhi Basu
Re: Zeppelin Kerberos error
Hi Jongyoul- I followed the exact same steps for compiling and setting up
the new build from source as 0.5.6 (only difference is, I acquired the
source for latest build using "git clone")
hive-site.xml was copied to conf directory. But, the spark interpreter is
not talking to the hive metastore. Both the 0.5.6 & the latest builds are
running in the same machine. In 0.5.6 when i run the below command, I see
116 databases listed, as per my expectations and I'm able to run my
notebooks built on those databases.
[image: Inline image 1]
Thanks,
Pradeep
On Wed, Aug 31, 2016 at 2:52 AM, Jongyoul Lee wrote:
> Hello,
>
> Do you copy your hive-site.xml in a proper position?
>
> On Wed, Aug 31, 2016 at 3:52 PM, Pradeep Reddy <
> pradeepreddy.a...@gmail.com> wrote:
>
>> nothing obvious. I will stick to 0.5.6 build, until the latest builds
>> stabilize.
>>
>> On Wed, Aug 31, 2016 at 1:39 AM, Jeff Zhang wrote:
>>
>>> Then I guess maybe you are connecting to different database. Why not
>>> using 'z.show(sql("databases"))' to display the databases ? Then you
>>> will get a hint what's going on.
>>>
>>> On Wed, Aug 31, 2016 at 2:36 PM, Pradeep Reddy <
>>> pradeepreddy.a...@gmail.com> wrote:
>>>
Yes...I didn't wish to show the names of the databases that we have in
our data lake on that screen shot. so thats why I chose to display the
count. The latest zeppelin build just shows 1 count which is "default"
database.
Thanks,
Pradeep
On Wed, Aug 31, 2016 at 1:33 AM, Jeff Zhang wrote:
> 116 is the databases count number. Do you expect a list of database ?
> then you need to use 'z.show(sql("databases"))'
>
> On Wed, Aug 31, 2016 at 2:26 PM, Pradeep Reddy <
> pradeepreddy.a...@gmail.com> wrote:
>
>> Here it is Jeff
>>
>> [image: Inline image 1]
>>
>> On Wed, Aug 31, 2016 at 1:24 AM, Jeff Zhang wrote:
>>
>>> Hi Pradeep,
>>>
>>> I don't see the databases on your screenshot (second one for 0.5.6).
>>> I think the output is correct.
>>>
>>>
>>>
>>> On Wed, Aug 31, 2016 at 12:55 PM, Pradeep Reddy <
>>> pradeepreddy.a...@gmail.com> wrote:
>>>
Hi Jeff- I was able to make Kerberos work in 0.5.6 zeppelin build.
It seems like Kerberos not working & spark not able to talk to the
shared
hive meta store are defects in the current build.
On Tue, Aug 30, 2016 at 11:09 PM, Pradeep Reddy <
pradeepreddy.a...@gmail.com> wrote:
> Hi Jeff-
>
> I switched to local mode now, I'm able to summon the implicit
> objects like sc, sqlContext etc., but it doesn't show my databases &
> tables, just shows 1 database "default".
>
> Zeppelin Latest Build
>
> [image: Inline image 3]
>
> Zeppelin 0.5.6, running on the same machine, is able to show my
> databases and tables.
>
> [image: Inline image 4]
>
> On Tue, Aug 30, 2016 at 8:20 PM, Jeff Zhang
> wrote:
>
>> > the spark interpreter is not showing my tables & databases,
>> may be its running in an isolated mode... I'm just getting empty
>> list, so I
>> attempted to do kerberos authentication to workaround that issue, and
>> bumped into this road block.
>>
>> kerberos would not help here, actually I think it would make the
>> problem more complicated. You need to first check the log why you
>> get
>> empty list.
>>
>> On Wed, Aug 31, 2016 at 8:56 AM, Pradeep Reddy <
>> pradeepreddy.a...@gmail.com> wrote:
>>
>>> Jeff- I was also successfully able to run spark shell, after
>>> running kdestroy, with the below command and was able to get to my
>>> hive
>>> tables.
>>>
>>> spark-shell --conf spark.yarn.keytab=$HOME/pradeep.x.alla.keytab
>>> --conf spark.yarn.principal=pradeep.x.alla --deploy-mode client
>>> --master yarn --queue
>>>
>>> On Tue, Aug 30, 2016 at 7:34 PM, Pradeep Reddy <
>>> pradeepreddy.a...@gmail.com> wrote:
>>>
Thanks Jeff..I have always used zeppelin in local mode, but
when I migrated from 0.5.6 to this version, the spark interpreter
is not
showing my tables & databases, may be its running in an isolated
mode...
I'm just getting empty list, so I attempted to do kerberos
authentication
to workaround that issue, and bumped into this road block.
Below is the configuration, I also tested my keytab file and
its working fine.
*Kerberos test:*
$ kdestroy
$ klist
*klist: No credentials cache found (ticket cache
>>>
Re: Is it possible to add widget on zeppelin UI
Canan The idea behind Helium is precisely what you mention. Think of Helium as a way to extend Zeppelin UI. Zeppelin had 2 existing extensibility points: interpreters and notebook storage. Helium is adding the 3rd extensibility point, to Zeppelin the UI. See the demo at https://cwiki.apache.org/confluence/display/ZEPPELIN/Helium+proposal (note the UI widget for word cloud.) On Wed, Aug 31, 2016 at 8:00 PM, canan chen wrote: > > I notice Helium of zeppelin, not sure whether we can add widget on > zeppelin UI if Helium is implemented. Thanks >
Is it possible to add widget on zeppelin UI
I notice Helium of zeppelin, not sure whether we can add widget on zeppelin UI if Helium is implemented. Thanks
Re: Zeppelin Kerberos error
Hello,
Do you copy your hive-site.xml in a proper position?
On Wed, Aug 31, 2016 at 3:52 PM, Pradeep Reddy
wrote:
> nothing obvious. I will stick to 0.5.6 build, until the latest builds
> stabilize.
>
> On Wed, Aug 31, 2016 at 1:39 AM, Jeff Zhang wrote:
>
>> Then I guess maybe you are connecting to different database. Why not
>> using 'z.show(sql("databases"))' to display the databases ? Then you
>> will get a hint what's going on.
>>
>> On Wed, Aug 31, 2016 at 2:36 PM, Pradeep Reddy <
>> pradeepreddy.a...@gmail.com> wrote:
>>
>>> Yes...I didn't wish to show the names of the databases that we have in
>>> our data lake on that screen shot. so thats why I chose to display the
>>> count. The latest zeppelin build just shows 1 count which is "default"
>>> database.
>>>
>>> Thanks,
>>> Pradeep
>>>
>>> On Wed, Aug 31, 2016 at 1:33 AM, Jeff Zhang wrote:
>>>
116 is the databases count number. Do you expect a list of database ?
then you need to use 'z.show(sql("databases"))'
On Wed, Aug 31, 2016 at 2:26 PM, Pradeep Reddy <
pradeepreddy.a...@gmail.com> wrote:
> Here it is Jeff
>
> [image: Inline image 1]
>
> On Wed, Aug 31, 2016 at 1:24 AM, Jeff Zhang wrote:
>
>> Hi Pradeep,
>>
>> I don't see the databases on your screenshot (second one for 0.5.6).
>> I think the output is correct.
>>
>>
>>
>> On Wed, Aug 31, 2016 at 12:55 PM, Pradeep Reddy <
>> pradeepreddy.a...@gmail.com> wrote:
>>
>>> Hi Jeff- I was able to make Kerberos work in 0.5.6 zeppelin build.
>>> It seems like Kerberos not working & spark not able to talk to the
>>> shared
>>> hive meta store are defects in the current build.
>>>
>>> On Tue, Aug 30, 2016 at 11:09 PM, Pradeep Reddy <
>>> pradeepreddy.a...@gmail.com> wrote:
>>>
Hi Jeff-
I switched to local mode now, I'm able to summon the implicit
objects like sc, sqlContext etc., but it doesn't show my databases &
tables, just shows 1 database "default".
Zeppelin Latest Build
[image: Inline image 3]
Zeppelin 0.5.6, running on the same machine, is able to show my
databases and tables.
[image: Inline image 4]
On Tue, Aug 30, 2016 at 8:20 PM, Jeff Zhang
wrote:
> > the spark interpreter is not showing my tables & databases, may
> be its running in an isolated mode... I'm just getting empty list, so
> I
> attempted to do kerberos authentication to workaround that issue, and
> bumped into this road block.
>
> kerberos would not help here, actually I think it would make the
> problem more complicated. You need to first check the log why you get
> empty list.
>
> On Wed, Aug 31, 2016 at 8:56 AM, Pradeep Reddy <
> pradeepreddy.a...@gmail.com> wrote:
>
>> Jeff- I was also successfully able to run spark shell, after
>> running kdestroy, with the below command and was able to get to my
>> hive
>> tables.
>>
>> spark-shell --conf spark.yarn.keytab=$HOME/pradeep.x.alla.keytab
>> --conf spark.yarn.principal=pradeep.x.alla --deploy-mode client
>> --master yarn --queue
>>
>> On Tue, Aug 30, 2016 at 7:34 PM, Pradeep Reddy <
>> pradeepreddy.a...@gmail.com> wrote:
>>
>>> Thanks Jeff..I have always used zeppelin in local mode, but when
>>> I migrated from 0.5.6 to this version, the spark interpreter is not
>>> showing
>>> my tables & databases, may be its running in an isolated mode...
>>> I'm just
>>> getting empty list, so I attempted to do kerberos authentication to
>>> workaround that issue, and bumped into this road block.
>>>
>>> Below is the configuration, I also tested my keytab file and its
>>> working fine.
>>>
>>> *Kerberos test:*
>>> $ kdestroy
>>>
>>> $ klist
>>> *klist: No credentials cache found (ticket cache
>>> FILE:/tmp/krb5cc_12027)*
>>>
>>> $ kinit -kt pradeep_x_alla.keytab -V pradeep.x.alla
>>> *Using default cache: /tmp/krb5cc_12027*
>>> *Using principal: pradeep.x.alla@*
>>> *Using keytab: pradeep_x_alla.keytab*
>>> *Authenticated to Kerberos v5*
>>>
>>> $ klist
>>> *Ticket cache: FILE:/tmp/krb5cc_12027*
>>> *Default principal: pradeep.x.alla@*
>>>
>>> *Valid starting ExpiresService principal*
>>> *08/30/16 20:25:19 08/31/16 06:25:19
>>> krbtgt/@*
>>> *renew until 08/31/16 20:25:19*
>>>
>>> *zeppelin-env.sh*
>>>
>>> export HADOOP_CONF_DIR=/etc/hadoop/conf:/etc/hive/conf
>>> export SPARK_HOME=/o
