Re: Having issues with Hive RuntimeException in runing Zeppelin notebook applicaiton

[Muhammad Rezaul Karim]

 Hi,
I can run the same code in SPARK_HOME/bin/spark-shell. However, it does not 
allow me to execute the SQL command. 

 

On Thursday, November 17, 2016 6:01 PM, moon soo Lee  
wrote:
 

 Are you able to run the same code in SPARK_HOME/bin/spark-shell?
Thanks,moon
On Thu, Nov 17, 2016 at 9:47 AM Muhammad Rezaul Karim  
wrote:

Hi,
Thanks a lot. Yes, I have removed the metastore_db directory too. 
 
 

On Thursday, November 17, 2016 5:38 PM, moon soo Lee  
wrote:
 

 Although "export PATH=$PATH..."  is not really necessary in zeppelin-env.sh, i 
think your configuration looks okay.
Have you tried remove /home/asif/zeppelin-0.6.2-bin-all/metastore_db ?
Thanks,moon
On Wed, Nov 16, 2016 at 6:48 PM Muhammad Rezaul Karim  
wrote:

Hi Moon,
I have set those variables as follows (a partial view of the zeppelin-env.sh 
file). Is this okay? 

#!/bin/bash
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export PATH=$PATH:$JAVA_HOME/bin

export SPARK_HOME=/home/asif/spark-2.0.0-bin-hadoop2.7
export PATH=$PATH:$SPARK_HOME/bin    

# Options read in YARN client mode
export HADOOP_CONF_DIR=/home/asif/Downloads/hadoop-2.7.3/etc 
export PATH=$PATH:$HADOOP_CONF_DIR/hadoop           
 
 

On Thursday, November 17, 2016 1:45 AM, Md. Rezaul Karim 
 wrote:
 

 Hi Moon, No,  I did not set those two environmental variables. I'm still how 
to do that. I have installed Spark on my machine and have set the SPARK_HOME. 
However, do I also need to install and configure the Hadoop. But the error is 
all about the Hive. Am I some how wrong? A sample zeppelin-env.sh file would be 
very helpful. 
On Nov 16, 2016 10:47 PM, moon soo Lee  wrote:

Hi,
It's strange, Do you have SPARK_HOME or HADOOP_CONF_DIR defined in 
conf/zeppelin-env.sh?
You can stop Zeppelin, delete /home/asif/zeppelin-0.6.2-bin-all/metastore_db, 
start Zeppelin and try again.
Thanks,moon
On Tue, Nov 15, 2016 at 4:05 PM Muhammad Rezaul Karim  
wrote:




Hi,I am a new user of Apache Zeppelin and I am running a simple notebook app on 
Zeppelin (version 0.6.2-bin-all) using Scala based on Spark. 

My source code is as follows:
val bankText = 
sc.textFile("/home/rezkar/zeppelin-0.6.2-bin-all/bin/bank-full.csv")case class 
Bank(age:String, job:String, marital:String, education:String, balance:String)
val bank = bankText.map(s=>s.split(";")).map(
s=>Bank(s(0), 
s(1).replaceAll("\"", ""),
s(2).replaceAll("\"", ""),
s(3).replaceAll("\"", ""),
s(5)
)
)
bank.collect()val mydf=bank.toDF()
mydf.registerTempTable("mydf");
Upto bank.collect() it works pretty well. However, I'm getting the following 
error on Ubuntu 14.04 when trying to execute the last two lines of my code:
java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at 
org.apache.spark.sql.hive.client.HiveClientImpl.(HiveClientImpl.scala:171)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at org.apache.spark.sql.SQLImplicits.rddToDatasetHolder(SQLImplicits.scala:163)
... 46 elided
Caused by: java.lang.RuntimeException: Unable to instantiate 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86)
... 70 more
Caused by: java.lang.reflect.InvocationTargetException: 
javax.jdo.JDOFatalDataStoreException: Unable to open a test connection to the 
given database. JDBC url = jdbc:derby:;databaseName=metastore_db;create=true, 
username = APP. Terminating connection pool (set lazyInit to true if you expect 
to start your database after your app). Original Exception: --
at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)

Caused by: java.sql.SQLException: Failed to start database 'metastore_db' with 
class loader 
org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@48f2b61e, see the 
next exception for details.
 at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown 
Source)
 at 
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
 Source)
 ... 136 more
Caused by: java.sql.SQLException: Database at 
/home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible format with 
the current version of the software. The database was created by or upgraded by 
version 10.11.
 at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown 
Source)
 at 
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
 Source)
 at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown 
Source)
 at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
 ... 133 

Re: Having issues with Hive RuntimeException in runing Zeppelin notebook applicaiton

[moon soo Lee]

Are you able to run the same code in SPARK_HOME/bin/spark-shell?

Thanks,
moon

On Thu, Nov 17, 2016 at 9:47 AM Muhammad Rezaul Karim 
wrote:

> Hi,
>
> Thanks a lot. Yes, I have removed the metastore_db directory too.
>
>
>
>
> On Thursday, November 17, 2016 5:38 PM, moon soo Lee 
> wrote:
>
>
> Although "export PATH=$PATH..."  is not really necessary in
> zeppelin-env.sh, i think your configuration looks okay.
>
> Have you tried remove /home/asif/zeppelin-0.6.2-bin-all/metastore_db ?
>
> Thanks,
> moon
>
> On Wed, Nov 16, 2016 at 6:48 PM Muhammad Rezaul Karim <
> reza_cse...@yahoo.com> wrote:
>
> Hi Moon,
>
> I have set those variables as follows (a partial view of the zeppelin-env.sh
> file). Is this okay?
>
> #!/bin/bash
> export JAVA_HOME=/usr/lib/jvm/java-8-oracle
> export PATH=$PATH:$JAVA_HOME/bin
>
> export SPARK_HOME=/home/asif/spark-2.0.0-bin-hadoop2.7
> export PATH=$PATH:$SPARK_HOME/bin
>
> # Options read in YARN client mode
> export HADOOP_CONF_DIR=/home/asif/Downloads/hadoop-2.7.3/etc
> export PATH=$PATH:$HADOOP_CONF_DIR/hadoop
>
>
>
>
> On Thursday, November 17, 2016 1:45 AM, Md. Rezaul Karim <
> reza_cse...@yahoo.com> wrote:
>
>
> Hi Moon,
> No,  I did not set those two environmental variables. I'm still how to do
> that. I have installed Spark on my machine and have set the SPARK_HOME.
> However, do I also need to install and configure the Hadoop. But the error
> is all about the Hive. Am I some how wrong?
> A sample zeppelin-env.sh file would be very helpful.
>
> On Nov 16, 2016 10:47 PM, moon soo Lee  wrote:
>
> Hi,
>
> It's strange,
> Do you have SPARK_HOME or HADOOP_CONF_DIR defined in conf/zeppelin-env.sh?
>
> You can stop Zeppelin, delete /home/asif/zeppelin-0.6.2-bin-all/metastore_db,
> start Zeppelin and try again.
>
> Thanks,
> moon
>
> On Tue, Nov 15, 2016 at 4:05 PM Muhammad Rezaul Karim <
> reza_cse...@yahoo.com> wrote:
>
>
>
>
> Hi,
> I am a new user of Apache Zeppelin and I am running a simple notebook app
> on Zeppelin (version 0.6.2-bin-all) using Scala based on Spark.
>
> My source code is as follows:
>
> val bankText =
> sc.textFile("/home/rezkar/zeppelin-0.6.2-bin-all/bin/bank-full.csv")
> case class Bank(age:String, job:String, marital:String, education:String,
> balance:String)
> val bank = bankText.map(s=>s.split(";")).map(
> s=>Bank(s(0),
> s(1).replaceAll("\"", ""),
> s(2).replaceAll("\"", ""),
> s(3).replaceAll("\"", ""),
> s(5)
> )
> )
> bank.collect()
> val mydf=bank.toDF()
> mydf.registerTempTable("mydf");
>
> Upto bank.collect() it works pretty well. However, I'm getting the
> following error on Ubuntu 14.04 when trying to execute the last two lines
> of my code:
>
> *java.lang.RuntimeException: java.lang.RuntimeException: Unable to
> instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient*
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.(HiveClientImpl.scala:171)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> org.apache.spark.sql.SQLImplicits.rddToDatasetHolder(SQLImplicits.scala:163)
> ... 46 elided
> *Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient*
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86)
> ... 70 more
> Caused by: java.lang.reflect.InvocationTargetException:
> javax.jdo.JDOFatalDataStoreException: Unable to open a test connection to
> the given database. JDBC url =
> jdbc:derby:;databaseName=metastore_db;create=true, username = APP.
> Terminating connection pool (set lazyInit to true if you expect to start
> your database after your app). Original Exception: --
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
>
> *Caused by: java.sql.SQLException: Failed to start database 'metastore_db'
> with class loader
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@48f2b61e, see
> the next exception for details.*
> at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
> Source)
> ... 136 more
> Caused by: java.sql.SQLException: Database at
> /home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible format
> with the current version of the software. The database was created by or
> upgraded by version 10.11.
> at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
> Source)
> at org.apache.derby.impl.jdbc

Re: Having issues with Hive RuntimeException in runing Zeppelin notebook applicaiton

[Muhammad Rezaul Karim]

Hi,
Thanks a lot. Yes, I have removed the metastore_db directory too. 
 
 

On Thursday, November 17, 2016 5:38 PM, moon soo Lee  
wrote:
 

 Although "export PATH=$PATH..."  is not really necessary in zeppelin-env.sh, i 
think your configuration looks okay.
Have you tried remove /home/asif/zeppelin-0.6.2-bin-all/metastore_db ?
Thanks,moon
On Wed, Nov 16, 2016 at 6:48 PM Muhammad Rezaul Karim  
wrote:

Hi Moon,
I have set those variables as follows (a partial view of the zeppelin-env.sh 
file). Is this okay? 

#!/bin/bash
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export PATH=$PATH:$JAVA_HOME/bin

export SPARK_HOME=/home/asif/spark-2.0.0-bin-hadoop2.7
export PATH=$PATH:$SPARK_HOME/bin    

# Options read in YARN client mode
export HADOOP_CONF_DIR=/home/asif/Downloads/hadoop-2.7.3/etc 
export PATH=$PATH:$HADOOP_CONF_DIR/hadoop           
 
 

On Thursday, November 17, 2016 1:45 AM, Md. Rezaul Karim 
 wrote:
 

 Hi Moon, No,  I did not set those two environmental variables. I'm still how 
to do that. I have installed Spark on my machine and have set the SPARK_HOME. 
However, do I also need to install and configure the Hadoop. But the error is 
all about the Hive. Am I some how wrong? A sample zeppelin-env.sh file would be 
very helpful. 
On Nov 16, 2016 10:47 PM, moon soo Lee  wrote:

Hi,
It's strange, Do you have SPARK_HOME or HADOOP_CONF_DIR defined in 
conf/zeppelin-env.sh?
You can stop Zeppelin, delete /home/asif/zeppelin-0.6.2-bin-all/metastore_db, 
start Zeppelin and try again.
Thanks,moon
On Tue, Nov 15, 2016 at 4:05 PM Muhammad Rezaul Karim  
wrote:




Hi,I am a new user of Apache Zeppelin and I am running a simple notebook app on 
Zeppelin (version 0.6.2-bin-all) using Scala based on Spark. 

My source code is as follows:
val bankText = 
sc.textFile("/home/rezkar/zeppelin-0.6.2-bin-all/bin/bank-full.csv")case class 
Bank(age:String, job:String, marital:String, education:String, balance:String)
val bank = bankText.map(s=>s.split(";")).map(
s=>Bank(s(0), 
s(1).replaceAll("\"", ""),
s(2).replaceAll("\"", ""),
s(3).replaceAll("\"", ""),
s(5)
)
)
bank.collect()val mydf=bank.toDF()
mydf.registerTempTable("mydf");
Upto bank.collect() it works pretty well. However, I'm getting the following 
error on Ubuntu 14.04 when trying to execute the last two lines of my code:
java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at 
org.apache.spark.sql.hive.client.HiveClientImpl.(HiveClientImpl.scala:171)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at org.apache.spark.sql.SQLImplicits.rddToDatasetHolder(SQLImplicits.scala:163)
... 46 elided
Caused by: java.lang.RuntimeException: Unable to instantiate 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86)
... 70 more
Caused by: java.lang.reflect.InvocationTargetException: 
javax.jdo.JDOFatalDataStoreException: Unable to open a test connection to the 
given database. JDBC url = jdbc:derby:;databaseName=metastore_db;create=true, 
username = APP. Terminating connection pool (set lazyInit to true if you expect 
to start your database after your app). Original Exception: --
at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)

Caused by: java.sql.SQLException: Failed to start database 'metastore_db' with 
class loader 
org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@48f2b61e, see the 
next exception for details.
 at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown 
Source)
 at 
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
 Source)
 ... 136 more
Caused by: java.sql.SQLException: Database at 
/home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible format with 
the current version of the software. The database was created by or upgraded by 
version 10.11.
 at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown 
Source)
 at 
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
 Source)
 at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown 
Source)
 at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
 ... 133 more
Caused by: ERROR XSLAN: Database at 
/home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible format with 
the current version of the software. The database was created by or upgraded by 
version 10.11.
 at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
--
at sun.reflect.NativeConstructo

Re: Having issues with Hive RuntimeException in runing Zeppelin notebook applicaiton

[moon soo Lee]

Although "export PATH=$PATH..."  is not really necessary in
zeppelin-env.sh, i think your configuration looks okay.

Have you tried remove /home/asif/zeppelin-0.6.2-bin-all/metastore_db ?

Thanks,
moon

On Wed, Nov 16, 2016 at 6:48 PM Muhammad Rezaul Karim 
wrote:

Hi Moon,

I have set those variables as follows (a partial view of the zeppelin-env.sh
file). Is this okay?

#!/bin/bash
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export PATH=$PATH:$JAVA_HOME/bin

export SPARK_HOME=/home/asif/spark-2.0.0-bin-hadoop2.7
export PATH=$PATH:$SPARK_HOME/bin

# Options read in YARN client mode
export HADOOP_CONF_DIR=/home/asif/Downloads/hadoop-2.7.3/etc
export PATH=$PATH:$HADOOP_CONF_DIR/hadoop




On Thursday, November 17, 2016 1:45 AM, Md. Rezaul Karim <
reza_cse...@yahoo.com> wrote:


Hi Moon,
No,  I did not set those two environmental variables. I'm still how to do
that. I have installed Spark on my machine and have set the SPARK_HOME.
However, do I also need to install and configure the Hadoop. But the error
is all about the Hive. Am I some how wrong?
A sample zeppelin-env.sh file would be very helpful.

On Nov 16, 2016 10:47 PM, moon soo Lee  wrote:

Hi,

It's strange,
Do you have SPARK_HOME or HADOOP_CONF_DIR defined in conf/zeppelin-env.sh?

You can stop Zeppelin, delete /home/asif/zeppelin-0.6.2-bin-all/metastore_db,
start Zeppelin and try again.

Thanks,
moon

On Tue, Nov 15, 2016 at 4:05 PM Muhammad Rezaul Karim 
wrote:




Hi,
I am a new user of Apache Zeppelin and I am running a simple notebook app
on Zeppelin (version 0.6.2-bin-all) using Scala based on Spark.

My source code is as follows:

val bankText =
sc.textFile("/home/rezkar/zeppelin-0.6.2-bin-all/bin/bank-full.csv")
case class Bank(age:String, job:String, marital:String, education:String,
balance:String)
val bank = bankText.map(s=>s.split(";")).map(
s=>Bank(s(0),
s(1).replaceAll("\"", ""),
s(2).replaceAll("\"", ""),
s(3).replaceAll("\"", ""),
s(5)
)
)
bank.collect()
val mydf=bank.toDF()
mydf.registerTempTable("mydf");

Upto bank.collect() it works pretty well. However, I'm getting the
following error on Ubuntu 14.04 when trying to execute the last two lines
of my code:

*java.lang.RuntimeException: java.lang.RuntimeException: Unable to
instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient*
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at
org.apache.spark.sql.hive.client.HiveClientImpl.(HiveClientImpl.scala:171)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
org.apache.spark.sql.SQLImplicits.rddToDatasetHolder(SQLImplicits.scala:163)
... 46 elided
*Caused by: java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient*
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86)
... 70 more
Caused by: java.lang.reflect.InvocationTargetException:
javax.jdo.JDOFatalDataStoreException: Unable to open a test connection to
the given database. JDBC url =
jdbc:derby:;databaseName=metastore_db;create=true, username = APP.
Terminating connection pool (set lazyInit to true if you expect to start
your database after your app). Original Exception: --
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)

*Caused by: java.sql.SQLException: Failed to start database 'metastore_db'
with class loader
org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@48f2b61e, see
the next exception for details.*
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
... 136 more
Caused by: java.sql.SQLException: Database at
/home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible format
with the current version of the software. The database was created by or
upgraded by version 10.11.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
... 133 more


*Caused by: ERROR XSLAN: Database at
/home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible format
with the current version of the software. The database was created by or
upgraded by version 10.11. at
org.apache.derby.iapi.error.StandardException.newException(Unknown
Source)--*
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Del

Re: Having issues with Hive RuntimeException in runing Zeppelin notebook applicaiton

[Muhammad Rezaul Karim]

Hi Moon,
I have set those variables as follows (a partial view of the zeppelin-env.sh 
file). Is this okay? 

#!/bin/bash
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export PATH=$PATH:$JAVA_HOME/bin

export SPARK_HOME=/home/asif/spark-2.0.0-bin-hadoop2.7
export PATH=$PATH:$SPARK_HOME/bin    

# Options read in YARN client mode
export HADOOP_CONF_DIR=/home/asif/Downloads/hadoop-2.7.3/etc 
export PATH=$PATH:$HADOOP_CONF_DIR/hadoop           
 
 

On Thursday, November 17, 2016 1:45 AM, Md. Rezaul Karim 
 wrote:
 

 Hi Moon, No,  I did not set those two environmental variables. I'm still how 
to do that. I have installed Spark on my machine and have set the SPARK_HOME. 
However, do I also need to install and configure the Hadoop. But the error is 
all about the Hive. Am I some how wrong? A sample zeppelin-env.sh file would be 
very helpful. 
On Nov 16, 2016 10:47 PM, moon soo Lee  wrote:

Hi,
It's strange, Do you have SPARK_HOME or HADOOP_CONF_DIR defined in 
conf/zeppelin-env.sh?
You can stop Zeppelin, delete /home/asif/zeppelin-0.6.2-bin-all/metastore_db, 
start Zeppelin and try again.
Thanks,moon
On Tue, Nov 15, 2016 at 4:05 PM Muhammad Rezaul Karim  
wrote:




Hi,I am a new user of Apache Zeppelin and I am running a simple notebook app on 
Zeppelin (version 0.6.2-bin-all) using Scala based on Spark. 

My source code is as follows:
val bankText = 
sc.textFile("/home/rezkar/zeppelin-0.6.2-bin-all/bin/bank-full.csv")case class 
Bank(age:String, job:String, marital:String, education:String, balance:String)
val bank = bankText.map(s=>s.split(";")).map(
s=>Bank(s(0), 
s(1).replaceAll("\"", ""),
s(2).replaceAll("\"", ""),
s(3).replaceAll("\"", ""),
s(5)
)
)
bank.collect()val mydf=bank.toDF()
mydf.registerTempTable("mydf");
Upto bank.collect() it works pretty well. However, I'm getting the following 
error on Ubuntu 14.04 when trying to execute the last two lines of my code:
java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at 
org.apache.spark.sql.hive.client.HiveClientImpl.(HiveClientImpl.scala:171)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at org.apache.spark.sql.SQLImplicits.rddToDatasetHolder(SQLImplicits.scala:163)
... 46 elided
Caused by: java.lang.RuntimeException: Unable to instantiate 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86)
... 70 more
Caused by: java.lang.reflect.InvocationTargetException: 
javax.jdo.JDOFatalDataStoreException: Unable to open a test connection to the 
given database. JDBC url = jdbc:derby:;databaseName=metastore_db;create=true, 
username = APP. Terminating connection pool (set lazyInit to true if you expect 
to start your database after your app). Original Exception: --
at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)

Caused by: java.sql.SQLException: Failed to start database 'metastore_db' with 
class loader 
org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@48f2b61e, see the 
next exception for details.
 at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown 
Source)
 at 
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
 Source)
 ... 136 more
Caused by: java.sql.SQLException: Database at 
/home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible format with 
the current version of the software. The database was created by or upgraded by 
version 10.11.
 at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown 
Source)
 at 
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
 Source)
 at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown 
Source)
 at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
 ... 133 more
Caused by: ERROR XSLAN: Database at 
/home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible format with 
the current version of the software. The database was created by or upgraded by 
version 10.11.
 at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
--
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
... 76 mo

Re: Having issues with Hive RuntimeException in runing Zeppelin notebook applicaiton

[Md. Rezaul Karim]

Hi Moon, 
No,  I did not set those two environmental variables. I'm still how to do that. I have installed Spark on my machine and have set the SPARK_HOME. However, do I also need to install and configure the Hadoop. But the error is all about the Hive. Am I some how wrong? 
A sample zeppelin-env.sh file would be very helpful. 
On Nov 16, 2016 10:47 PM, moon soo Lee  wrote:Hi,It's strange, Do you have SPARK_HOME or HADOOP_CONF_DIR defined in conf/zeppelin-env.sh?You can stop Zeppelin, delete /home/asif/zeppelin-0.6.2-bin-all/metastore_db, start Zeppelin and try again.Thanks,moonOn Tue, Nov 15, 2016 at 4:05 PM Muhammad Rezaul Karim  wrote:Hi,

I am a new user of Apache Zeppelin and I am running a simple notebook app on Zeppelin (version 0.6.2-bin-all) using Scala based on Spark. My source code is as follows:val bankText = sc.textFile("/home/rezkar/zeppelin-0.6.2-bin-all/bin/bank-full.csv")

case class Bank(age:String, job:String, marital:String, education:String, balance:String)val bank = bankText.map(s=>s.split(";")).map(s=>Bank(s(0), s(1).replaceAll("\"", ""),s(2).replaceAll("\"", ""),s(3).replaceAll("\"", ""),s(5)))bank.collect()val mydf=bank.toDF()mydf.registerTempTable("mydf");


Upto bank.collect() it works pretty well. However, I'm getting the following error on Ubuntu 14.04 when trying to execute the last two lines of my code:java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClientat org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)at org.apache.spark.sql.hive.client.HiveClientImpl.(HiveClientImpl.scala:171)at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at org.apache.spark.sql.SQLImplicits.rddToDatasetHolder(SQLImplicits.scala:163)... 46 elidedCaused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClientat org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86)... 70 moreCaused by: java.lang.reflect.InvocationTargetException: javax.jdo.JDOFatalDataStoreException: Unable to open a test connection to the given database. JDBC url = "" username = APP. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: --at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)Caused by: java.sql.SQLException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@48f2b61e, see the next exception for details.	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)	... 136 moreCaused by: java.sql.SQLException: Database at /home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible format with the current version of the software. The database was created by or upgraded by version 10.11.	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)	at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)	... 133 moreCaused by: ERROR XSLAN: Database at /home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible format with the current version of the software. The database was created by or upgraded by version 10.11.	at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)--at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)... 76 moreCaused by: javax.jdo.JDOFatalDataStoreException: Unable to open a test connection to the given database. JDBC url = "" username = APP. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: --java.sql.SQLException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@48f2b61e, see the next exception for details.Caused by: java.sql.SQLException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@48f2b61e, see the next exception for details.	at o

Re: Having issues with Hive RuntimeException in runing Zeppelin notebook applicaiton

[Jianfeng (Jeff) Zhang]

>>> at /home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible 
>>> format with the current version of the software. The database was created 
>>> by or upgraded by version 10.11.

Try to delete this folder and rerun it again.


Best Regard,
Jeff Zhang


From: moon soo Lee mailto:m...@apache.org>>
Reply-To: "users@zeppelin.apache.org<mailto:users@zeppelin.apache.org>" 
mailto:users@zeppelin.apache.org>>
Date: Thursday, November 17, 2016 at 6:47 AM
To: "users@zeppelin.apache.org<mailto:users@zeppelin.apache.org>" 
mailto:users@zeppelin.apache.org>>, Muhammad Rezaul 
Karim mailto:reza_cse...@yahoo.com>>
Subject: Re: Having issues with Hive RuntimeException in runing Zeppelin 
notebook applicaiton

Hi,

It's strange,
Do you have SPARK_HOME or HADOOP_CONF_DIR defined in conf/zeppelin-env.sh?

You can stop Zeppelin, delete /home/asif/zeppelin-0.6.2-bin-all/metastore_db, 
start Zeppelin and try again.

Thanks,
moon

On Tue, Nov 15, 2016 at 4:05 PM Muhammad Rezaul Karim 
mailto:reza_cse...@yahoo.com>> wrote:



Hi,
I am a new user of Apache Zeppelin and I am running a simple notebook app on 
Zeppelin (version 0.6.2-bin-all) using Scala based on Spark.

My source code is as follows:

val bankText = 
sc.textFile("/home/rezkar/zeppelin-0.6.2-bin-all/bin/bank-full.csv")
case class Bank(age:String, job:String, marital:String, education:String, 
balance:String)
val bank = bankText.map(s=>s.split(";")).map(
s=>Bank(s(0),
s(1).replaceAll("\"", ""),
s(2).replaceAll("\"", ""),
s(3).replaceAll("\"", ""),
s(5)
)
)
bank.collect()
val mydf=bank.toDF()
mydf.registerTempTable("mydf");

Upto bank.collect() it works pretty well. However, I'm getting the following 
error on Ubuntu 14.04 when trying to execute the last two lines of my code:

java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at 
org.apache.spark.sql.hive.client.HiveClientImpl.(HiveClientImpl.scala:171)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at org.apache.spark.sql.SQLImplicits.rddToDatasetHolder(SQLImplicits.scala:163)
... 46 elided
Caused by: java.lang.RuntimeException: Unable to instantiate 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86)
... 70 more
Caused by: java.lang.reflect.InvocationTargetException: 
javax.jdo.JDOFatalDataStoreException: Unable to open a test connection to the 
given database. JDBC url = jdbc:derby:;databaseName=metastore_db;create=true, 
username = APP. Terminating connection pool (set lazyInit to true if you expect 
to start your database after your app). Original Exception: --
at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)

Caused by: java.sql.SQLException: Failed to start database 'metastore_db' with 
class loader 
org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@48f2b61e, see the 
next exception for details.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown 
Source)
at 
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
 Source)
... 136 more
Caused by: java.sql.SQLException: Database at 
/home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible format with 
the current version of the software. The database was created by or upgraded by 
version 10.11.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown 
Source)
at 
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
 Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown 
Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
... 133 more
Caused by: ERROR XSLAN: Database at 
/home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible format with 
the current version of the software. The database was created by or upgraded by 
version 10.11.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
--
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(M

Re: Having issues with Hive RuntimeException in runing Zeppelin notebook applicaiton

[moon soo Lee]

Hi,

It's strange,
Do you have SPARK_HOME or HADOOP_CONF_DIR defined in conf/zeppelin-env.sh?

You can stop Zeppelin, delete /home/asif/zeppelin-0.6.2-bin-all/metastore_db,
start Zeppelin and try again.

Thanks,
moon

On Tue, Nov 15, 2016 at 4:05 PM Muhammad Rezaul Karim 
wrote:

>
>
>
> Hi,
> I am a new user of Apache Zeppelin and I am running a simple notebook app
> on Zeppelin (version 0.6.2-bin-all) using Scala based on Spark.
>
> My source code is as follows:
>
> val bankText =
> sc.textFile("/home/rezkar/zeppelin-0.6.2-bin-all/bin/bank-full.csv")
> case class Bank(age:String, job:String, marital:String, education:String,
> balance:String)
> val bank = bankText.map(s=>s.split(";")).map(
> s=>Bank(s(0),
> s(1).replaceAll("\"", ""),
> s(2).replaceAll("\"", ""),
> s(3).replaceAll("\"", ""),
> s(5)
> )
> )
> bank.collect()
> val mydf=bank.toDF()
> mydf.registerTempTable("mydf");
>
> Upto bank.collect() it works pretty well. However, I'm getting the
> following error on Ubuntu 14.04 when trying to execute the last two lines
> of my code:
>
> *java.lang.RuntimeException: java.lang.RuntimeException: Unable to
> instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient*
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.(HiveClientImpl.scala:171)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> org.apache.spark.sql.SQLImplicits.rddToDatasetHolder(SQLImplicits.scala:163)
> ... 46 elided
> *Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient*
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86)
> ... 70 more
> Caused by: java.lang.reflect.InvocationTargetException:
> javax.jdo.JDOFatalDataStoreException: Unable to open a test connection to
> the given database. JDBC url =
> jdbc:derby:;databaseName=metastore_db;create=true, username = APP.
> Terminating connection pool (set lazyInit to true if you expect to start
> your database after your app). Original Exception: --
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
>
> *Caused by: java.sql.SQLException: Failed to start database 'metastore_db'
> with class loader
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@48f2b61e, see
> the next exception for details.*
> at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
> Source)
> ... 136 more
> Caused by: java.sql.SQLException: Database at
> /home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible format
> with the current version of the software. The database was created by or
> upgraded by version 10.11.
> at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
> ... 133 more
>
>
> *Caused by: ERROR XSLAN: Database at
> /home/asif/zeppelin-0.6.2-bin-all/metastore_db has an incompatible format
> with the current version of the software. The database was created by or
> upgraded by version 10.11. at
> org.apache.derby.iapi.error.StandardException.newException(Unknown
> Source)--*
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
> ... 76 more
> *Caused by: javax.jdo.JDOFatalDataStoreException: Unable to open a test
> connection to the given database. JDBC url =
> jdbc:derby:;databaseName=metastore_db;create=true, username = APP.
> Terminating connection pool (set lazyInit to true if you expect to start
> your database after your app). Original Exception: --*
> java.sql.SQLException: Failed to start database 'metastore_db' with class
> loader
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@48f2b61e,
> see the next exception for details.
>
>
> *Caused by: java.sql.SQLException: Failed to start database 'metastore_db'
> with class loader
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@48f2b61e, see
> the next exception for details. at
> org.apache.derby.impl.jdbc.SQ