Using TestHiveContext/HiveContext in unit tests

[Sahil Sareen]

I'm trying to do this in unit tests:

val sConf = new SparkConf()
  .setAppName("RandomAppName")
  .setMaster("local")
val sc = new SparkContext(sConf)
val sqlContext = new TestHiveContext(sc)  // tried new HiveContext(sc)
as well


But I get this:

*[scalatest] **Exception encountered when invoking run on a nested suite -
java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.metastore.HiveMetaStoreClient *** ABORTED 

*[scalatest]   java.lang.RuntimeException: java.lang.RuntimeException:
Unable to instantiate
org.apache.hadoop.hive.metastore.HiveMetaStoreClient*[scalatest]
  at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
[scalatest]   at
org.apache.spark.sql.hive.client.ClientWrapper.(ClientWrapper.scala:120)
[scalatest]   at
org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:163)
[scalatest]   at
org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:161)
[scalatest]   at
org.apache.spark.sql.hive.HiveContext.(HiveContext.scala:168)
[scalatest]   at
org.apache.spark.sql.hive.test.TestHiveContext.(TestHive.scala:72)
[scalatest]   at mypackage.NewHiveTest.beforeAll(NewHiveTest.scala:48)
[scalatest]   at
org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAfterAll.scala:187)
[scalatest]   at mypackage.NewHiveTest.beforeAll(NewHiveTest.scala:35)
[scalatest]   at
org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:253)
[scalatest]   at mypackage.NewHiveTest.run(NewHiveTest.scala:35)
[scalatest]   at
org.scalatest.Suite$class.callExecuteOnSuite$1(Suite.scala:1491)

The code works perfectly when I run using spark-submit, but not in unit
tests. Any inputs??

-Sahil

Re: Using TestHiveContext/HiveContext in unit tests

[Michael Armbrust]

Just use TestHive.  Its a global singlton that you can share across test
cases.  It has a reset function if you want to clear out the state at the
begining of a test.

On Fri, Dec 11, 2015 at 2:06 AM, Sahil Sareen  wrote:

> I'm trying to do this in unit tests:
>
> val sConf = new SparkConf()
>   .setAppName("RandomAppName")
>   .setMaster("local")
> val sc = new SparkContext(sConf)
> val sqlContext = new TestHiveContext(sc)  // tried new
> HiveContext(sc) as well
>
>
> But I get this:
>
> *[scalatest] **Exception encountered when invoking run on a nested suite
> - java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient *** ABORTED 
>
> *[scalatest]   java.lang.RuntimeException: java.lang.RuntimeException:
> Unable to instantiate 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient*[scalatest]
>   at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
> [scalatest]   at
> org.apache.spark.sql.hive.client.ClientWrapper.(ClientWrapper.scala:120)
> [scalatest]   at
> org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:163)
> [scalatest]   at
> org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:161)
> [scalatest]   at
> org.apache.spark.sql.hive.HiveContext.(HiveContext.scala:168)
> [scalatest]   at
> org.apache.spark.sql.hive.test.TestHiveContext.(TestHive.scala:72)
> [scalatest]   at mypackage.NewHiveTest.beforeAll(NewHiveTest.scala:48)
> [scalatest]   at
> org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAfterAll.scala:187)
> [scalatest]   at mypackage.NewHiveTest.beforeAll(NewHiveTest.scala:35)
> [scalatest]   at
> org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:253)
> [scalatest]   at mypackage.NewHiveTest.run(NewHiveTest.scala:35)
> [scalatest]   at
> org.scalatest.Suite$class.callExecuteOnSuite$1(Suite.scala:1491)
>
> The code works perfectly when I run using spark-submit, but not in unit
> tests. Any inputs??
>
> -Sahil
>