Well, I try this approach, and still have issues. Apparently TestHive can
not delete the hive metastore directory. The complete error that I have is:
15/08/06 15:01:29 ERROR Driver: FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
15/08/06 15:01:29 ERROR TestHive:
======================
HIVE FAILURE OUTPUT
======================
SET spark.sql.test=
SET
javax.jdo.option.ConnectionURL=jdbc:derby:;databaseName=C:\cygwin64\tmp\sparkHiveMetastore1376676777991703945;create=true
SET
hive.metastore.warehouse.dir=C:\cygwin64\tmp\sparkHiveWarehouse5264564710014125096
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
======================
END HIVE FAILURE OUTPUT
======================
[error] Uncaught exception when running
com.dotomi.pipeline.utilitytransformers.SorterTransformerSuite:
java.lang.ExceptionInInitializerError
[trace] Stack trace suppressed: run last pipeline/test:testOnly for the
full output.
15/08/06 15:01:29 ERROR Utils: Exception while deleting Spark temp dir:
C:\cygwin64\tmp\sparkHiveMetastore1376676777991703945
java.io.IOException: Failed to delete:
C:\cygwin64\tmp\sparkHiveMetastore1376676777991703945
at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:932)
at
org.apache.spark.util.Utils$$anon$4$$anonfun$run$1$$anonfun$apply$mcV$sp$2.apply(Utils.scala:181)
at
org.apache.spark.util.Utils$$anon$4$$anonfun$run$1$$anonfun$apply$mcV$sp$2.apply(Utils.scala:179)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
at
org.apache.spark.util.Utils$$anon$4$$anonfun$run$1.apply$mcV$sp(Utils.scala:179)
at
org.apache.spark.util.Utils$$anon$4$$anonfun$run$1.apply(Utils.scala:177)
at
org.apache.spark.util.Utils$$anon$4$$anonfun$run$1.apply(Utils.scala:177)
at
org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
at org.apache.spark.util.Utils$$anon$4.run(Utils.scala:177)
Any new idea about how to avoid this error? I think the problem may be
running the tests on sbt, as the created directories are locked until I
exit the sbt command shell from where I run the tests. Please let me know
if you have any other suggestion.
Thanks
On Mon, Aug 3, 2015 at 5:56 PM, Michael Armbrust <mich...@databricks.com>
wrote:
> TestHive takes care of creating a temporary directory for each invocation
> so that multiple test runs won't conflict.
>
> On Mon, Aug 3, 2015 at 3:09 PM, Cesar Flores <ces...@gmail.com> wrote:
>
>>
>> We are using a local hive context in order to run unit tests. Our unit
>> tests runs perfectly fine if we run why by one using sbt as the next
>> example:
>>
>> >sbt test-only com.company.pipeline.scalers.ScalerSuite.scala
>> >sbt test-only com.company.pipeline.labels.ActiveUsersLabelsSuite.scala
>>
>> However, if we try to run them as:
>>
>> >sbt test-only com.company.pipeline.*
>>
>> we start to run into issues. It appears that the issue is that the hive
>> context is not properly shutdown after finishing the first test. Does any
>> one know how to attack this problem? The test part in my build.sbt file
>> looks like:
>>
>> libraryDependencies += "org.scalatest" % "scalatest_2.10" % "2.0" %
>> "test",
>> parallelExecution in Test := false,
>> fork := true,
>> javaOptions ++= Seq("-Xms512M", "-Xmx2048M", "-XX:MaxPermSize=2048M",
>> "-XX:+CMSClassUnloadingEnabled")
>>
>> We are working under Spark 1.3.0
>>
>>
>> Thanks
>> --
>> Cesar Flores
>>
>
>
--
Cesar Flores
