Well. I managed to solve that issue after running my tests on a linux
system instead of windows (which I was originally using). However, now I
have an error when I try to reset the hive context using hc.reset(). It
tries to create a file inside directory /user/my_user_name instead of the
usual linux path /home/my_user_name, which fails.
On Thu, Aug 6, 2015 at 3:12 PM, Cesar Flores <ces...@gmail.com> wrote:
> Well, I try this approach, and still have issues. Apparently TestHive can
> not delete the hive metastore directory. The complete error that I have is:
>
> 15/08/06 15:01:29 ERROR Driver: FAILED: Execution Error, return code 1
> from org.apache.hadoop.hive.ql.exec.DDLTask.
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NullPointerException
> 15/08/06 15:01:29 ERROR TestHive:
> ======================
> HIVE FAILURE OUTPUT
> ======================
> SET spark.sql.test=
> SET
> javax.jdo.option.ConnectionURL=jdbc:derby:;databaseName=C:\cygwin64\tmp\sparkHiveMetastore1376676777991703945;create=true
> SET
> hive.metastore.warehouse.dir=C:\cygwin64\tmp\sparkHiveWarehouse5264564710014125096
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NullPointerException
>
> ======================
> END HIVE FAILURE OUTPUT
> ======================
>
> [error] Uncaught exception when running
> com.dotomi.pipeline.utilitytransformers.SorterTransformerSuite:
> java.lang.ExceptionInInitializerError
> [trace] Stack trace suppressed: run last pipeline/test:testOnly for the
> full output.
> 15/08/06 15:01:29 ERROR Utils: Exception while deleting Spark temp dir:
> C:\cygwin64\tmp\sparkHiveMetastore1376676777991703945
> java.io.IOException: Failed to delete:
> C:\cygwin64\tmp\sparkHiveMetastore1376676777991703945
> at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:932)
> at
> org.apache.spark.util.Utils$$anon$4$$anonfun$run$1$$anonfun$apply$mcV$sp$2.apply(Utils.scala:181)
> at
> org.apache.spark.util.Utils$$anon$4$$anonfun$run$1$$anonfun$apply$mcV$sp$2.apply(Utils.scala:179)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
> at
> org.apache.spark.util.Utils$$anon$4$$anonfun$run$1.apply$mcV$sp(Utils.scala:179)
> at
> org.apache.spark.util.Utils$$anon$4$$anonfun$run$1.apply(Utils.scala:177)
> at
> org.apache.spark.util.Utils$$anon$4$$anonfun$run$1.apply(Utils.scala:177)
> at
> org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
> at org.apache.spark.util.Utils$$anon$4.run(Utils.scala:177)
>
> Any new idea about how to avoid this error? I think the problem may be
> running the tests on sbt, as the created directories are locked until I
> exit the sbt command shell from where I run the tests. Please let me know
> if you have any other suggestion.
>
>
> Thanks
>
> On Mon, Aug 3, 2015 at 5:56 PM, Michael Armbrust <mich...@databricks.com>
> wrote:
>
>> TestHive takes care of creating a temporary directory for each invocation
>> so that multiple test runs won't conflict.
>>
>> On Mon, Aug 3, 2015 at 3:09 PM, Cesar Flores <ces...@gmail.com> wrote:
>>
>>>
>>> We are using a local hive context in order to run unit tests. Our unit
>>> tests runs perfectly fine if we run why by one using sbt as the next
>>> example:
>>>
>>> >sbt test-only com.company.pipeline.scalers.ScalerSuite.scala
>>> >sbt test-only com.company.pipeline.labels.ActiveUsersLabelsSuite.scala
>>>
>>> However, if we try to run them as:
>>>
>>> >sbt test-only com.company.pipeline.*
>>>
>>> we start to run into issues. It appears that the issue is that the hive
>>> context is not properly shutdown after finishing the first test. Does any
>>> one know how to attack this problem? The test part in my build.sbt file
>>> looks like:
>>>
>>> libraryDependencies += "org.scalatest" % "scalatest_2.10" % "2.0" %
>>> "test",
>>> parallelExecution in Test := false,
>>> fork := true,
>>> javaOptions ++= Seq("-Xms512M", "-Xmx2048M", "-XX:MaxPermSize=2048M",
>>> "-XX:+CMSClassUnloadingEnabled")
>>>
>>> We are working under Spark 1.3.0
>>>
>>>
>>> Thanks
>>> --
>>> Cesar Flores
>>>
>>
>>
>
>
> --
> Cesar Flores
>
--
Cesar Flores
