Justin, If you are creating multiple HiveContexts in tests, you need to assign a temporary metastore location for every HiveContext (like what we do at here <https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala#L527-L543>). Otherwise, they all try to connect to the metastore in the current dir (look at metastore_db).
Peter, Do you also have the same use case as Justin (creating multiple HiveContexts in tests)? Can you explain what you meant by "all tests"? I am probably missing some context at here. Thanks, Yin On Thu, May 28, 2015 at 11:28 AM, Peter Rudenko <petro.rude...@gmail.com> wrote: > Also have the same issue - all tests fail because of HiveContext / derby > lock. > > Cause: javax.jdo.JDOFatalDataStoreException: Unable to open a test connection > to the given database. JDBC url = > jdbc:derby:;databaseName=metastore_db;create=true, username = APP. > Terminating connection pool (set lazyInit to true if you expect to start your > database after your app). Original Exception: ------ > [info] java.sql.SQLException: Failed to start database 'metastore_db' with > class loader > org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@8066e0e, see > the next exception for details. > > Also is there build for hadoop2.6? Don’t see it here: > <http://people.apache.org/%7Epwendell/spark-releases/spark-1.4.0-rc2-bin/> > http://people.apache.org/~pwendell/spark-releases/spark-1.4.0-rc2-bin/ > > Thanks, > Peter Rudenko > > On 2015-05-22 22:56, Justin Uang wrote: > > I'm working on one of the Palantir teams using Spark, and here is our > feedback: > > We have encountered three issues when upgrading to spark 1.4.0. I'm not > sure they qualify as a -1, as they come from using non-public APIs and > multiple spark contexts for the purposes of testing, but I do want to bring > them up for awareness =) > > 1. Our UDT was serializing to a StringType, but now strings are > represented internally as UTF8String, so we had to change our UDT to use > UTF8String.apply() and UTF8String.toString() to convert back to String. > 2. createDataFrame when using UDTs used to accept things in the > serialized catalyst form. Now, they're supposed to be in the UDT java class > form (I think this change would've affected us in 1.3.1 already, since we > were in 1.3.0) > 3. derby database lifecycle management issue with HiveContext. We have > been using a SparkContextResource JUnit Rule that we wrote, and it sets up > then tears down a SparkContext and HiveContext between unit test runs > within the same process (possibly the same thread as well). Multiple > contexts are not being used at once. It used to work in 1.3.0, but now when > we try to create the HiveContext for the second unit test, then it > complains with the following exception. I have a feeling it might have > something to do with the Hive object being thread local, and us not > explicitly closing the HiveContext and everything it holds. The full stack > trace is here: > <https://gist.github.com/justinuang/0403d49cdeedf91727cd> > https://gist.github.com/justinuang/0403d49cdeedf91727cd > > Caused by: java.sql.SQLException: Failed to start database 'metastore_db' > with class loader > org.apache.spark.sql.hive.client.IsolatedClientLoader$anon$1@5dea2446, see > the next exception for details. > at > org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) > > > On Wed, May 20, 2015 at 10:35 AM Imran Rashid <iras...@cloudera.com> > wrote: > >> -1 >> >> discovered I accidentally removed master & worker json endpoints, will >> restore >> https://issues.apache.org/jira/browse/SPARK-7760 >> >> On Tue, May 19, 2015 at 11:10 AM, Patrick Wendell < <pwend...@gmail.com> >> pwend...@gmail.com> wrote: >> >>> Please vote on releasing the following candidate as Apache Spark version >>> 1.4.0! >>> >>> The tag to be voted on is v1.4.0-rc1 (commit 777a081): >>> >>> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=777a08166f1fb144146ba32581d4632c3466541e >>> >>> The release files, including signatures, digests, etc. can be found at: >>> http://people.apache.org/~pwendell/spark-1.4.0-rc1/ >>> >>> Release artifacts are signed with the following key: >>> https://people.apache.org/keys/committer/pwendell.asc >>> >>> The staging repository for this release can be found at: >>> https://repository.apache.org/content/repositories/orgapachespark-1092/ >>> >>> The documentation corresponding to this release can be found at: >>> http://people.apache.org/~pwendell/spark-1.4.0-rc1-docs/ >>> >>> Please vote on releasing this package as Apache Spark 1.4.0! >>> >>> The vote is open until Friday, May 22, at 17:03 UTC and passes >>> if a majority of at least 3 +1 PMC votes are cast. >>> >>> [ ] +1 Release this package as Apache Spark 1.4.0 >>> [ ] -1 Do not release this package because ... >>> >>> To learn more about Apache Spark, please see >>> http://spark.apache.org/ >>> >>> == How can I help test this release? == >>> If you are a Spark user, you can help us test this release by >>> taking a Spark 1.3 workload and running on this release candidate, >>> then reporting any regressions. >>> >>> == What justifies a -1 vote for this release? == >>> This vote is happening towards the end of the 1.4 QA period, >>> so -1 votes should only occur for significant regressions from 1.3.1. >>> Bugs already present in 1.3.X, minor regressions, or bugs related >>> to new features will not block this release. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >>> For additional commands, e-mail: <dev-h...@spark.apache.org> >>> dev-h...@spark.apache.org >>> >>> >> >
