Just find this is because below lines in make_distribution.sh doesn't work:
if [ "$SPARK_HIVE" == "true" ]; then cp "$FWDIR"/lib_managed/jars/datanucleus*.jar "$DISTDIR/lib/" fi There is no definition of $SPARK_HIVE in make_distribution.sh. I should set it explicitly. On Wed, Aug 13, 2014 at 1:10 PM, ZHENG, Xu-dong <dong...@gmail.com> wrote: > Hi Cheng, > > I also meet some issues when I try to start ThriftServer based a build > from master branch (I could successfully run it from the branch-1.0-jdbc > branch). Below is my build command: > > ./make-distribution.sh --skip-java-test -Phadoop-2.4 -Phive -Pyarn > -Dyarn.version=2.4.0 -Dhadoop.version=2.4.0 -Phive-thriftserver > > And below are the printed errors: > > ERROR CompositeService: Error starting services HiveServer2 > org.apache.hive.service.ServiceException: Unable to connect to MetaStore! > at org.apache.hive.service.cli.CLIService.start(CLIService.java:85) > at > org.apache.hive.service.CompositeService.start(CompositeService.java:70) > at > org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:73) > at > org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:71) > at > org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:314) > at > org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:73) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: javax.jdo.JDOFatalUserException: Class > org.datanucleus.api.jdo.JDOPersistenceManagerFactory was not found. > NestedThrowables: > java.lang.ClassNotFoundException: > org.datanucleus.api.jdo.JDOPersistenceManagerFactory > at > javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1175) > at > javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808) > at > javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701) > at > org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:275) > at > org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:304) > at > org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:234) > at > org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:209) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) > at > org.apache.hadoop.hive.metastore.RetryingRawStore.<init>(RetryingRawStore.java:64) > at > org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:73) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:415) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:402) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:441) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:326) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:286) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59) > at > org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4060) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:121) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:104) > at org.apache.hive.service.cli.CLIService.start(CLIService.java:82) > ... 11 more > Caused by: java.lang.ClassNotFoundException: > org.datanucleus.api.jdo.JDOPersistenceManagerFactory > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:270) > at javax.jdo.JDOHelper$18.run(JDOHelper.java:2018) > at javax.jdo.JDOHelper$18.run(JDOHelper.java:2016) > at java.security.AccessController.doPrivileged(Native Method) > at javax.jdo.JDOHelper.forName(JDOHelper.java:2015) > at > javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1162) > ... 32 more > 14/08/13 13:08:48 INFO AbstractService: Service:OperationManager is > stopped. > 14/08/13 13:08:48 INFO AbstractService: Service:SessionManager is stopped. > 14/08/13 13:08:48 INFO AbstractService: Service:CLIService is stopped. > 14/08/13 13:08:48 ERROR HiveThriftServer2: Error starting HiveThriftServer2 > org.apache.hive.service.ServiceException: Failed to Start HiveServer2 > at > org.apache.hive.service.CompositeService.start(CompositeService.java:80) > at > org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:73) > at > org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:71) > at > org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:314) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:73) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: org.apache.hive.service.ServiceException: Unable to connect to > MetaStore! > at org.apache.hive.service.cli.CLIService.start(CLIService.java:85) > at > org.apache.hive.service.CompositeService.start(CompositeService.java:70) > ... 10 more > Caused by: javax.jdo.JDOFatalUserException: Class > org.datanucleus.api.jdo.JDOPersistenceManagerFactory was not found. > NestedThrowables: > java.lang.ClassNotFoundException: > org.datanucleus.api.jdo.JDOPersistenceManagerFactory > at > javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1175) > at > javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808) > at > javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701) > at > org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:275) > at > org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:304) > at > org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:234) > at > org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:209) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) > at > org.apache.hadoop.hive.metastore.RetryingRawStore.<init>(RetryingRawStore.java:64) > at > org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:73) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:415) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:402) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:441) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:326) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:286) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59) > at > org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4060) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:121) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:104) > at org.apache.hive.service.cli.CLIService.start(CLIService.java:82) > ... 11 more > Caused by: java.lang.ClassNotFoundException: > org.datanucleus.api.jdo.JDOPersistenceManagerFactory > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:270) > at javax.jdo.JDOHelper$18.run(JDOHelper.java:2018) > at javax.jdo.JDOHelper$18.run(JDOHelper.java:2016) > at java.security.AccessController.doPrivileged(Native Method) > at javax.jdo.JDOHelper.forName(JDOHelper.java:2015) > at > javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1162) > ... 32 more > > > Any suggestions? > > > On Tue, Aug 12, 2014 at 12:47 AM, Cheng Lian <lian.cs....@gmail.com> > wrote: > >> Hi John, the JDBC Thrift server resides in its own build profile and need >> to be enabled explicitly by ./sbt/sbt -Phive-thriftserver assembly. >> >> >> >> On Tue, Aug 5, 2014 at 4:54 AM, John Omernik <j...@omernik.com> wrote: >> >>> I am using spark-1.1.0-SNAPSHOT right now and trying to get familiar >>> with the JDBC thrift server. I have everything compiled correctly, I can >>> access data in spark-shell on yarn from my hive installation. Cached >>> tables, etc all work. >>> >>> When I execute ./sbin/start-thriftserver.sh >>> >>> I get the error below. Shouldn't it just ready my spark-env? I guess I >>> am lost on how to make this work. >>> >>> Thanks1 >>> >>> $ ./start-thriftserver.sh >>> >>> >>> Spark assembly has been built with Hive, including Datanucleus jars on >>> classpath >>> >>> Exception in thread "main" java.lang.ClassNotFoundException: >>> org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 >>> >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >>> >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>> >>> at java.security.AccessController.doPrivileged(Native Method) >>> >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >>> >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >>> >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >>> >>> at java.lang.Class.forName0(Native Method) >>> >>> at java.lang.Class.forName(Class.java:270) >>> >>> at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:311) >>> >>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:73) >>> >>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>> >> >> > > > -- > 郑旭东 > ZHENG, Xu-dong > > -- 郑旭东 ZHENG, Xu-dong
