Unsubscribe
Re: Error communicating with metastore
Thanks for all who replied. After several retries in vain, I felt the issue might be because of version mismatch. So I took Hadoop 2.7.1, Spark 1.3.1 and Hive 1.2.1. Tried with these combination and I could successfully test the hive transactions without any issue. Regards, Sarath. On Fri, Aug 7, 2015 at 10:32 PM, Alan Gates alanfga...@gmail.com wrote: What version of Hive did you say you were using? In 0.14 we switched to make sure all jdbc connections are either serializable or read_committed, yet the error message below seems to indicate you're seeing a connection request that doesn't match this. The only other thing I know to try i switching the JDBC pool provider. By default Hive uses bonecp (which is what you're using according to the logs). I have seen issues with Oracle 12 and bonecp that went away when you switch to using dbcp. You can try this by setting datanucleus.connectionPoolingType to dbcp on your thrift metastore and then restarting the thrift metastore. Alan. Sarath Chandra sarathchandra.jos...@algofusiontech.com August 7, 2015 at 6:13 Thanks Eugene, Alan. @Alan, As suggested checked the logs, here is what I found - - On starting metastore server, I'm seeing following messages in the log file - *2015-08-07 18:32:56,678 ERROR [Thread-7]: compactor.Initiator (Initiator.java:run(134)) - Caught an exception in the main loop of compactor initiator, exiting MetaException(message:Unable to get jdbc connection from pool, READ_COMMITTED and SERIALIZABLE are the only valid transaction levels)* *at org.apache.hadoop.hive.metastore.txn.TxnHandler.getDbConn(TxnHandler.java:811)* *at org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.revokeFromLocalWorkers(CompactionTxnHandler.java:443)* *at org.apache.hadoop.hive.ql.txn.compactor.Initiator.recoverFailedCompactions(Initiator.java:147)* *at org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:64)* - On bringing up the hive shell, I get the following messages - tion - enable connectionWatch for additional debugging assistance or set disableConnectionTracking to true to disable this feature entirely. 2015-08-07 18:38:51,614 WARN [org.spark-project.guava.common.base.internal.Finalizer]: bonecp.ConnectionPartition (ConnectionPartition.java:finalizeReferent(162)) - BoneCP detected an unclosed connection and will now attempt to close it for you. You should be closing this connection in your application - enable connectionWatch for additional debugging assistance or set disableConnectionTracking to true to disable this feature entirely. 2015-08-07 18:38:51,768 DEBUG [pool-3-thread-1]: metastore.ObjectStore (ObjectStore.java:debugLog(6435)) - Commit transaction: count = 0, isactive true at: org.apache.hadoop.hive.metastore.ObjectStore.getFunctions(ObjectStore.java:6657) - On firing show tables command, I get the following messages in the log file - 2015-08-07 18:41:02,511 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(297)) - Trying to connect to metastore with URI thrift://sarath:9083 2015-08-07 18:41:02,511 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(385)) - Connected to metastore. 2015-08-07 18:41:22,549 ERROR [main]: ql.Driver (SessionState.java:printError(545)) - FAILED: Error in determing valid transactions: Error communicating with the metastore org.apache.hadoop.hive.ql.lockmgr.LockException: Error communicating with the metastore at org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.getValidTxns(DbTxnManager.java:281) at org.apache.hadoop.hive.ql.Driver.recordValidTxns(Driver.java:842) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1036) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129
Re: Error communicating with metastore
Thanks Sharath for your reply. I'm specifying these parameters already. I've followed the documentation at https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions But it still throws me this error On Mon, Aug 3, 2015 at 8:45 PM, Sharath Punreddy srpunre...@gmail.com wrote: Here is another one: http://stackoverflow.com/questions/28867368/hive-transactions-are-crashing On Mon, Aug 3, 2015 at 10:13 AM, Sharath Punreddy srpunre...@gmail.com wrote: Check out: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.1/bk_dataintegration/content/ch_using-hive-transactions.html You need to specify the following properties as well: - hive.txn.driver - hive.compactor.initiator.on - hive.compactor.worker.threads On Mon, Aug 3, 2015 at 8:29 AM, Sarath Chandra sarathchandra.jos...@algofusiontech.com wrote: Hi All, Earlier I was using hive 0.13.0 and now trying to migrate to latest version to utilize the transaction support introduced from hive 0.14.0. I downloaded hive 1.2.1, created a metastore in oracle database and provided all the required configuration parameters in conf/hive-site.xml to enable transactions. For the parameter hive.txn.manager given the value org.apache.hadoop.hive.ql.lockmgr.DbTxnManager. From the hive prompt when I fire the command show tables; I'm getting the below exception - *FAILED: Error in determining valid transactions: Error communicating with the metastore* But if disable the hive.txn.manager parameter in hive-site.xml then the command works fine. Is there anything else to be configured which I'm missing? Thanks Regards, Sarath. -- *Sharath Punreddy* *Phone:626-470-7867 626-470-7867* -- *Sharath Punreddy* *Phone:626-470-7867*
Re: Error communicating with metastore
Thanks Alan. Yes I've run metastore scripts for oracle instance. Infact I've removed my previous metastore and created a fresh one by running the schema creation script for 1.2.1. I've looked into the new schema and able to see the table TXNS. I've also removed the hdfs location /user/hive/warehouse and created a fresh one. But still I'm facing this issue. On Mon, Aug 3, 2015 at 8:59 PM, Alan Gates alanfga...@gmail.com wrote: Did you run the hive metastore upgrade scripts for your oracle instance? This error message usually means the transaction related tables have not been created in your database. Somewhere in your distribution there should be a set of upgrade scripts. Look for scripts of the form: scripts/metastore/upgrade/oracle/upgrade-0.13.0-to-0.14.0.oracle.sql You'll want to run all of the ones from 0.13 to 1.2 (0.13-0.14, 0.14-1.1, 1.1-1.2). The 0.13-0.14 scripts assume that you added the transaction tables as part of upgrading to Hive 0.13. If you did not you will need to first run hive-txn-schema-0.13.0.oracle.sql which will create the initial transaction tables. You can determine whether this was done by looking for a table named TXNS in the hive schema on your Oracle db. Alan. Sarath Chandra sarathchandra.jos...@algofusiontech.com August 3, 2015 at 6:29 Hi All, Earlier I was using hive 0.13.0 and now trying to migrate to latest version to utilize the transaction support introduced from hive 0.14.0. I downloaded hive 1.2.1, created a metastore in oracle database and provided all the required configuration parameters in conf/hive-site.xml to enable transactions. For the parameter hive.txn.manager given the value org.apache.hadoop.hive.ql.lockmgr.DbTxnManager. From the hive prompt when I fire the command show tables; I'm getting the below exception - *FAILED: Error in determining valid transactions: Error communicating with the metastore* But if disable the hive.txn.manager parameter in hive-site.xml then the command works fine. Is there anything else to be configured which I'm missing? Thanks Regards, Sarath.
Error communicating with metastore
Hi All, Earlier I was using hive 0.13.0 and now trying to migrate to latest version to utilize the transaction support introduced from hive 0.14.0. I downloaded hive 1.2.1, created a metastore in oracle database and provided all the required configuration parameters in conf/hive-site.xml to enable transactions. For the parameter hive.txn.manager given the value org.apache.hadoop.hive.ql.lockmgr.DbTxnManager. From the hive prompt when I fire the command show tables; I'm getting the below exception - *FAILED: Error in determining valid transactions: Error communicating with the metastore* But if disable the hive.txn.manager parameter in hive-site.xml then the command works fine. Is there anything else to be configured which I'm missing? Thanks Regards, Sarath.
Case sensitiveness of hive partition column
Hi All, I've created a external table in hive as - create table table1(...) partitioned by(CONTEXT_ID string) row format delimited fields terminated by '\\;' location '/user/sarath/table1/'; Then I created a HDFS folder as '/user/sarath/table1/CONTEXT_ID=A123' and copied the data file to this location. I altered the table from hive shell to add this partition as - alter table table1 add partition(CONTEXT_ID='A123'); When I tried select * from table1 I was not getting any data. I searched several blogs and did several trials. Then I figured out that if I change the case of partition column from CONTEXT_ID to context_id and create HDFS folder with the same case then it works. I was of opinion that hive column names are case insensitive in Hive. Is there something I'm missing? Regards, Sarath.
Re: Case sensitiveness of hive partition column
Thanks Nitin. I'm using 0.13.1 version. I think they documentation needs to be updated to highlight this so that beginner users like me doesn't need to struggle to figure it out. On Tue, May 12, 2015 at 5:09 PM, nitinpathakala . nitinpathak...@gmail.com wrote: Hi Sarath, Hive columns are case sensitive.I've noticed it in hive 0.14 version.I remember column name was insensitive for earlier versions. Regards, Nitin On May 12, 2015 5:03 PM, Sarath Chandra sarathchandra.jos...@algofusiontech.com wrote: Hi All, I've created a external table in hive as - create table table1(...) partitioned by(CONTEXT_ID string) row format delimited fields terminated by '\\;' location '/user/sarath/table1/'; Then I created a HDFS folder as '/user/sarath/table1/CONTEXT_ID=A123' and copied the data file to this location. I altered the table from hive shell to add this partition as - alter table table1 add partition(CONTEXT_ID='A123'); When I tried select * from table1 I was not getting any data. I searched several blogs and did several trials. Then I figured out that if I change the case of partition column from CONTEXT_ID to context_id and create HDFS folder with the same case then it works. I was of opinion that hive column names are case insensitive in Hive. Is there something I'm missing? Regards, Sarath.
Re: Issue while running Hive 0.13
Thanks Jason. It worked. There was a different version of SLF4J libraries existing in $HADOOP_HOME/lib. Once I synced both the libraries, it started working. On Fri, Jul 11, 2014 at 11:25 PM, Jason Dere jd...@hortonworks.com wrote: Looking at that error online, I see http://slf4j.org/faq.html#compatibility Maybe try to find what version of the slf libraries you have installed (in hadoop? hive?), and try updating to later version. On Jul 10, 2014, at 9:57 PM, Sarath Chandra sarathchandra.jos...@algofusiontech.com wrote: I'm using Hadoop 1.0.4. Suspecting some compatibility issues I moved from Hive 0.13 to Hive 0.12. But the exceptions related to SL4J still persist. Unable to move forward with hive to finalize a critical product design. Can somebody please help me? On Wed, Jul 9, 2014 at 11:25 AM, Sarath Chandra sarathchandra.jos...@algofusiontech.com wrote: Thanks Deepesh. To use hive with embedded derby mode, I have put the below configuration in hive-site.xml. As suggested on the net, I ran schematool -dbType derby -initSchema and it created $HIVE_HOME/metastore_db folder. Then as suggested by you, I ran hive --service metastore. Strangely I'm getting exceptions related to SL4J -- *java.lang.IllegalAccessError: tried to access field org.slf4j.impl.StaticLoggerBinder.SINGLETON from class org.slf4j.LoggerFactory* Is there anything more to configure before starting? *hive-site.xml* configuration property namejavax.jdo.option.ConnectionURL/name valuejdbc:derby:;databaseName=metastore_db;create=true/value /property property namehive.metastore.warehouse.dir/name value/user/hive/warehouse/value /property property namehive.exec.scratchdir/name value/tmp/hduser/value /property /configuration On Tue, Jul 8, 2014 at 11:20 PM, D K deepe...@gmail.com wrote: Did you start the Hive Metastore? You can start that by running hive --service metastore On Tue, Jul 8, 2014 at 5:27 AM, Sarath Chandra sarathchandra.jos...@algofusiontech.com wrote: Thanks Santhosh. So before going to launch hive shell, we need to start hive server is what I understand. I tried starting hive server by running ./bin/hiveserver2. It just prompts Starting HiveServer2 and keeps waiting. Nothing is happening even after waiting for several minutes. On Tue, Jul 8, 2014 at 4:16 PM, Santhosh Thomas santhosh.tho...@yahoo.com wrote: how did you start hive? Use hive-server2 -- *From:* Sarath Chandra sarathchandra.jos...@algofusiontech.com *To:* user@hive.apache.org *Sent:* Tuesday, July 8, 2014 4:02 PM *Subject:* Issue while running Hive 0.13 Hi, I'm a newbie to Hive. Facing an issue while installing hive stable version (0.13). I downloaded the tar file from the site ( apache-hive-0.13.1-bin.tar.gz http://apache.cs.utah.edu/hive/hive-0.13.1/apache-hive-0.13.1-bin.tar.gz) and followed the instructions given on Hive Wiki https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-InstallationandConfiguration. On running the hive command to get to the hive shell, I'm getting the below exception. Request for a help in this regard. What am I missing? Is there any further configuration to be done? Logging initialized using configuration in file:/usr/local/hive-0.13.1/conf/hive-log4j.properties Exception in thread main java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(RetryingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340) ... 7 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39
Re: Issue while running Hive 0.13
I'm using Hadoop 1.0.4. Suspecting some compatibility issues I moved from Hive 0.13 to Hive 0.12. But the exceptions related to SL4J still persist. Unable to move forward with hive to finalize a critical product design. Can somebody please help me? On Wed, Jul 9, 2014 at 11:25 AM, Sarath Chandra sarathchandra.jos...@algofusiontech.com wrote: Thanks Deepesh. To use hive with embedded derby mode, I have put the below configuration in hive-site.xml. As suggested on the net, I ran schematool -dbType derby -initSchema and it created $HIVE_HOME/metastore_db folder. Then as suggested by you, I ran hive --service metastore. Strangely I'm getting exceptions related to SL4J -- *java.lang.IllegalAccessError: tried to access field org.slf4j.impl.StaticLoggerBinder.SINGLETON from class org.slf4j.LoggerFactory* Is there anything more to configure before starting? *hive-site.xml* configuration property namejavax.jdo.option.ConnectionURL/name valuejdbc:derby:;databaseName=metastore_db;create=true/value /property property namehive.metastore.warehouse.dir/name value/user/hive/warehouse/value /property property namehive.exec.scratchdir/name value/tmp/hduser/value /property /configuration On Tue, Jul 8, 2014 at 11:20 PM, D K deepe...@gmail.com wrote: Did you start the Hive Metastore? You can start that by running hive --service metastore On Tue, Jul 8, 2014 at 5:27 AM, Sarath Chandra sarathchandra.jos...@algofusiontech.com wrote: Thanks Santhosh. So before going to launch hive shell, we need to start hive server is what I understand. I tried starting hive server by running ./bin/hiveserver2. It just prompts Starting HiveServer2 and keeps waiting. Nothing is happening even after waiting for several minutes. On Tue, Jul 8, 2014 at 4:16 PM, Santhosh Thomas santhosh.tho...@yahoo.com wrote: how did you start hive? Use hive-server2 -- *From:* Sarath Chandra sarathchandra.jos...@algofusiontech.com *To:* user@hive.apache.org *Sent:* Tuesday, July 8, 2014 4:02 PM *Subject:* Issue while running Hive 0.13 Hi, I'm a newbie to Hive. Facing an issue while installing hive stable version (0.13). I downloaded the tar file from the site ( apache-hive-0.13.1-bin.tar.gz http://apache.cs.utah.edu/hive/hive-0.13.1/apache-hive-0.13.1-bin.tar.gz) and followed the instructions given on Hive Wiki https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-InstallationandConfiguration. On running the hive command to get to the hive shell, I'm getting the below exception. Request for a help in this regard. What am I missing? Is there any further configuration to be done? Logging initialized using configuration in file:/usr/local/hive-0.13.1/conf/hive-log4j.properties Exception in thread main java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(RetryingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340) ... 7 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410) ... 12 more Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory NestedThrowables: java.lang.reflect.InvocationTargetException at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:587
Issue while running Hive 0.13
Hi, I'm a newbie to Hive. Facing an issue while installing hive stable version (0.13). I downloaded the tar file from the site ( apache-hive-0.13.1-bin.tar.gz http://apache.cs.utah.edu/hive/hive-0.13.1/apache-hive-0.13.1-bin.tar.gz) and followed the instructions given on Hive Wiki https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-InstallationandConfiguration. On running the hive command to get to the hive shell, I'm getting the below exception. Request for a help in this regard. What am I missing? Is there any further configuration to be done? Logging initialized using configuration in file:/usr/local/hive-0.13.1/conf/hive-log4j.properties Exception in thread main java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(RetryingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340) ... 7 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410) ... 12 more Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory NestedThrowables: java.lang.reflect.InvocationTargetException at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:587) at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:788) at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333) at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965) at java.security.AccessController.doPrivileged(Native Method) at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960) at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166) at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808) at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701) at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:310) at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:339) at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:248) at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:223) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.hive.metastore.RawStoreProxy.init(RawStoreProxy.java:58) at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:497) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:475) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:356) at
Re: Issue while running Hive 0.13
Thanks Santhosh. So before going to launch hive shell, we need to start hive server is what I understand. I tried starting hive server by running ./bin/hiveserver2. It just prompts Starting HiveServer2 and keeps waiting. Nothing is happening even after waiting for several minutes. On Tue, Jul 8, 2014 at 4:16 PM, Santhosh Thomas santhosh.tho...@yahoo.com wrote: how did you start hive? Use hive-server2 -- *From:* Sarath Chandra sarathchandra.jos...@algofusiontech.com *To:* user@hive.apache.org *Sent:* Tuesday, July 8, 2014 4:02 PM *Subject:* Issue while running Hive 0.13 Hi, I'm a newbie to Hive. Facing an issue while installing hive stable version (0.13). I downloaded the tar file from the site ( apache-hive-0.13.1-bin.tar.gz http://apache.cs.utah.edu/hive/hive-0.13.1/apache-hive-0.13.1-bin.tar.gz) and followed the instructions given on Hive Wiki https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-InstallationandConfiguration. On running the hive command to get to the hive shell, I'm getting the below exception. Request for a help in this regard. What am I missing? Is there any further configuration to be done? Logging initialized using configuration in file:/usr/local/hive-0.13.1/conf/hive-log4j.properties Exception in thread main java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(RetryingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340) ... 7 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410) ... 12 more Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory NestedThrowables: java.lang.reflect.InvocationTargetException at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:587) at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:788) at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333) at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965) at java.security.AccessController.doPrivileged(Native Method) at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960) at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166) at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808) at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701) at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:310) at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:339) at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:248) at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:223) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62
Re: Issue while running Hive 0.13
Thanks Deepesh. To use hive with embedded derby mode, I have put the below configuration in hive-site.xml. As suggested on the net, I ran schematool -dbType derby -initSchema and it created $HIVE_HOME/metastore_db folder. Then as suggested by you, I ran hive --service metastore. Strangely I'm getting exceptions related to SL4J -- *java.lang.IllegalAccessError: tried to access field org.slf4j.impl.StaticLoggerBinder.SINGLETON from class org.slf4j.LoggerFactory* Is there anything more to configure before starting? *hive-site.xml* configuration property namejavax.jdo.option.ConnectionURL/name valuejdbc:derby:;databaseName=metastore_db;create=true/value /property property namehive.metastore.warehouse.dir/name value/user/hive/warehouse/value /property property namehive.exec.scratchdir/name value/tmp/hduser/value /property /configuration On Tue, Jul 8, 2014 at 11:20 PM, D K deepe...@gmail.com wrote: Did you start the Hive Metastore? You can start that by running hive --service metastore On Tue, Jul 8, 2014 at 5:27 AM, Sarath Chandra sarathchandra.jos...@algofusiontech.com wrote: Thanks Santhosh. So before going to launch hive shell, we need to start hive server is what I understand. I tried starting hive server by running ./bin/hiveserver2. It just prompts Starting HiveServer2 and keeps waiting. Nothing is happening even after waiting for several minutes. On Tue, Jul 8, 2014 at 4:16 PM, Santhosh Thomas santhosh.tho...@yahoo.com wrote: how did you start hive? Use hive-server2 -- *From:* Sarath Chandra sarathchandra.jos...@algofusiontech.com *To:* user@hive.apache.org *Sent:* Tuesday, July 8, 2014 4:02 PM *Subject:* Issue while running Hive 0.13 Hi, I'm a newbie to Hive. Facing an issue while installing hive stable version (0.13). I downloaded the tar file from the site ( apache-hive-0.13.1-bin.tar.gz http://apache.cs.utah.edu/hive/hive-0.13.1/apache-hive-0.13.1-bin.tar.gz) and followed the instructions given on Hive Wiki https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-InstallationandConfiguration. On running the hive command to get to the hive shell, I'm getting the below exception. Request for a help in this regard. What am I missing? Is there any further configuration to be done? Logging initialized using configuration in file:/usr/local/hive-0.13.1/conf/hive-log4j.properties Exception in thread main java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(RetryingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340) ... 7 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410) ... 12 more Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory NestedThrowables: java.lang.reflect.InvocationTargetException at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:587) at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:788) at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333) at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202
Re: Issue while using hive 0.9
Is there any solution for my issue? Otherwise I will quit my hive exploration and move on with other product for my requirement. I thought of giving it a try instead of Pig as I was not comfortable in writing complex data mining logic using Pig. If there is no other way I can get Hive on my machine then looks like I don't have enough choice. On Tue, Sep 18, 2012 at 11:21 AM, Sarath sarathchandra.jos...@algofusiontech.com wrote: No I don't see any antlr jars in the output. Infact not even the default version that exists in the hive lib folder. Do I need to explicitly mention in the classpath? On Tuesday 18 September 2012 11:16 AM, Ashutosh Chauhan wrote: Do ps aux | grep hive n check the classpath of hive process. See if you can spot antlr jar versions in it. On Mon, Sep 17, 2012 at 10:40 PM, Sarath sarathchandra.jos...@algofusiontech.com wrote: Thanks for the quick reply. Tried it. But same issue. Pointed system to JDK 1.6 (update 32). Changed hadoop conf to use this JDK. java -version prints the 1.6 version. Replaced antlr JAR with the same version as it came in hive binary (3.0.1). Anything else I have to do? Do I need to make any changes in hive-conf folder? On Tuesday 18 September 2012 10:35 AM, Ashutosh Chauhan wrote: Only thing which is different then usual setups is jdk1.7 Is it possible for you to test with jdk1.6 ? Ashutosh On Mon, Sep 17, 2012 at 10:02 PM, Sarath Chandra sarathchandra.jos...@algofusiontech.com wrote: @Ashutosh I haven't built my hive from sources. I just downloaded the binaries from the site. The downloaded tar has come with 3.0.1 version of antlr. I didn't do any changes. Just set the environment variables and invoked hive and got this error. I checked out sources and tried to build hive as directed in hive wiki. But the build failed. @Rekha Downloaded 3.4.1 version from the site mentioned and tried again. No luck. Same issue. Just wondering now, how is it working for others out there. What am I missing? Am I doing something wrong? Is this right combination - jdk 1.7 / hadoop 1.0.3 / hive 0.9.0? Regards, Sarath. On Tue, Sep 18, 2012 at 9:00 AM, Joshi, Rekha rekha_jo...@intuit.comwrote: Hi Sarath, Sorry for delay – I don't see this mail folder all the time. Anyhow do try with exactly antler 3.4.1 – available at http://www.antlr.org/wiki/display/ANTLR3/ANTLR+3+Wiki+Home Even antler 3.0.1 might work. Thanks Rekha From: Sarath sarathchandra.jos...@algofusiontech.com Reply-To: user@hive.apache.org Date: Mon, 17 Sep 2012 17:10:01 +0530 To: user@hive.apache.org Subject: Re: Issue while using hive 0.9 From ANTLR site downloaded the latest JAR - antlr-3.4-complete.jar. Copied this to lib folder and tried again. Same issue persists. Thanks, Sarath. On Monday 17 September 2012 04:42 PM, Joshi, Rekha wrote: Try with antlr 3.4.1 jar? Thanks Rekha From: Sarath sarathchandra.jos...@algofusiontech.com Reply-To: user@hive.apache.org Date: Mon, 17 Sep 2012 16:24:02 +0530 To: user@hive.apache.org Subject: Issue while using hive 0.9 Hi, I'm new to hive. As first step downloaded version 0.9 and upacked it. Set the HIVE_HOME to the unpacked folder. I already have hadoop 1.0.3 installed and its configuration is in classpath. Then issued the hive command and got the hive prompt. After this whatever I try to execute, results in the below error - *Exception in thread main java.lang.NoSuchFieldError: type at org.apache.hadoop.hive.ql.parse.HiveLexer.mKW_CREATE(HiveLexer.java:1602) at org.apache.hadoop.hive.ql.parse.HiveLexer.mTokens(HiveLexer.java:6380) at org.antlr.runtime.Lexer.nextToken(Lexer.java:89) at org.antlr.runtime.BufferedTokenStream.fetch(BufferedTokenStream.java:133) at org.antlr.runtime.BufferedTokenStream.sync(BufferedTokenStream.java:127) at org.antlr.runtime.CommonTokenStream.setup(CommonTokenStream.java:132) at org.antlr.runtime.CommonTokenStream.LT (CommonTokenStream.java:91) at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:547) at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:438) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:416) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:689) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:557) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke
Re: Issue while using hive 0.9
@Ashutosh I haven't built my hive from sources. I just downloaded the binaries from the site. The downloaded tar has come with 3.0.1 version of antlr. I didn't do any changes. Just set the environment variables and invoked hive and got this error. I checked out sources and tried to build hive as directed in hive wiki. But the build failed. @Rekha Downloaded 3.4.1 version from the site mentioned and tried again. No luck. Same issue. Just wondering now, how is it working for others out there. What am I missing? Am I doing something wrong? Is this right combination - jdk 1.7 / hadoop 1.0.3 / hive 0.9.0? Regards, Sarath. On Tue, Sep 18, 2012 at 9:00 AM, Joshi, Rekha rekha_jo...@intuit.comwrote: Hi Sarath, Sorry for delay – I don't see this mail folder all the time. Anyhow do try with exactly antler 3.4.1 – available at http://www.antlr.org/wiki/display/ANTLR3/ANTLR+3+Wiki+Home Even antler 3.0.1 might work. Thanks Rekha From: Sarath sarathchandra.jos...@algofusiontech.com Reply-To: user@hive.apache.org Date: Mon, 17 Sep 2012 17:10:01 +0530 To: user@hive.apache.org Subject: Re: Issue while using hive 0.9 From ANTLR site downloaded the latest JAR - antlr-3.4-complete.jar. Copied this to lib folder and tried again. Same issue persists. Thanks, Sarath. On Monday 17 September 2012 04:42 PM, Joshi, Rekha wrote: Try with antlr 3.4.1 jar? Thanks Rekha From: Sarath sarathchandra.jos...@algofusiontech.com Reply-To: user@hive.apache.org Date: Mon, 17 Sep 2012 16:24:02 +0530 To: user@hive.apache.org Subject: Issue while using hive 0.9 Hi, I'm new to hive. As first step downloaded version 0.9 and upacked it. Set the HIVE_HOME to the unpacked folder. I already have hadoop 1.0.3 installed and its configuration is in classpath. Then issued the hive command and got the hive prompt. After this whatever I try to execute, results in the below error - *Exception in thread main java.lang.NoSuchFieldError: type at org.apache.hadoop.hive.ql.parse.HiveLexer.mKW_CREATE(HiveLexer.java:1602) at org.apache.hadoop.hive.ql.parse.HiveLexer.mTokens(HiveLexer.java:6380) at org.antlr.runtime.Lexer.nextToken(Lexer.java:89) at org.antlr.runtime.BufferedTokenStream.fetch(BufferedTokenStream.java:133) at org.antlr.runtime.BufferedTokenStream.sync(BufferedTokenStream.java:127) at org.antlr.runtime.CommonTokenStream.setup(CommonTokenStream.java:132) at org.antlr.runtime.CommonTokenStream.LT(CommonTokenStream.java:91) at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:547) at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:438) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:416) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:336) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:909) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:689) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:557) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:156)* Googled and found several posts with the same issue. Every where it says that issue is with version of antlr.jar. But I find that antlr.jar is already with correct version inside hive's lib directory and I don't think there's any other antlr.jar in my classpath. 1. What is going wrong? 2. Where and what else to be checked? 3. How to resolve and start using Hive? Please guide me. Regards, Sarath.
