[ https://issues.apache.org/jira/browse/HIVE-9543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
fuqiang cao reassigned HIVE-9543: --------------------------------- Assignee: fuqiang cao > MetaException(message:Metastore contains multiple versions) > ----------------------------------------------------------- > > Key: HIVE-9543 > URL: https://issues.apache.org/jira/browse/HIVE-9543 > Project: Hive > Issue Type: Bug > Components: Metastore > Affects Versions: 0.13.1 > Reporter: Junyong Li > Assignee: fuqiang cao > > When i run bin/hive command, i got the following exception: > {noformat} > Logging initialized using configuration in > jar:file:/home/hadoop/apache-hive-0.13.1-bin/lib/hive-common-0.13.1.jar!/hive-log4j.properties > Exception in thread "main" java.lang.RuntimeException: > java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > Caused by: java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) > at > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453) > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465) > at > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340) > ... 7 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410) > ... 12 more > Caused by: MetaException(message:Metastore contains multiple versions) > at > org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6368) > at > org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:6330) > at > org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6289) > at > org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6277) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108) > at com.sun.proxy.$Proxy9.verifySchema(Unknown Source) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:476) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59) > at > org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171) > ... 17 more > {noformat} > And i have found two record in metastore table VERSION. after reading source > code, i found following code maybe cause the problem: > In the org.apache.hadoop.hive.metastore.ObjectStore.java:6289: > {noformat} > String schemaVer = getMetaStoreSchemaVersion(); > if (schemaVer == null) { > // metastore has no schema version information > if (strictValidation) { > throw new MetaException("Version information not found in > metastore. "); > } else { > LOG.warn("Version information not found in metastore. " > + HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION.toString() + > " is not enabled so recording the schema version " + > MetaStoreSchemaInfo.getHiveSchemaVersion()); > > setMetaStoreSchemaVersion(MetaStoreSchemaInfo.getHiveSchemaVersion(), > "Set by MetaStore"); > } > } > {noformat} > If there is exception in the {{getMetaStoreSchemaVersion()}}, Hive will > insert a duplication record > And i also found my another exception maybe the same cause as above.the > exception is : > {noformat} > Exception in thread "main" java.lang.RuntimeException: > java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > Caused by: java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) > at > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453) > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465) > at > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340) > ... 7 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410) > ... 12 more > Caused by: javax.jdo.JDODataStoreException: Insert of object > "org.apache.hadoop.hive.metastore.model.MRoleMap@2fd02622" using statement > "INSERT INTO `ROLE_MAP` (`ROLE_GRANT_ID`,`GRANTOR`,`A > DD_TIME`,`ROLE_ID`,`GRANTOR_TYPE`,`GRANT_OPTION`,`PRINCIPAL_NAME`,`PRINCIPAL_TYPE`) > VALUES (?,?,?,?,?,?,?,?)" failed : Duplicate entry 'hadoop-1-admin-ROLE' for > key 'USERROLEMAPINDEX' > NestedThrowables: > com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: > Duplicate entry 'hadoop-1-admin-ROLE' for key 'USERROLEMAPINDEX' > at > org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) > at > org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732) > at > org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752) > at > org.apache.hadoop.hive.metastore.ObjectStore.grantRole(ObjectStore.java:3058) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108) > at com.sun.proxy.$Proxy9.grantRole(Unknown Source) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addAdminUsers(HiveMetaStore.java:611) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:399) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:55) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:60) > at > org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171) > ... 17 more > Caused by: > com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: > Duplicate entry 'hadoop-1-admin-ROLE' for key 'USERROLEMAPINDEX' > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:406) > at com.mysql.jdbc.Util.getInstance(Util.java:381) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3515) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3447) > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1951) > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2101) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2554) > at > com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1761) > at > com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2046) > at > com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1964) > at > com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1949) > at > com.jolbox.bonecp.PreparedStatementHandle.executeUpdate(PreparedStatementHandle.java:205) > at > org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeUpdate(ParamLoggingPreparedStatement.java:399) > at > org.datanucleus.store.rdbms.SQLController.executeStatementUpdate(SQLController.java:439) > at > org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:410) > at > org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:167) > at > org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:143) > at > org.datanucleus.state.JDOStateManager.internalMakePersistent(JDOStateManager.java:3784) > at > org.datanucleus.state.JDOStateManager.makePersistent(JDOStateManager.java:3760) > at > org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2219) > at > org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:2065) > at > org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1913) > at > org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217) > at > org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:727) > ... 32 more > {noformat} > The following code maybe cause the exception, in In the > org.apache.hadoop.hive.metastore.ObjectStore.java:3018: > {noformat} > MRole nameCheck = this.getMRole(roleName); > if (nameCheck != null) { > throw new InvalidObjectException("Role " + roleName + " already > exists."); > } > int now = (int)(System.currentTimeMillis()/1000); > MRole mRole = new MRole(roleName, now, > ownerName); > pm.makePersistent(mRole); > commited = commitTransaction(); > success = true; > {noformat} > If there is exception in the {{getMRole(roleName)}} function, Hive will try > to insert a duplicate record into table ROLE_MAP, this will cause exception > above. > So can we just throw the exception instead of ignoring the exception in the > {{getMRole(ObjectStore.java:3292)}} and in the > {{getMetaStoreSchemaVersion(ObjectStore.java:6326)}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)