Xiao Yu created HIVE-25481:
------------------------------
Summary: Schema Validation with MySQL Throws Exceptions
Key: HIVE-25481
URL: https://issues.apache.org/jira/browse/HIVE-25481
Project: Hive
Issue Type: Bug
Components: Database/Schema
Affects Versions: 2.3.6
Environment: * Bigtop 1.5.0 distributed Hadoop (Hive 2.3.6)
* MariaDB (but also likely will happen with MySQL as well)
* MySQL JDBC connector (8.0.26)
Reporter: Xiao Yu
When running with a MySQL/MariaDB backend setting
{{datanucleus.schema.validateTables}} to true causes warnings like the
following to be thrown:
{code}
2021-08-24 17:56:54,765 [WARN ] DataNucleus.Query: Query for candidates of
org.apache.hadoop.hive.metastore.model.MPartition and subclasses resulted in no
possible candidates
org.datanucleus.store.rdbms.exceptions.NotATableException: Table "`PARTITIONS`"
is of wrong type SYSTEM VIEW. Should be a table
at org.datanucleus.store.rdbms.table.TableImpl.validate(TableImpl.java:161)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:3407)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2896)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:119)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.RDBMSStoreManager.manageClasses(RDBMSStoreManager.java:1627)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.RDBMSStoreManager.getDatastoreClass(RDBMSStoreManager.java:672)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.query.RDBMSQueryUtils.getStatementForCandidates(RDBMSQueryUtils.java:425)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.query.JDOQLQuery.compileQueryFull(JDOQLQuery.java:865)
[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.query.JDOQLQuery.compileInternal(JDOQLQuery.java:347)
[datanucleus-rdbms-4.1.19.jar:?]
at org.datanucleus.store.query.Query.executeQuery(Query.java:1816)
[datanucleus-core-4.1.17.jar:?]
at org.datanucleus.store.query.Query.executeWithArray(Query.java:1744)
[datanucleus-core-4.1.17.jar:?]
at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:368)
[datanucleus-api-jdo-4.2.4.jar:?]
at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:241)
[datanucleus-api-jdo-4.2.4.jar:?]
at
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionNamesNoTxn(ObjectStore.java:2498)
[hive-exec-2.3.6.jar:2.3.6]
at
org.apache.hadoop.hive.metastore.ObjectStore.listPartitionNames(ObjectStore.java:2231)
[hive-exec-2.3.6.jar:2.3.6]
at jdk.internal.reflect.GeneratedMethodAccessor122.invoke(Unknown Source)
~[?:?]
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
[hive-exec-2.3.6.jar:2.3.6]
at com.sun.proxy.$Proxy41.listPartitionNames(Unknown Source) [?:?]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition_names(HiveMetaStore.java:3708)
[hive-exec-2.3.6.jar:2.3.6]
at jdk.internal.reflect.GeneratedMethodAccessor121.invoke(Unknown Source)
~[?:?]
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
[hive-exec-2.3.6.jar:2.3.6]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
[hive-exec-2.3.6.jar:2.3.6]
at com.sun.proxy.$Proxy42.get_partition_names(Unknown Source) [?:?]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partition_names.getResult(ThriftHiveMetastore.java:12287)
[hive-exec-2.3.6.jar:2.3.6]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partition_names.getResult(ThriftHiveMetastore.java:12271)
[hive-exec-2.3.6.jar:2.3.6]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
[hive-exec-2.3.6.jar:2.3.6]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
[hive-exec-2.3.6.jar:2.3.6]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
[hive-exec-2.3.6.jar:2.3.6]
at java.base/java.security.AccessController.doPrivileged(Native Method)
~[?:?]
at java.base/javax.security.auth.Subject.doAs(Subject.java:423) [?:?]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1926)
[hadoop-common-2.10.1.jar:?]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
[hive-exec-2.3.6.jar:2.3.6]
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
[hive-exec-2.3.6.jar:2.3.6]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
{code}
It appears this is because the schema validation is mistaking the internal
{{INFORMATION_SCHEMA.PARTITIONS}} table with the Hive created
{{[$HIVE_DB].PARTITIONS}} table. From the MySQL side it appears this is
happening because the validation routine is iterating through all databases
that it has access to not just using the specific DB configured via
{{javax.jdo.option.ConnectionURL}}. We see the following set of queries issued
when this error occurs:
{code}
SET autocommit=0
SELECT @@session.tx_isolation
SHOW DATABASES
SHOW FULL TABLES FROM `heartbeat` LIKE PARTITIONS
SHOW FULL TABLES FROM `information_schema` LIKE PARTITIONS
SHOW FULL TABLES FROM `wyeast-hive` LIKE PARTITIONS
SHOW DATABASES
SHOW FULL TABLES FROM `heartbeat` LIKE PARTITIONS
SHOW FULL TABLES FROM `information_schema` LIKE PARTITIONS
SHOW FULL TABLES FROM `wyeast-hive` LIKE PARTITIONS
rollback
commit
rollback
SET autocommit=1
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
