[ https://issues.apache.org/jira/browse/FLINK-27999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chesnay Schepler updated FLINK-27999: ------------------------------------- Description: VirtualColumn.VIRTUAL_COLUMN_NAMES uses different types in Hive2 / Hive3, causing NoSuchMethodErrors when the Hive dialect is used. When use 'mvn test -PHive3.1.1' Hive connector, it will throw the following exception "Caused by: MetaException(message:Required table missing : "DBS" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.schema.autoCreateTables")". >From the error message, we can know the reason is the configuration >"datanucleus.schema.autoCreateTables" is not true. But when create >HiveCatalog, we do really set the configuration >"datanucleus.schema.autoCreateTables" to true in the hive-site.xml. After some debuging, I found the reason is that some test modify a static variable, and then boil the other test. When running HiveCatalogFactoryTest, it will call HiveCatalog#createHiveConf(@Nullable String hiveConfDir, @Nullable String hadoopConfDir) to create a HiveConf for HiveCatalog. In this method, it will set the static variable "hiveSiteURL" to null. Then, if we run "HiveCatalogHiveMetadataTest", it will call HiveTestUtils#createHiveConf() to create HiveConf. The following code will create a empty HiveConf; {code:java} HiveConf hiveConf = new HiveConf(); {code} But in the initialize function of HiveConf, it first will apply all default variables, so "datanucleus.schema.autoCreateTables" will be set to false. And the "hiveSiteURL" is null, so skip add the resource. Then, it will check ""hive.metastore.schema.verification" is true or false. If it's true, it will set "datanucleus.schema.autoCreateTables" to false. This have a higher priority and thus overwrite the value we configure in hive-site.xml. To fix it, we only need to reset the static variable "hiveSiteURL" with our hive-site.xml. was: When use 'mvn test -PHive3.1.1' Hive connector, it will throw the following exception "Caused by: MetaException(message:Required table missing : "DBS" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.schema.autoCreateTables")". >From the error message, we can know the reason is the configuration >"datanucleus.schema.autoCreateTables" is not true. But when create >HiveCatalog, we do really set the configuration >"datanucleus.schema.autoCreateTables" to true in the hive-site.xml. After some debuging, I found the reason is that some test modify a static variable, and then boil the other test. When running HiveCatalogFactoryTest, it will call HiveCatalog#createHiveConf(@Nullable String hiveConfDir, @Nullable String hadoopConfDir) to create a HiveConf for HiveCatalog. In this method, it will set the static variable "hiveSiteURL" to null. Then, if we run "HiveCatalogHiveMetadataTest", it will call HiveTestUtils#createHiveConf() to create HiveConf. The following code will create a empty HiveConf; {code:java} HiveConf hiveConf = new HiveConf(); {code} But in the initialize function of HiveConf, it first will apply all default variables, so "datanucleus.schema.autoCreateTables" will be set to false. And the "hiveSiteURL" is null, so skip add the resource. Then, it will check ""hive.metastore.schema.verification" is true or false. If it's true, it will set "datanucleus.schema.autoCreateTables" to false. This have a higher priority and thus overwrite the value we configure in hive-site.xml. To fix it, we only need to reset the static variable "hiveSiteURL" with our hive-site.xml. > NoSuchMethodError when using Hive 3 dialect > ------------------------------------------- > > Key: FLINK-27999 > URL: https://issues.apache.org/jira/browse/FLINK-27999 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive > Affects Versions: 1.15.0 > Reporter: luoyuxia > Assignee: luoyuxia > Priority: Major > Labels: pull-request-available > Fix For: 1.16.0, 1.15.2 > > > VirtualColumn.VIRTUAL_COLUMN_NAMES uses different types in Hive2 / Hive3, > causing NoSuchMethodErrors when the Hive dialect is used. > When use 'mvn test -PHive3.1.1' Hive connector, it will throw the following > exception "Caused by: MetaException(message:Required table missing : "DBS" in > Catalog "" Schema "". DataNucleus requires this table to perform its > persistence operations. Either your MetaData is incorrect, or you need to > enable "datanucleus.schema.autoCreateTables")". > From the error message, we can know the reason is the configuration > "datanucleus.schema.autoCreateTables" is not true. But when create > HiveCatalog, we do really set the configuration > "datanucleus.schema.autoCreateTables" to true in the hive-site.xml. > After some debuging, I found the reason is that some test modify a static > variable, and then boil the other test. > When running HiveCatalogFactoryTest, it will call > HiveCatalog#createHiveConf(@Nullable String hiveConfDir, @Nullable String > hadoopConfDir) to create a HiveConf for HiveCatalog. In this method, it will > set the static variable "hiveSiteURL" to null. > Then, if we run "HiveCatalogHiveMetadataTest", it will call > HiveTestUtils#createHiveConf() to create HiveConf. The following code will > create a empty HiveConf; > > {code:java} > HiveConf hiveConf = new HiveConf(); {code} > But in the initialize function of HiveConf, it first will apply all default > variables, so "datanucleus.schema.autoCreateTables" will be set to false. And > the "hiveSiteURL" is null, so skip add the resource. Then, it will check > ""hive.metastore.schema.verification" is true or false. If it's true, it will > set "datanucleus.schema.autoCreateTables" to false. This have a higher > priority and thus overwrite the value we configure in hive-site.xml. > To fix it, we only need to reset the static variable "hiveSiteURL" with our > hive-site.xml. > > > > -- This message was sent by Atlassian Jira (v8.20.7#820007)