[
https://issues.apache.org/jira/browse/FLINK-27999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-27999:
-------------------------------------
Description:
VirtualColumn.VIRTUAL_COLUMN_NAMES uses different types in Hive2 / Hive3,
causing NoSuchMethodErrors when the Hive dialect is used.
When use 'mvn test -PHive3.1.1' Hive connector, it will throw the following
exception "Caused by: MetaException(message:Required table missing : "DBS" in
Catalog "" Schema "". DataNucleus requires this table to perform its
persistence operations. Either your MetaData is incorrect, or you need to
enable "datanucleus.schema.autoCreateTables")".
>From the error message, we can know the reason is the configuration
>"datanucleus.schema.autoCreateTables" is not true. But when create
>HiveCatalog, we do really set the configuration
>"datanucleus.schema.autoCreateTables" to true in the hive-site.xml.
After some debuging, I found the reason is that some test modify a static
variable, and then boil the other test.
When running HiveCatalogFactoryTest, it will call
HiveCatalog#createHiveConf(@Nullable String hiveConfDir, @Nullable String
hadoopConfDir) to create a HiveConf for HiveCatalog. In this method, it will
set the static variable "hiveSiteURL" to null.
Then, if we run "HiveCatalogHiveMetadataTest", it will call
HiveTestUtils#createHiveConf() to create HiveConf. The following code will
create a empty HiveConf;
{code:java}
HiveConf hiveConf = new HiveConf(); {code}
But in the initialize function of HiveConf, it first will apply all default
variables, so "datanucleus.schema.autoCreateTables" will be set to false. And
the "hiveSiteURL" is null, so skip add the resource. Then, it will check
""hive.metastore.schema.verification" is true or false. If it's true, it will
set "datanucleus.schema.autoCreateTables" to false. This have a higher priority
and thus overwrite the value we configure in hive-site.xml.
To fix it, we only need to reset the static variable "hiveSiteURL" with our
hive-site.xml.
was:
When use 'mvn test -PHive3.1.1' Hive connector, it will throw the following
exception "Caused by: MetaException(message:Required table missing : "DBS" in
Catalog "" Schema "". DataNucleus requires this table to perform its
persistence operations. Either your MetaData is incorrect, or you need to
enable "datanucleus.schema.autoCreateTables")".
>From the error message, we can know the reason is the configuration
>"datanucleus.schema.autoCreateTables" is not true. But when create
>HiveCatalog, we do really set the configuration
>"datanucleus.schema.autoCreateTables" to true in the hive-site.xml.
After some debuging, I found the reason is that some test modify a static
variable, and then boil the other test.
When running HiveCatalogFactoryTest, it will call
HiveCatalog#createHiveConf(@Nullable String hiveConfDir, @Nullable String
hadoopConfDir) to create a HiveConf for HiveCatalog. In this method, it will
set the static variable "hiveSiteURL" to null.
Then, if we run "HiveCatalogHiveMetadataTest", it will call
HiveTestUtils#createHiveConf() to create HiveConf. The following code will
create a empty HiveConf;
{code:java}
HiveConf hiveConf = new HiveConf(); {code}
But in the initialize function of HiveConf, it first will apply all default
variables, so "datanucleus.schema.autoCreateTables" will be set to false. And
the "hiveSiteURL" is null, so skip add the resource. Then, it will check
""hive.metastore.schema.verification" is true or false. If it's true, it will
set "datanucleus.schema.autoCreateTables" to false. This have a higher priority
and thus overwrite the value we configure in hive-site.xml.
To fix it, we only need to reset the static variable "hiveSiteURL" with our
hive-site.xml.
> NoSuchMethodError when using Hive 3 dialect
> -------------------------------------------
>
> Key: FLINK-27999
> URL: https://issues.apache.org/jira/browse/FLINK-27999
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
> Affects Versions: 1.15.0
> Reporter: luoyuxia
> Assignee: luoyuxia
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0, 1.15.2
>
>
> VirtualColumn.VIRTUAL_COLUMN_NAMES uses different types in Hive2 / Hive3,
> causing NoSuchMethodErrors when the Hive dialect is used.
> When use 'mvn test -PHive3.1.1' Hive connector, it will throw the following
> exception "Caused by: MetaException(message:Required table missing : "DBS" in
> Catalog "" Schema "". DataNucleus requires this table to perform its
> persistence operations. Either your MetaData is incorrect, or you need to
> enable "datanucleus.schema.autoCreateTables")".
> From the error message, we can know the reason is the configuration
> "datanucleus.schema.autoCreateTables" is not true. But when create
> HiveCatalog, we do really set the configuration
> "datanucleus.schema.autoCreateTables" to true in the hive-site.xml.
> After some debuging, I found the reason is that some test modify a static
> variable, and then boil the other test.
> When running HiveCatalogFactoryTest, it will call
> HiveCatalog#createHiveConf(@Nullable String hiveConfDir, @Nullable String
> hadoopConfDir) to create a HiveConf for HiveCatalog. In this method, it will
> set the static variable "hiveSiteURL" to null.
> Then, if we run "HiveCatalogHiveMetadataTest", it will call
> HiveTestUtils#createHiveConf() to create HiveConf. The following code will
> create a empty HiveConf;
>
> {code:java}
> HiveConf hiveConf = new HiveConf(); {code}
> But in the initialize function of HiveConf, it first will apply all default
> variables, so "datanucleus.schema.autoCreateTables" will be set to false. And
> the "hiveSiteURL" is null, so skip add the resource. Then, it will check
> ""hive.metastore.schema.verification" is true or false. If it's true, it will
> set "datanucleus.schema.autoCreateTables" to false. This have a higher
> priority and thus overwrite the value we configure in hive-site.xml.
> To fix it, we only need to reset the static variable "hiveSiteURL" with our
> hive-site.xml.
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
