kingkongpoon opened a new issue #2517:
URL: https://github.com/apache/hudi/issues/2517
hive version 2.3.5
mysql's hive.VERSION SCHEMA_VERSION=2.3.0
When I successfully run my spark hudi job
```
8999 [main] INFO io.javalin.Javalin - Starting Javalin ...
9116 [main] INFO io.javalin.Javalin - Listening on http://localhost:40474/
9116 [main] INFO io.javalin.Javalin - Javalin started in 121ms \o/
15347 [main] INFO DataNucleus.Persistence - Property
hive.metastore.integral.jdo.pushdown unknown - will be ignored
15347 [main] INFO DataNucleus.Persistence - Property
datanucleus.cache.level2 unknown - will be ignored
17028 [main] INFO DataNucleus.Datastore - The class
"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
"embedded-only" so does not have its own datastore table.
17029 [main] INFO DataNucleus.Datastore - The class
"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so
does not have its own datastore table.
17137 [main] INFO DataNucleus.Datastore - The class
"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
"embedded-only" so does not have its own datastore table.
17137 [main] INFO DataNucleus.Datastore - The class
"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so
does not have its own datastore table.
17193 [main] INFO DataNucleus.Query - Reading in results for query
"org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is
closing
17336 [main] ERROR org.apache.hadoop.hive.metastore.ObjectStore - Version
information found in metastore differs 2.3.0 from expected schema version
1.2.0. Schema verififcation is disabled hive.metastore.schema.verification so
setting version.
17641 [main] INFO DataNucleus.Datastore - The class
"org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as
"embedded-only" so does not have its own datastore table.
17683 [main] INFO hive.metastore - Mestastore configuration
hive.metastore.warehouse.dir changed from /user/hive/warehouse to
file:/root/spark-warehouse/
17690 [main] INFO DataNucleus.Query - Reading in results for query
"org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is
closing
17948 [main] INFO io.javalin.Javalin - Stopping Javalin ...
17957 [main] INFO io.javalin.Javalin - Javalin has stopped
```
it throws an error ,17336 [main] ERROR
then I find it comes from
```
package org.apache.hadoop.hive.metastore
...
private synchronized void checkSchema() throws MetaException {
// recheck if it got verified by another thread while we were waiting
if (isSchemaVerified.get()) {
return;
}
boolean strictValidation =
HiveConf.getBoolVar(getConf(),
HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION);
// read the schema version stored in metastore db
String dbSchemaVer = getMetaStoreSchemaVersion();
// version of schema for this version of hive
String hiveSchemaVer = MetaStoreSchemaInfo.getHiveSchemaVersion();
if (dbSchemaVer == null) {
if (strictValidation) {
throw new MetaException("Version information not found in metastore.
");
} else {
LOG.warn("Version information not found in metastore. "
+ HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION.toString() +
" is not enabled so recording the schema version " +
hiveSchemaVer);
setMetaStoreSchemaVersion(hiveSchemaVer,
"Set by MetaStore " + USER + "@" + HOSTNAME);
}
} else {
if (MetaStoreSchemaInfo.isVersionCompatible(hiveSchemaVer,
dbSchemaVer)) {
LOG.debug("Found expected HMS version of " + dbSchemaVer);
} else {
// metastore schema version is different than Hive distribution needs
if (strictValidation) {
throw new MetaException("Hive Schema version " + hiveSchemaVer +
" does not match metastore's schema version " + dbSchemaVer +
" Metastore is not upgraded or corrupt");
} else {
LOG.error("Version information found in metastore differs " +
dbSchemaVer +
" from expected schema version " + hiveSchemaVer +
". Schema verififcation is disabled " +
HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION);
setMetaStoreSchemaVersion(hiveSchemaVer,
"Set by MetaStore " + USER + "@" + HOSTNAME);
}
}
}
isSchemaVerified.set(true);
return;
}
...
```
Then the value (in mysql)hive.VERSION SCHEMA_VERSION=2.3.0 change to 1.2.0
And then I rerun my spark hudi jar ,it will not throw that error, but I
can not restart my hive metastre ,because it's version change to 1.2.0 and
hive --service metastore will throw error,
I put hive-site.xml in my maven project, include the mysql's user and
password
I want to know why the spark hudi job can change the hive metadata which in
mysql???
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
