[
https://issues.apache.org/jira/browse/HIVE-24279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xianyin Xin updated HIVE-24279:
-------------------------------
Description:
I got some exception when i configured transaction,
{code}
2020-10-15T11:05:41,356 ERROR [Thread-7] compactor.Initiator: Caught an
exception in the main loop of compactor initiator, exiting
MetaException(message:Unable to connect to transaction database
java.sql.SQLSyntaxErrorException: Table/View 'COMPACTION_QUEUE' does not exist.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeLargeUpdate(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(Unknown
Source)
at
com.zaxxer.hikari.pool.ProxyStatement.executeUpdate(ProxyStatement.java:117)
at
com.zaxxer.hikari.pool.HikariProxyStatement.executeUpdate(HikariProxyStatement.java)
at
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.revokeFromLocalWorkers(CompactionTxnHandler.java:646)
at
org.apache.hadoop.hive.ql.txn.compactor.Initiator.recoverFailedCompactions(Initiator.java:208)
at
org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:74)
Caused by: ERROR 42X05: Table/View 'COMPACTION_QUEUE' does not exist.
{code}
After some debugging, i found the conf that passed to the {{CompactorThread}}
was:
{code}
public void setConf(Configuration configuration) {
// TODO MS-SPLIT for now, keep a copy of HiveConf around as we need to call
other methods with
// it. This should be changed to Configuration once everything that this
calls that requires
// HiveConf is moved to the standalone metastore.
conf = (configuration instanceof HiveConf) ? (HiveConf)configuration :
new HiveConf(configuration, HiveConf.class);
}
{code}
However, {{new HiveConf(configuration, HiveConf.class)}} would not inherit all
the configs come from {{configuration}}, actually the {{configuration}} will be
overwrite by the default values those are not nulls:
{code}
private void initialize(Class<?> cls) {
hiveJar = (new JobConf(cls)).getJar();
// preserve the original configuration
origProp = getAllProperties();
// Overlay the ConfVars. Note that this ignores ConfVars with null values
addResource(getConfVarInputStream());
// Overlay hive-site.xml if it exists
if (hiveSiteURL != null) {
addResource(hiveSiteURL);
}
// if embedded metastore is to be used as per config so far
// then this is considered like the metastore server case
String msUri = this.getVar(HiveConf.ConfVars.METASTOREURIS);
// This is hackery, but having hive-common depend on standalone-metastore
is really bad
// because it will pull all of the metastore code into every module. We
need to check that
// we aren't using the standalone metastore. If we are, we should treat it
the same as a
// remote metastore situation.
if (msUri == null || msUri.isEmpty()) {
msUri = this.get("metastore.thrift.uris");
}
LOG.debug("Found metastore URI of " + msUri);
if(HiveConfUtil.isEmbeddedMetaStore(msUri)){
setLoadMetastoreConfig(true);
}
// load hivemetastore-site.xml if this is metastore and file exists
if (isLoadMetastoreConfig() && hivemetastoreSiteUrl != null) {
addResource(hivemetastoreSiteUrl);
}
{code}
That is, {{new HiveConf(configuration, HiveConf.class)}} is merely a new
{{hiveConf}} but the configs which have null default values will be overwrite
by {{configuration}}. A {{hiveConf}} would not load hivemetastore-site except
that it is an embedded metastore. If hive-site doesn't have the db connection
info, {{CompactorThread}} would connect to the default derby.
was:
I got some exception when i configured transaction,
{code}
2020-10-15T11:05:41,356 ERROR [Thread-7] compactor.Initiator: Caught an
exception in the main loop of compactor initiator, exiting
MetaException(message:Unable to connect to transaction database
java.sql.SQLSyntaxErrorException: Table/View 'COMPACTION_QUEUE' does not exist.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeLargeUpdate(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(Unknown
Source)
at
com.zaxxer.hikari.pool.ProxyStatement.executeUpdate(ProxyStatement.java:117)
at
com.zaxxer.hikari.pool.HikariProxyStatement.executeUpdate(HikariProxyStatement.java)
at
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.revokeFromLocalWorkers(CompactionTxnHandler.java:646)
at
org.apache.hadoop.hive.ql.txn.compactor.Initiator.recoverFailedCompactions(Initiator.java:208)
at
org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:74)
Caused by: ERROR 42X05: Table/View 'COMPACTION_QUEUE' does not exist.
{code}
After some debugging, i found the conf that passed to the {{CompactorThread}}
was:
{code}
public void setConf(Configuration configuration) {
// TODO MS-SPLIT for now, keep a copy of HiveConf around as we need to call
other methods with
// it. This should be changed to Configuration once everything that this
calls that requires
// HiveConf is moved to the standalone metastore.
conf = (configuration instanceof HiveConf) ? (HiveConf)configuration :
new HiveConf(configuration, HiveConf.class);
}
{code}
However, {{new HiveConf(configuration, HiveConf.class)}} would not inherit all
the configs come from {{configuration}}, actually the {{configuration}} will be
overwrite by the default values those are not nulls:
{code}
private void initialize(Class<?> cls) {
hiveJar = (new JobConf(cls)).getJar();
// preserve the original configuration
origProp = getAllProperties();
// Overlay the ConfVars. Note that this ignores ConfVars with null values
addResource(getConfVarInputStream());
// Overlay hive-site.xml if it exists
if (hiveSiteURL != null) {
addResource(hiveSiteURL);
}
// if embedded metastore is to be used as per config so far
// then this is considered like the metastore server case
String msUri = this.getVar(HiveConf.ConfVars.METASTOREURIS);
// This is hackery, but having hive-common depend on standalone-metastore
is really bad
// because it will pull all of the metastore code into every module. We
need to check that
// we aren't using the standalone metastore. If we are, we should treat it
the same as a
// remote metastore situation.
if (msUri == null || msUri.isEmpty()) {
msUri = this.get("metastore.thrift.uris");
}
LOG.debug("Found metastore URI of " + msUri);
if(HiveConfUtil.isEmbeddedMetaStore(msUri)){
setLoadMetastoreConfig(true);
}
// load hivemetastore-site.xml if this is metastore and file exists
if (isLoadMetastoreConfig() && hivemetastoreSiteUrl != null) {
addResource(hivemetastoreSiteUrl);
}
{code}
That is, {{new HiveConf(configuration, HiveConf.class)}} is merely a new
{{hiveConf}} but the configs which have null default values will be overwrite
by {{configuration}}. A {{hiveConf}} would not load hivemetastore-site except
that it is an embedded metastore. If hive-site doesn't have the db connection
info, {{CompactorThread}} would connect to the default derby which causes the
failure.
> Hive CompactorThread fails to connect to metastore if the connectionURL was
> only configs in metastore-site
> ----------------------------------------------------------------------------------------------------------
>
> Key: HIVE-24279
> URL: https://issues.apache.org/jira/browse/HIVE-24279
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
> Affects Versions: 3.1.2
> Reporter: Xianyin Xin
> Priority: Major
>
> I got some exception when i configured transaction,
> {code}
> 2020-10-15T11:05:41,356 ERROR [Thread-7] compactor.Initiator: Caught an
> exception in the main loop of compactor initiator, exiting
> MetaException(message:Unable to connect to transaction database
> java.sql.SQLSyntaxErrorException: Table/View 'COMPACTION_QUEUE' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at
> org.apache.derby.impl.jdbc.EmbedStatement.executeLargeUpdate(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(Unknown
> Source)
> at
> com.zaxxer.hikari.pool.ProxyStatement.executeUpdate(ProxyStatement.java:117)
> at
> com.zaxxer.hikari.pool.HikariProxyStatement.executeUpdate(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.revokeFromLocalWorkers(CompactionTxnHandler.java:646)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.recoverFailedCompactions(Initiator.java:208)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:74)
> Caused by: ERROR 42X05: Table/View 'COMPACTION_QUEUE' does not exist.
> {code}
> After some debugging, i found the conf that passed to the {{CompactorThread}}
> was:
> {code}
> public void setConf(Configuration configuration) {
> // TODO MS-SPLIT for now, keep a copy of HiveConf around as we need to
> call other methods with
> // it. This should be changed to Configuration once everything that this
> calls that requires
> // HiveConf is moved to the standalone metastore.
> conf = (configuration instanceof HiveConf) ? (HiveConf)configuration :
> new HiveConf(configuration, HiveConf.class);
> }
> {code}
> However, {{new HiveConf(configuration, HiveConf.class)}} would not inherit
> all the configs come from {{configuration}}, actually the {{configuration}}
> will be overwrite by the default values those are not nulls:
> {code}
> private void initialize(Class<?> cls) {
> hiveJar = (new JobConf(cls)).getJar();
> // preserve the original configuration
> origProp = getAllProperties();
> // Overlay the ConfVars. Note that this ignores ConfVars with null values
> addResource(getConfVarInputStream());
> // Overlay hive-site.xml if it exists
> if (hiveSiteURL != null) {
> addResource(hiveSiteURL);
> }
> // if embedded metastore is to be used as per config so far
> // then this is considered like the metastore server case
> String msUri = this.getVar(HiveConf.ConfVars.METASTOREURIS);
> // This is hackery, but having hive-common depend on standalone-metastore
> is really bad
> // because it will pull all of the metastore code into every module. We
> need to check that
> // we aren't using the standalone metastore. If we are, we should treat
> it the same as a
> // remote metastore situation.
> if (msUri == null || msUri.isEmpty()) {
> msUri = this.get("metastore.thrift.uris");
> }
> LOG.debug("Found metastore URI of " + msUri);
> if(HiveConfUtil.isEmbeddedMetaStore(msUri)){
> setLoadMetastoreConfig(true);
> }
> // load hivemetastore-site.xml if this is metastore and file exists
> if (isLoadMetastoreConfig() && hivemetastoreSiteUrl != null) {
> addResource(hivemetastoreSiteUrl);
> }
> {code}
> That is, {{new HiveConf(configuration, HiveConf.class)}} is merely a new
> {{hiveConf}} but the configs which have null default values will be overwrite
> by {{configuration}}. A {{hiveConf}} would not load hivemetastore-site except
> that it is an embedded metastore. If hive-site doesn't have the db connection
> info, {{CompactorThread}} would connect to the default derby.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
