[ 
https://issues.apache.org/jira/browse/HIVE-24279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xianyin Xin updated HIVE-24279:
-------------------------------
    Description: 
I got some exception when i configured transaction,
{code}
2020-10-15T11:05:41,356 ERROR [Thread-7] compactor.Initiator: Caught an 
exception in the main loop of compactor initiator, exiting 
MetaException(message:Unable to connect to transaction database 
java.sql.SQLSyntaxErrorException: Table/View 'COMPACTION_QUEUE' does not exist.
        at 
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown 
Source)
        at 
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown 
Source)
        at 
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown 
Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown 
Source)
        at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown 
Source)
        at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedStatement.executeLargeUpdate(Unknown 
Source)
        at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(Unknown 
Source)
        at 
com.zaxxer.hikari.pool.ProxyStatement.executeUpdate(ProxyStatement.java:117)
        at 
com.zaxxer.hikari.pool.HikariProxyStatement.executeUpdate(HikariProxyStatement.java)
        at 
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.revokeFromLocalWorkers(CompactionTxnHandler.java:646)
        at 
org.apache.hadoop.hive.ql.txn.compactor.Initiator.recoverFailedCompactions(Initiator.java:208)
        at 
org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:74)
Caused by: ERROR 42X05: Table/View 'COMPACTION_QUEUE' does not exist.
{code}

After some debugging, i found the conf that passed to the {{CompactorThread}} 
was:
{code}
  public void setConf(Configuration configuration) {
    // TODO MS-SPLIT for now, keep a copy of HiveConf around as we need to call 
other methods with
    // it. This should be changed to Configuration once everything that this 
calls that requires
    // HiveConf is moved to the standalone metastore.
    conf = (configuration instanceof HiveConf) ? (HiveConf)configuration :
        new HiveConf(configuration, HiveConf.class);
  }
{code}

However, {{new HiveConf(configuration, HiveConf.class)}} would not inherit all 
the configs come from {{configuration}}, actually the {{configuration}} will be 
overwrite by the default values those are not nulls:

{code}
  private void initialize(Class<?> cls) {
    hiveJar = (new JobConf(cls)).getJar();

    // preserve the original configuration
    origProp = getAllProperties();

    // Overlay the ConfVars. Note that this ignores ConfVars with null values
    addResource(getConfVarInputStream());

    // Overlay hive-site.xml if it exists
    if (hiveSiteURL != null) {
      addResource(hiveSiteURL);
    }

    // if embedded metastore is to be used as per config so far
    // then this is considered like the metastore server case
    String msUri = this.getVar(HiveConf.ConfVars.METASTOREURIS);
    // This is hackery, but having hive-common depend on standalone-metastore 
is really bad
    // because it will pull all of the metastore code into every module.  We 
need to check that
    // we aren't using the standalone metastore.  If we are, we should treat it 
the same as a
    // remote metastore situation.
    if (msUri == null || msUri.isEmpty()) {
      msUri = this.get("metastore.thrift.uris");
    }
    LOG.debug("Found metastore URI of " + msUri);
    if(HiveConfUtil.isEmbeddedMetaStore(msUri)){
      setLoadMetastoreConfig(true);
    }

    // load hivemetastore-site.xml if this is metastore and file exists
    if (isLoadMetastoreConfig() && hivemetastoreSiteUrl != null) {
      addResource(hivemetastoreSiteUrl);
    }
{code}

That is, {{new HiveConf(configuration, HiveConf.class)}} is merely a new 
{{hiveConf}} but the configs which have null default values will be overwrite 
by {{configuration}}. A {{hiveConf}} would not load hivemetastore-site except 
that it is an embedded metastore. If hive-site doesn't have the db connection 
info, {{CompactorThread}} would connect to the default derby.

  was:
I got some exception when i configured transaction,
{code}
2020-10-15T11:05:41,356 ERROR [Thread-7] compactor.Initiator: Caught an 
exception in the main loop of compactor initiator, exiting 
MetaException(message:Unable to connect to transaction database 
java.sql.SQLSyntaxErrorException: Table/View 'COMPACTION_QUEUE' does not exist.
        at 
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown 
Source)
        at 
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown 
Source)
        at 
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown 
Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown 
Source)
        at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown 
Source)
        at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedStatement.executeLargeUpdate(Unknown 
Source)
        at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(Unknown 
Source)
        at 
com.zaxxer.hikari.pool.ProxyStatement.executeUpdate(ProxyStatement.java:117)
        at 
com.zaxxer.hikari.pool.HikariProxyStatement.executeUpdate(HikariProxyStatement.java)
        at 
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.revokeFromLocalWorkers(CompactionTxnHandler.java:646)
        at 
org.apache.hadoop.hive.ql.txn.compactor.Initiator.recoverFailedCompactions(Initiator.java:208)
        at 
org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:74)
Caused by: ERROR 42X05: Table/View 'COMPACTION_QUEUE' does not exist.
{code}

After some debugging, i found the conf that passed to the {{CompactorThread}} 
was:
{code}
  public void setConf(Configuration configuration) {
    // TODO MS-SPLIT for now, keep a copy of HiveConf around as we need to call 
other methods with
    // it. This should be changed to Configuration once everything that this 
calls that requires
    // HiveConf is moved to the standalone metastore.
    conf = (configuration instanceof HiveConf) ? (HiveConf)configuration :
        new HiveConf(configuration, HiveConf.class);
  }
{code}

However, {{new HiveConf(configuration, HiveConf.class)}} would not inherit all 
the configs come from {{configuration}}, actually the {{configuration}} will be 
overwrite by the default values those are not nulls:

{code}
  private void initialize(Class<?> cls) {
    hiveJar = (new JobConf(cls)).getJar();

    // preserve the original configuration
    origProp = getAllProperties();

    // Overlay the ConfVars. Note that this ignores ConfVars with null values
    addResource(getConfVarInputStream());

    // Overlay hive-site.xml if it exists
    if (hiveSiteURL != null) {
      addResource(hiveSiteURL);
    }

    // if embedded metastore is to be used as per config so far
    // then this is considered like the metastore server case
    String msUri = this.getVar(HiveConf.ConfVars.METASTOREURIS);
    // This is hackery, but having hive-common depend on standalone-metastore 
is really bad
    // because it will pull all of the metastore code into every module.  We 
need to check that
    // we aren't using the standalone metastore.  If we are, we should treat it 
the same as a
    // remote metastore situation.
    if (msUri == null || msUri.isEmpty()) {
      msUri = this.get("metastore.thrift.uris");
    }
    LOG.debug("Found metastore URI of " + msUri);
    if(HiveConfUtil.isEmbeddedMetaStore(msUri)){
      setLoadMetastoreConfig(true);
    }

    // load hivemetastore-site.xml if this is metastore and file exists
    if (isLoadMetastoreConfig() && hivemetastoreSiteUrl != null) {
      addResource(hivemetastoreSiteUrl);
    }
{code}

That is, {{new HiveConf(configuration, HiveConf.class)}} is merely a new 
{{hiveConf}} but the configs which have null default values will be overwrite 
by {{configuration}}. A {{hiveConf}} would not load hivemetastore-site except 
that it is an embedded metastore. If hive-site doesn't have the db connection 
info, {{CompactorThread}} would connect to the default derby which causes the 
failure.


> Hive CompactorThread fails to connect to metastore if the connectionURL was 
> only configs in metastore-site
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-24279
>                 URL: https://issues.apache.org/jira/browse/HIVE-24279
>             Project: Hive
>          Issue Type: Improvement
>          Components: Metastore
>    Affects Versions: 3.1.2
>            Reporter: Xianyin Xin
>            Priority: Major
>
> I got some exception when i configured transaction,
> {code}
> 2020-10-15T11:05:41,356 ERROR [Thread-7] compactor.Initiator: Caught an 
> exception in the main loop of compactor initiator, exiting 
> MetaException(message:Unable to connect to transaction database 
> java.sql.SQLSyntaxErrorException: Table/View 'COMPACTION_QUEUE' does not 
> exist.
>         at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
>         at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown 
> Source)
>         at 
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown 
> Source)
>         at 
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown 
> Source)
>         at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown 
> Source)
>         at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown 
> Source)
>         at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
>         at 
> org.apache.derby.impl.jdbc.EmbedStatement.executeLargeUpdate(Unknown Source)
>         at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(Unknown 
> Source)
>         at 
> com.zaxxer.hikari.pool.ProxyStatement.executeUpdate(ProxyStatement.java:117)
>         at 
> com.zaxxer.hikari.pool.HikariProxyStatement.executeUpdate(HikariProxyStatement.java)
>         at 
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.revokeFromLocalWorkers(CompactionTxnHandler.java:646)
>         at 
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.recoverFailedCompactions(Initiator.java:208)
>         at 
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:74)
> Caused by: ERROR 42X05: Table/View 'COMPACTION_QUEUE' does not exist.
> {code}
> After some debugging, i found the conf that passed to the {{CompactorThread}} 
> was:
> {code}
>   public void setConf(Configuration configuration) {
>     // TODO MS-SPLIT for now, keep a copy of HiveConf around as we need to 
> call other methods with
>     // it. This should be changed to Configuration once everything that this 
> calls that requires
>     // HiveConf is moved to the standalone metastore.
>     conf = (configuration instanceof HiveConf) ? (HiveConf)configuration :
>         new HiveConf(configuration, HiveConf.class);
>   }
> {code}
> However, {{new HiveConf(configuration, HiveConf.class)}} would not inherit 
> all the configs come from {{configuration}}, actually the {{configuration}} 
> will be overwrite by the default values those are not nulls:
> {code}
>   private void initialize(Class<?> cls) {
>     hiveJar = (new JobConf(cls)).getJar();
>     // preserve the original configuration
>     origProp = getAllProperties();
>     // Overlay the ConfVars. Note that this ignores ConfVars with null values
>     addResource(getConfVarInputStream());
>     // Overlay hive-site.xml if it exists
>     if (hiveSiteURL != null) {
>       addResource(hiveSiteURL);
>     }
>     // if embedded metastore is to be used as per config so far
>     // then this is considered like the metastore server case
>     String msUri = this.getVar(HiveConf.ConfVars.METASTOREURIS);
>     // This is hackery, but having hive-common depend on standalone-metastore 
> is really bad
>     // because it will pull all of the metastore code into every module.  We 
> need to check that
>     // we aren't using the standalone metastore.  If we are, we should treat 
> it the same as a
>     // remote metastore situation.
>     if (msUri == null || msUri.isEmpty()) {
>       msUri = this.get("metastore.thrift.uris");
>     }
>     LOG.debug("Found metastore URI of " + msUri);
>     if(HiveConfUtil.isEmbeddedMetaStore(msUri)){
>       setLoadMetastoreConfig(true);
>     }
>     // load hivemetastore-site.xml if this is metastore and file exists
>     if (isLoadMetastoreConfig() && hivemetastoreSiteUrl != null) {
>       addResource(hivemetastoreSiteUrl);
>     }
> {code}
> That is, {{new HiveConf(configuration, HiveConf.class)}} is merely a new 
> {{hiveConf}} but the configs which have null default values will be overwrite 
> by {{configuration}}. A {{hiveConf}} would not load hivemetastore-site except 
> that it is an embedded metastore. If hive-site doesn't have the db connection 
> info, {{CompactorThread}} would connect to the default derby.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to