[
https://issues.apache.org/jira/browse/HIVE-28578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17901761#comment-17901761
]
Zsolt Miskolczi commented on HIVE-28578:
----------------------------------------
Steps to reproduce and test:
* create a cluster with multiple HMS instances.
* bombard the HMS instances with concurrent calls when the tables have
statistics both for Hive and Impala
Script to do the testing (Repro.java is attached):
{code:java}
export JAVA_HOME=/usr/java/jdk1.8.0_232-cloudera
export JAVA_OPTS="-Xmx1g"
export PATH="$JAVA_HOME/bin:$PATH"
export CONF_DIR=/var/run/cloudera-scm-agent/process/104-hive_on_tez-HIVESERVER2/
export CDH_HCAT_HOME=/opt/cloudera/parcels/CDH/lib/hive-hcatalog/
export CDH_HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive
CLASSPATH=":$CONF_DIR"
CLASSPATH="$CLASSPATH:$CONF_DIR/hadoop-conf"
CLASSPATH="$CLASSPATH:$CONF_DIR/hive-conf"
CLASSPATH="$CLASSPATH:$(hadoop classpath)"
CLASSPATH="$CLASSPATH:$CDH_HIVE_HOME/*"
CLASSPATH="$CLASSPATH:$CDH_HIVE_HOME/lib/*"
CLASSPATH="$CLASSPATH:${CDH_HCAT_HOME}/share/webhcat/java-client/hive-webhcat-java-client.jar"
CLASSPATH="$CLASSPATH:${CDH_HCAT_HOME}/share/hcatalog/hive-hcatalog-core.jar"
kinit -kt /cdep/keytabs/hive.keytab hive
hive -e 'create database test;'
javac -cp $CLASSPATH Repro.java
java -cp $CLASSPATH Repro 100 {code}
> Concurrency issue in updateTableColumnStatistics
> ------------------------------------------------
>
> Key: HIVE-28578
> URL: https://issues.apache.org/jira/browse/HIVE-28578
> Project: Hive
> Issue Type: Bug
> Security Level: Public(Viewable by anyone)
> Components: Standalone Metastore
> Reporter: Zsolt Miskolczi
> Assignee: Zsolt Miskolczi
> Priority: Major
> Attachments: Repro.java
>
>
> Summary:
> updateTableColumnStatistics can throw
> SQLIntegrityConstraintViolationException during replication if HA is on and
> two different HMS instance gets the same call but with different engine.
>
> Workaround:
> Update table column statistics in single threaded.
>
> Details:
> updateTableColumnStatistics has a relative long running transaction. In that
> transaction, it validates the actual parameters, queries the metastore db
> against the TABLE_PARAMS that are already stored and makes a decision based
> on that. After this, it uses data nucleus to persist the new statistics.
> From the two HMS instances, one can save the column statistics. And the other
> cannot as the first instance already saved them.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
