[jira] [Resolved] (HIVE-28311) Backward compatibility of java.sql.Date and java.sql.Timestamp in hive-serde
[ https://issues.apache.org/jira/browse/HIVE-28311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar resolved HIVE-28311. --- Fix Version/s: Not Applicable Resolution: Won't Fix > Backward compatibility of java.sql.Date and java.sql.Timestamp in hive-serde > > > Key: HIVE-28311 > URL: https://issues.apache.org/jira/browse/HIVE-28311 > Project: Hive > Issue Type: Bug >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > Fix For: Not Applicable > > > HIVE-20007 introduced {{org.apache.hadoop.hive.common.type.Date}} and > {{org.apache.hadoop.hive.common.type.Timestamp}} to replace {{java.sql.Date}} > and {{{}java.sql.Timestamp{}}}. > It's a huge improvements but it also produce incompatibility issues for > clients without this update. > {code:bash} > Caused by: java.lang.ClassCastException: java.sql.Timestamp cannot be cast to > org.apache.hadoop.hive.common.type.Timestamp > at > org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaTimestampObjectInspector.getPrimitiveWritableObject(JavaTimestampObjectInspector.java:33) > at > org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getTimestamp(PrimitiveObjectInspectorUtils.java:1232) > at > org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TimestampConverter.convert(PrimitiveObjectInspectorConverter.java:291) > at > org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getConstantObjectInspector(ObjectInspectorUtils.java:1397) > at > org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc.getWritableObjectInspector(ExprNodeConstantDesc.java:93) > at > org.apache.hadoop.hive.ql.exec.ExprNodeConstantEvaluator.(ExprNodeConstantEvaluator.java:41) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:49) > at > org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.(ExprNodeGenericFuncEvaluator.java:101) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:58) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:43) > at > org.apache.hadoop.hive.ql.optimizer.ppr.PartExprEvalUtils.prepareExpr(PartExprEvalUtils.java:118) > at > org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prunePartitionNames(PartitionPruner.java:551) > at > org.apache.hadoop.hive.ql.optimizer.ppr.PartitionExpressionForMetastore.filterPartitionsByExpr(PartitionExpressionForMetastore.java:73) > at > org.apache.hadoop.hive.metastore.ObjectStore.getPartitionNamesPrunedByExprNoTxn(ObjectStore.java:3606) > at > org.apache.hadoop.hive.metastore.ObjectStore.access$1000(ObjectStore.java:241) > at > org.apache.hadoop.hive.metastore.ObjectStore$16.getJdoResult(ObjectStore.java:4157) > at > org.apache.hadoop.hive.metastore.ObjectStore$16.getJdoResult(ObjectStore.java:4124) > at > org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3913) > ... 30 more > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HIVE-28311) Backward compatibility of java.sql.Date and java.sql.Timestamp in hive-serde
[ https://issues.apache.org/jira/browse/HIVE-28311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17860348#comment-17860348 ] Wechar edited comment on HIVE-28311 at 6/27/24 6:32 AM: [~zabetak] I found this issue in hive client 2.3.8 and server 3.1.2, and this patch can solve this issue. But the master has upgrade to Kryo5 in HIVE-22944, then hive client 2.x and 3.x serialization bytes can not be deserialized successfully in HMS 4.x. Luckily after applying patch HIVE-22944 in hive 2.x, it works well. I think we can upgrade kryo in low version clients to solve this issue. And we can close this ticket now. was (Author: wechar): [~zabetak] I found this issue in hive client 2.3.8 and server 3.1.2, and this patch can solve this issue. But the master has upgrade to Kryo5 in HIVE-22944, then hive client 2.x and 3.x serialization bytes can not be deserialized successfully in HMS 4.x. Luckily after applying patch HIVE-22944 in hive 2.x, it works well. I think we can upgrade kryo in low version clients to solve this issue. > Backward compatibility of java.sql.Date and java.sql.Timestamp in hive-serde > > > Key: HIVE-28311 > URL: https://issues.apache.org/jira/browse/HIVE-28311 > Project: Hive > Issue Type: Bug >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > > HIVE-20007 introduced {{org.apache.hadoop.hive.common.type.Date}} and > {{org.apache.hadoop.hive.common.type.Timestamp}} to replace {{java.sql.Date}} > and {{{}java.sql.Timestamp{}}}. > It's a huge improvements but it also produce incompatibility issues for > clients without this update. > {code:bash} > Caused by: java.lang.ClassCastException: java.sql.Timestamp cannot be cast to > org.apache.hadoop.hive.common.type.Timestamp > at > org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaTimestampObjectInspector.getPrimitiveWritableObject(JavaTimestampObjectInspector.java:33) > at > org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getTimestamp(PrimitiveObjectInspectorUtils.java:1232) > at > org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TimestampConverter.convert(PrimitiveObjectInspectorConverter.java:291) > at > org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getConstantObjectInspector(ObjectInspectorUtils.java:1397) > at > org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc.getWritableObjectInspector(ExprNodeConstantDesc.java:93) > at > org.apache.hadoop.hive.ql.exec.ExprNodeConstantEvaluator.(ExprNodeConstantEvaluator.java:41) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:49) > at > org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.(ExprNodeGenericFuncEvaluator.java:101) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:58) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:43) > at > org.apache.hadoop.hive.ql.optimizer.ppr.PartExprEvalUtils.prepareExpr(PartExprEvalUtils.java:118) > at > org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prunePartitionNames(PartitionPruner.java:551) > at > org.apache.hadoop.hive.ql.optimizer.ppr.PartitionExpressionForMetastore.filterPartitionsByExpr(PartitionExpressionForMetastore.java:73) > at > org.apache.hadoop.hive.metastore.ObjectStore.getPartitionNamesPrunedByExprNoTxn(ObjectStore.java:3606) > at > org.apache.hadoop.hive.metastore.ObjectStore.access$1000(ObjectStore.java:241) > at > org.apache.hadoop.hive.metastore.ObjectStore$16.getJdoResult(ObjectStore.java:4157) > at > org.apache.hadoop.hive.metastore.ObjectStore$16.getJdoResult(ObjectStore.java:4124) > at > org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3913) > ... 30 more > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-28311) Backward compatibility of java.sql.Date and java.sql.Timestamp in hive-serde
[ https://issues.apache.org/jira/browse/HIVE-28311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17860348#comment-17860348 ] Wechar commented on HIVE-28311: --- [~zabetak] I found this issue in hive client 2.3.8 and server 3.1.2, and this patch can solve this issue. But the master has upgrade to Kryo5 in HIVE-22944, then hive client 2.x and 3.x serialization bytes can not be deserialized successfully in HMS 4.x. Luckily after applying patch HIVE-22944 in hive 2.x, it works well. I think we can upgrade kryo in low version clients to solve this issue. > Backward compatibility of java.sql.Date and java.sql.Timestamp in hive-serde > > > Key: HIVE-28311 > URL: https://issues.apache.org/jira/browse/HIVE-28311 > Project: Hive > Issue Type: Bug >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > > HIVE-20007 introduced {{org.apache.hadoop.hive.common.type.Date}} and > {{org.apache.hadoop.hive.common.type.Timestamp}} to replace {{java.sql.Date}} > and {{{}java.sql.Timestamp{}}}. > It's a huge improvements but it also produce incompatibility issues for > clients without this update. > {code:bash} > Caused by: java.lang.ClassCastException: java.sql.Timestamp cannot be cast to > org.apache.hadoop.hive.common.type.Timestamp > at > org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaTimestampObjectInspector.getPrimitiveWritableObject(JavaTimestampObjectInspector.java:33) > at > org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getTimestamp(PrimitiveObjectInspectorUtils.java:1232) > at > org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TimestampConverter.convert(PrimitiveObjectInspectorConverter.java:291) > at > org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getConstantObjectInspector(ObjectInspectorUtils.java:1397) > at > org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc.getWritableObjectInspector(ExprNodeConstantDesc.java:93) > at > org.apache.hadoop.hive.ql.exec.ExprNodeConstantEvaluator.(ExprNodeConstantEvaluator.java:41) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:49) > at > org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.(ExprNodeGenericFuncEvaluator.java:101) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:58) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:43) > at > org.apache.hadoop.hive.ql.optimizer.ppr.PartExprEvalUtils.prepareExpr(PartExprEvalUtils.java:118) > at > org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prunePartitionNames(PartitionPruner.java:551) > at > org.apache.hadoop.hive.ql.optimizer.ppr.PartitionExpressionForMetastore.filterPartitionsByExpr(PartitionExpressionForMetastore.java:73) > at > org.apache.hadoop.hive.metastore.ObjectStore.getPartitionNamesPrunedByExprNoTxn(ObjectStore.java:3606) > at > org.apache.hadoop.hive.metastore.ObjectStore.access$1000(ObjectStore.java:241) > at > org.apache.hadoop.hive.metastore.ObjectStore$16.getJdoResult(ObjectStore.java:4157) > at > org.apache.hadoop.hive.metastore.ObjectStore$16.getJdoResult(ObjectStore.java:4124) > at > org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3913) > ... 30 more > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28338) Client connection count is not correct in HiveMetaStore#close
Wechar created HIVE-28338: - Summary: Client connection count is not correct in HiveMetaStore#close Key: HIVE-28338 URL: https://issues.apache.org/jira/browse/HIVE-28338 Project: Hive Issue Type: Bug Components: Hive Reporter: Wechar Assignee: Wechar HIVE-24349 introduced a bug in {{HiveMetaStoreClient}} for embedded metastore, where the log would print negative connection counts. *Root Cause* Connection count is only used in remote metastore, we do not need decrease connection counts when transport is null. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28307) Fast return when database exists and IF NOT EXISTS clause is enabled
[ https://issues.apache.org/jira/browse/HIVE-28307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-28307: -- Description: When use beeline for testing purpose: {code:bash} beeline -u jdbc:hive2:// {code} {{CREATE TABLE IF NOT EXISTS}} will not print {{{}AlreadyExistsException{}}}, but {{CREATE DATABASE IF NOT EXISTS}} will. The error log may be confused because the sql actually execute successfully. The root cause is that {{SemanticAnalyzer}} will fast-return if table exists: [https://github.com/apache/hive/blob/98d9d22398370f817fe64449368671b978fff096/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L14163-L14174] was: When use beeline for testing purpose: {code:bash} beeline -u jdbc:hive2:// {code} {{CREATE TABLE IF NOT EXISTS}} will not print {{AlreadyExistsException}}, but {{CREATE DATABASE IF NOT EXISTS}} will. The root cause is that {{SemanticAnalyzer}} will fast-return if table exists: https://github.com/apache/hive/blob/98d9d22398370f817fe64449368671b978fff096/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L14163-L14174 > Fast return when database exists and IF NOT EXISTS clause is enabled > > > Key: HIVE-28307 > URL: https://issues.apache.org/jira/browse/HIVE-28307 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > Attachments: create_database.png, create_table.png > > > When use beeline for testing purpose: > {code:bash} > beeline -u jdbc:hive2:// > {code} > {{CREATE TABLE IF NOT EXISTS}} will not print {{{}AlreadyExistsException{}}}, > but {{CREATE DATABASE IF NOT EXISTS}} will. > The error log may be confused because the sql actually execute successfully. > > The root cause is that {{SemanticAnalyzer}} will fast-return if table exists: > [https://github.com/apache/hive/blob/98d9d22398370f817fe64449368671b978fff096/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L14163-L14174] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28307) Fast return when database exists and IF NOT EXISTS clause is enabled
[ https://issues.apache.org/jira/browse/HIVE-28307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-28307: -- Attachment: create_table.png > Fast return when database exists and IF NOT EXISTS clause is enabled > > > Key: HIVE-28307 > URL: https://issues.apache.org/jira/browse/HIVE-28307 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > Attachments: create_database.png, create_table.png > > > When use beeline for testing purpose: > {code:bash} > beeline -u jdbc:hive2:// > {code} > {{CREATE TABLE IF NOT EXISTS}} will not print {{AlreadyExistsException}}, but > {{CREATE DATABASE IF NOT EXISTS}} will. > The root cause is that {{SemanticAnalyzer}} will fast-return if table exists: > https://github.com/apache/hive/blob/98d9d22398370f817fe64449368671b978fff096/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L14163-L14174 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28307) Fast return when database exists and IF NOT EXISTS clause is enabled
[ https://issues.apache.org/jira/browse/HIVE-28307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-28307: -- Attachment: create_database.png > Fast return when database exists and IF NOT EXISTS clause is enabled > > > Key: HIVE-28307 > URL: https://issues.apache.org/jira/browse/HIVE-28307 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > Attachments: create_database.png > > > When use beeline for testing purpose: > {code:bash} > beeline -u jdbc:hive2:// > {code} > {{CREATE TABLE IF NOT EXISTS}} will not print {{AlreadyExistsException}}, but > {{CREATE DATABASE IF NOT EXISTS}} will. > The root cause is that {{SemanticAnalyzer}} will fast-return if table exists: > https://github.com/apache/hive/blob/98d9d22398370f817fe64449368671b978fff096/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L14163-L14174 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28307) Fast return when database exists and IF NOT EXISTS clause is enabled
[ https://issues.apache.org/jira/browse/HIVE-28307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-28307: -- Summary: Fast return when database exists and IF NOT EXISTS clause is enabled (was: Enable IF NOT EXISTS in CREATE DATABASE statements.) > Fast return when database exists and IF NOT EXISTS clause is enabled > > > Key: HIVE-28307 > URL: https://issues.apache.org/jira/browse/HIVE-28307 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > > When use beeline for testing purpose: > {code:bash} > beeline -u jdbc:hive2:// > {code} > {{CREATE TABLE IF NOT EXISTS}} will not print {{AlreadyExistsException}}, but > {{CREATE DATABASE IF NOT EXISTS}} will. > The root cause is that {{SemanticAnalyzer}} will fast-return if table exists: > https://github.com/apache/hive/blob/98d9d22398370f817fe64449368671b978fff096/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L14163-L14174 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28307) Enable IF NOT EXISTS in CREATE DATABASE statements.
[ https://issues.apache.org/jira/browse/HIVE-28307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-28307: -- Description: When use beeline for testing purpose: {code:bash} beeline -u jdbc:hive2:// {code} {{CREATE TABLE IF NOT EXISTS}} will not print {{AlreadyExistsException}}, but {{CREATE DATABASE IF NOT EXISTS}} will. The root cause is that {{SemanticAnalyzer}} will fast-return if table exists: https://github.com/apache/hive/blob/98d9d22398370f817fe64449368671b978fff096/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L14163-L14174 was:If a database already exists, we should not throw exception if we try to create it with {{IF NOT EXISTS}} clause. > Enable IF NOT EXISTS in CREATE DATABASE statements. > --- > > Key: HIVE-28307 > URL: https://issues.apache.org/jira/browse/HIVE-28307 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > > When use beeline for testing purpose: > {code:bash} > beeline -u jdbc:hive2:// > {code} > {{CREATE TABLE IF NOT EXISTS}} will not print {{AlreadyExistsException}}, but > {{CREATE DATABASE IF NOT EXISTS}} will. > The root cause is that {{SemanticAnalyzer}} will fast-return if table exists: > https://github.com/apache/hive/blob/98d9d22398370f817fe64449368671b978fff096/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L14163-L14174 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28311) Backward compatibility of java.sql.Date and java.sql.Timestamp in hive-serde
[ https://issues.apache.org/jira/browse/HIVE-28311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-28311: -- Description: HIVE-20007 introduced {{org.apache.hadoop.hive.common.type.Date}} and {{org.apache.hadoop.hive.common.type.Timestamp}} to replace {{java.sql.Date}} and {{{}java.sql.Timestamp{}}}. It's a huge improvements but it also produce incompatibility issues for clients without this update. {code:bash} Caused by: java.lang.ClassCastException: java.sql.Timestamp cannot be cast to org.apache.hadoop.hive.common.type.Timestamp at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaTimestampObjectInspector.getPrimitiveWritableObject(JavaTimestampObjectInspector.java:33) at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getTimestamp(PrimitiveObjectInspectorUtils.java:1232) at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TimestampConverter.convert(PrimitiveObjectInspectorConverter.java:291) at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getConstantObjectInspector(ObjectInspectorUtils.java:1397) at org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc.getWritableObjectInspector(ExprNodeConstantDesc.java:93) at org.apache.hadoop.hive.ql.exec.ExprNodeConstantEvaluator.(ExprNodeConstantEvaluator.java:41) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:49) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.(ExprNodeGenericFuncEvaluator.java:101) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:58) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:43) at org.apache.hadoop.hive.ql.optimizer.ppr.PartExprEvalUtils.prepareExpr(PartExprEvalUtils.java:118) at org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prunePartitionNames(PartitionPruner.java:551) at org.apache.hadoop.hive.ql.optimizer.ppr.PartitionExpressionForMetastore.filterPartitionsByExpr(PartitionExpressionForMetastore.java:73) at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionNamesPrunedByExprNoTxn(ObjectStore.java:3606) at org.apache.hadoop.hive.metastore.ObjectStore.access$1000(ObjectStore.java:241) at org.apache.hadoop.hive.metastore.ObjectStore$16.getJdoResult(ObjectStore.java:4157) at org.apache.hadoop.hive.metastore.ObjectStore$16.getJdoResult(ObjectStore.java:4124) at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3913) ... 30 more {code} > Backward compatibility of java.sql.Date and java.sql.Timestamp in hive-serde > > > Key: HIVE-28311 > URL: https://issues.apache.org/jira/browse/HIVE-28311 > Project: Hive > Issue Type: Bug >Reporter: Wechar >Assignee: Wechar >Priority: Major > > HIVE-20007 introduced {{org.apache.hadoop.hive.common.type.Date}} and > {{org.apache.hadoop.hive.common.type.Timestamp}} to replace {{java.sql.Date}} > and {{{}java.sql.Timestamp{}}}. > It's a huge improvements but it also produce incompatibility issues for > clients without this update. > {code:bash} > Caused by: java.lang.ClassCastException: java.sql.Timestamp cannot be cast to > org.apache.hadoop.hive.common.type.Timestamp > at > org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaTimestampObjectInspector.getPrimitiveWritableObject(JavaTimestampObjectInspector.java:33) > at > org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getTimestamp(PrimitiveObjectInspectorUtils.java:1232) > at > org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TimestampConverter.convert(PrimitiveObjectInspectorConverter.java:291) > at > org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getConstantObjectInspector(ObjectInspectorUtils.java:1397) > at > org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc.getWritableObjectInspector(ExprNodeConstantDesc.java:93) > at > org.apache.hadoop.hive.ql.exec.ExprNodeConstantEvaluator.(ExprNodeConstantEvaluator.java:41) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:49) > at > org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.(ExprNodeGenericFuncEvaluator.java:101) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:58) > at > org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:43) > at > org.apache.hadoop.h
[jira] [Created] (HIVE-28311) Backward compatibility of java.sql.Date and java.sql.Timestamp in hive-serde
Wechar created HIVE-28311: - Summary: Backward compatibility of java.sql.Date and java.sql.Timestamp in hive-serde Key: HIVE-28311 URL: https://issues.apache.org/jira/browse/HIVE-28311 Project: Hive Issue Type: Bug Reporter: Wechar Assignee: Wechar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28307) Enable IF NOT EXISTS in CREATE DATABASE statements.
[ https://issues.apache.org/jira/browse/HIVE-28307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-28307: -- Description: If a database already exists, we should not throw exception if we try to create it with {{IF NOT EXISTS}} clause. > Enable IF NOT EXISTS in CREATE DATABASE statements. > --- > > Key: HIVE-28307 > URL: https://issues.apache.org/jira/browse/HIVE-28307 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > > If a database already exists, we should not throw exception if we try to > create it with {{IF NOT EXISTS}} clause. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28307) Enable IF NOT EXISTS in CREATE DATABASE statements.
Wechar created HIVE-28307: - Summary: Enable IF NOT EXISTS in CREATE DATABASE statements. Key: HIVE-28307 URL: https://issues.apache.org/jira/browse/HIVE-28307 Project: Hive Issue Type: Bug Components: Hive Reporter: Wechar Assignee: Wechar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-28297) Unify table pattern definition in Hive MetaStore
[ https://issues.apache.org/jira/browse/HIVE-28297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-28297 started by Wechar. - > Unify table pattern definition in Hive MetaStore > > > Key: HIVE-28297 > URL: https://issues.apache.org/jira/browse/HIVE-28297 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > > *Current Situation* > There are two cases for table pattern in Hive: > - Case 1: Wildcards ('*' and '|) > It's replaced by SQL type like expressions with '%' and '_' since Hive 4.0.0, > details in HIVE-23359. > DDL documents: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Show > - Case 2: Java regular expression > It's declared in hive code: > [https://github.com/apache/hive/blob/45867be6cb5308566e4cf16c7b4cf8081085b58c/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1882-L1894], > and also match the jdo {{matches(String pattern)}} method: > [https://www.datanucleus.org/products/accessplatform/jdo/query.html] > *Issues* > The different definitions cause the misuse of table pattern, in some codes it > use * to represent match everything, while it use .* in other codes. > The misuse would cause the wrong result because both client and server side > of HMS need a trick handle for '*' with code like: > {code:java} > patterns.split("\\|") > pattern.replaceAll("\\*", ".*")) > {code} > *Action* > We should honor the definition in HMS of Case 2, because Case 1 is just the > frontend of SQL, and it has been replaced. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28297) Unify table pattern definition in Hive MetaStore
[ https://issues.apache.org/jira/browse/HIVE-28297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-28297: -- Description: *Current Situation* There are two cases for table pattern in Hive: - Case 1: Wildcards ('*' and '|) It's replaced by SQL type like expressions with '%' and '_' since Hive 4.0.0, details in HIVE-23359. DDL documents: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Show - Case 2: Java regular expression It's declared in hive code: [https://github.com/apache/hive/blob/45867be6cb5308566e4cf16c7b4cf8081085b58c/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1882-L1894], and also match the jdo {{matches(String pattern)}} method: [https://www.datanucleus.org/products/accessplatform/jdo/query.html] *Issues* The different definitions cause the misuse of table pattern, in some codes it use * to represent match everything, while it use .* in other codes. The misuse would cause the wrong result because both client and server side of HMS need a trick handle for '*' with code like: {code:java} patterns.split("\\|") pattern.replaceAll("\\*", ".*")) {code} *Action* We should honor the definition in HMS of Case 2, because Case 1 is just the frontend of SQL, and it has been replaced. was: *Current Situation* There are two cases for table pattern in Hive: - Case 1: Wildcards ('*' and '|) It's replaced by SQL type like expressions with '%' and '_' since Hive 4.0.0, details in HIVE-23359 - Case 2: Java regular expression It's declared in hive code: [https://github.com/apache/hive/blob/45867be6cb5308566e4cf16c7b4cf8081085b58c/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1882-L1894], and also match the jdo {{matches(String pattern)}} method: [https://www.datanucleus.org/products/accessplatform/jdo/query.html] *Issues* The different definitions cause the misuse of table pattern, in some codes it use * to represent match everything, while it use .* in other codes. The misuse would cause the wrong result because both client and server side of HMS need a trick handle for '*' with code like: {code:java} patterns.split("\\|") pattern.replaceAll("\\*", ".*")) {code} *Action* We should honor the definition in HMS of Case 2, because Case 1 is just the frontend of SQL, and it has been replaced. > Unify table pattern definition in Hive MetaStore > > > Key: HIVE-28297 > URL: https://issues.apache.org/jira/browse/HIVE-28297 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > > *Current Situation* > There are two cases for table pattern in Hive: > - Case 1: Wildcards ('*' and '|) > It's replaced by SQL type like expressions with '%' and '_' since Hive 4.0.0, > details in HIVE-23359. > DDL documents: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Show > - Case 2: Java regular expression > It's declared in hive code: > [https://github.com/apache/hive/blob/45867be6cb5308566e4cf16c7b4cf8081085b58c/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1882-L1894], > and also match the jdo {{matches(String pattern)}} method: > [https://www.datanucleus.org/products/accessplatform/jdo/query.html] > *Issues* > The different definitions cause the misuse of table pattern, in some codes it > use * to represent match everything, while it use .* in other codes. > The misuse would cause the wrong result because both client and server side > of HMS need a trick handle for '*' with code like: > {code:java} > patterns.split("\\|") > pattern.replaceAll("\\*", ".*")) > {code} > *Action* > We should honor the definition in HMS of Case 2, because Case 1 is just the > frontend of SQL, and it has been replaced. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28297) Unify table pattern definition in Hive MetaStore
Wechar created HIVE-28297: - Summary: Unify table pattern definition in Hive MetaStore Key: HIVE-28297 URL: https://issues.apache.org/jira/browse/HIVE-28297 Project: Hive Issue Type: Improvement Components: Hive Reporter: Wechar Assignee: Wechar *Current Situation* There are two cases for table pattern in Hive: - Case 1: Wildcards ('*' and '|) It's replaced by SQL type like expressions with '%' and '_' since Hive 4.0.0, details in HIVE-23359 - Case 2: Java regular expression It's declared in hive code: [https://github.com/apache/hive/blob/45867be6cb5308566e4cf16c7b4cf8081085b58c/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1882-L1894], and also match the jdo {{matches(String pattern)}} method: [https://www.datanucleus.org/products/accessplatform/jdo/query.html] *Issues* The different definitions cause the misuse of table pattern, in some codes it use * to represent match everything, while it use .* in other codes. The misuse would cause the wrong result because both client and server side of HMS need a trick handle for '*' with code like: {code:java} patterns.split("\\|") pattern.replaceAll("\\*", ".*")) {code} *Action* We should honor the definition in HMS of Case 2, because Case 1 is just the frontend of SQL, and it has been replaced. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28292) Optimize SHOW TABLES|VIEWS statements
[ https://issues.apache.org/jira/browse/HIVE-28292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-28292: -- Description: There are some improvement points in {{SHOW TABLES}} statement: 1. pattern is not pushed to metastore server 2. get table objects even if only table name and table type is needed. 3. index {{UNIQUETABLE(TBL_NAME, DB_ID)}} in {{TBLS}} table is not efficient enough compared to index {{(DB_ID, TBL_NAME)}} when querying tables by name pattern. > Optimize SHOW TABLES|VIEWS statements > - > > Key: HIVE-28292 > URL: https://issues.apache.org/jira/browse/HIVE-28292 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > > There are some improvement points in {{SHOW TABLES}} statement: > 1. pattern is not pushed to metastore server > 2. get table objects even if only table name and table type is needed. > 3. index {{UNIQUETABLE(TBL_NAME, DB_ID)}} in {{TBLS}} table is not efficient > enough compared to index {{(DB_ID, TBL_NAME)}} when querying tables by name > pattern. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-28292) Optimize SHOW TABLES|VIEWS statements
[ https://issues.apache.org/jira/browse/HIVE-28292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-28292 started by Wechar. - > Optimize SHOW TABLES|VIEWS statements > - > > Key: HIVE-28292 > URL: https://issues.apache.org/jira/browse/HIVE-28292 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > > There are some improvement points in {{SHOW TABLES}} statement: > 1. pattern is not pushed to metastore server > 2. get table objects even if only table name and table type is needed. > 3. index {{UNIQUETABLE(TBL_NAME, DB_ID)}} in {{TBLS}} table is not efficient > enough compared to index {{(DB_ID, TBL_NAME)}} when querying tables by name > pattern. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28292) Optimize SHOW TABLES|VIEWS statements
Wechar created HIVE-28292: - Summary: Optimize SHOW TABLES|VIEWS statements Key: HIVE-28292 URL: https://issues.apache.org/jira/browse/HIVE-28292 Project: Hive Issue Type: Improvement Components: Hive Reporter: Wechar Assignee: Wechar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-28279) Output the database name for SHOW EXTENDED TABLES statement
[ https://issues.apache.org/jira/browse/HIVE-28279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar resolved HIVE-28279. --- Fix Version/s: Not Applicable Resolution: Won't Do > Output the database name for SHOW EXTENDED TABLES statement > --- > > Key: HIVE-28279 > URL: https://issues.apache.org/jira/browse/HIVE-28279 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > Fix For: Not Applicable > > > HIVE-21301 introduced {{SHOW EXTENDED TABLES}} statement which output table > name and table type while listing tables in a database. > In this patch, we aim to add a new output filed for database name with > following reasons: > 1. database name in {{SHOW EXTENDED TABLES}} statement is optional, output > the database is informal in this case. > 2. when statistic table names and database names by this statement for list > of databases, the output result including database name is much more helpful. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work stopped] (HIVE-28279) Output the database name for SHOW EXTENDED TABLES statement
[ https://issues.apache.org/jira/browse/HIVE-28279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-28279 stopped by Wechar. - > Output the database name for SHOW EXTENDED TABLES statement > --- > > Key: HIVE-28279 > URL: https://issues.apache.org/jira/browse/HIVE-28279 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > > HIVE-21301 introduced {{SHOW EXTENDED TABLES}} statement which output table > name and table type while listing tables in a database. > In this patch, we aim to add a new output filed for database name with > following reasons: > 1. database name in {{SHOW EXTENDED TABLES}} statement is optional, output > the database is informal in this case. > 2. when statistic table names and database names by this statement for list > of databases, the output result including database name is much more helpful. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-28279) Output the database name for SHOW EXTENDED TABLES statement
[ https://issues.apache.org/jira/browse/HIVE-28279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-28279 started by Wechar. - > Output the database name for SHOW EXTENDED TABLES statement > --- > > Key: HIVE-28279 > URL: https://issues.apache.org/jira/browse/HIVE-28279 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > > HIVE-21301 introduced {{SHOW EXTENDED TABLES}} statement which output table > name and table type while listing tables in a database. > In this patch, we aim to add a new output filed for database name with > following reasons: > 1. database name in {{SHOW EXTENDED TABLES}} statement is optional, output > the database is informal in this case. > 2. when statistic table names and database names by this statement for list > of databases, the output result including database name is much more helpful. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28279) Output the database name for SHOW EXTENDED TABLES statement
Wechar created HIVE-28279: - Summary: Output the database name for SHOW EXTENDED TABLES statement Key: HIVE-28279 URL: https://issues.apache.org/jira/browse/HIVE-28279 Project: Hive Issue Type: Task Components: Hive Reporter: Wechar Assignee: Wechar HIVE-21301 introduced {{SHOW EXTENDED TABLES}} statement which output table name and table type while listing tables in a database. In this patch, we aim to add a new output filed for database name with following reasons: 1. database name in {{SHOW EXTENDED TABLES}} statement is optional, output the database is informal in this case. 2. when statistic table names and database names by this statement for list of databases, the output result including database name is much more helpful. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28270) Fix missing partition paths bug on drop_database
Wechar created HIVE-28270: - Summary: Fix missing partition paths bug on drop_database Key: HIVE-28270 URL: https://issues.apache.org/jira/browse/HIVE-28270 Project: Hive Issue Type: Bug Components: Hive Reporter: Wechar Assignee: Wechar In {{HMSHandler#drop_database_core}}, it needs to collect all partition paths that were not in the subdirectory of the table path, but now it only fetch the last batch of paths. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28239) Fix bug on HMSHandler#checkLimitNumberOfPartitions
Wechar created HIVE-28239: - Summary: Fix bug on HMSHandler#checkLimitNumberOfPartitions Key: HIVE-28239 URL: https://issues.apache.org/jira/browse/HIVE-28239 Project: Hive Issue Type: Bug Components: Hive Reporter: Wechar Assignee: Wechar {{HMSHandler#checkLimitNumberOfPartitions}} should not compare request size, which can cause the incorrect limit check. Assume that HMS configure {{metastore.limit.partition.request}} as 100, the client calls {{get_partitions_by_filter}} with maxParts as 101, and the matching partition number is 50, in this case the HMS server should not throw MetaException by partition limit check. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-28024) Support sources profile for hive metastore modules
[ https://issues.apache.org/jira/browse/HIVE-28024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar resolved HIVE-28024. --- Fix Version/s: 4.0.0 Resolution: Won't Fix > Support sources profile for hive metastore modules > -- > > Key: HIVE-28024 > URL: https://issues.apache.org/jira/browse/HIVE-28024 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0-beta-1 >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Maven profile sources in Hive supports to generate source jar for all > submodules. Since Hive Metastore modules are separated from hive parent > module, it would be convenient to add this profile in standalone-metastore > module. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-28024) Support sources profile for hive metastore modules
[ https://issues.apache.org/jira/browse/HIVE-28024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842050#comment-17842050 ] Wechar commented on HIVE-28024: --- Thanks [~zabetak], close this jira now. > Support sources profile for hive metastore modules > -- > > Key: HIVE-28024 > URL: https://issues.apache.org/jira/browse/HIVE-28024 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0-beta-1 >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > > Maven profile sources in Hive supports to generate source jar for all > submodules. Since Hive Metastore modules are separated from hive parent > module, it would be convenient to add this profile in standalone-metastore > module. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-28024) Support sources profile for hive metastore modules
[ https://issues.apache.org/jira/browse/HIVE-28024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841307#comment-17841307 ] Wechar commented on HIVE-28024: --- Because the root pom.xml in Hive defines the [sources|https://github.com/apache/hive/blob/4614ce72a7f366674d89a3a78f687e419400cb89/pom.xml#L1940] profile explicitly, so I think it's also needed in metastore modules. But since apache-release profile is available, do we still need sources profile in Hive? > Support sources profile for hive metastore modules > -- > > Key: HIVE-28024 > URL: https://issues.apache.org/jira/browse/HIVE-28024 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0-beta-1 >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > > Maven profile sources in Hive supports to generate source jar for all > submodules. Since Hive Metastore modules are separated from hive parent > module, it would be convenient to add this profile in standalone-metastore > module. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28219) Support drop partitions by names in IMetaStoreClient
Wechar created HIVE-28219: - Summary: Support drop partitions by names in IMetaStoreClient Key: HIVE-28219 URL: https://issues.apache.org/jira/browse/HIVE-28219 Project: Hive Issue Type: New Feature Components: Hive Reporter: Wechar Assignee: Wechar In thrift api definition, HMS support to drop partitions by both partition names and expressions: {code:bash} // hive_metastore.thrift DropPartitionsResult drop_partitions_req(1: DropPartitionsRequest req) throws(1:NoSuchObjectException o1, 2:MetaException o2) // Request type for drop_partitions_req struct DropPartitionsRequest { // ... 3: required RequestPartsSpec parts, // ... } union RequestPartsSpec { 1: list names; 2: list exprs; } {code} But current api in `IMetaStoreClient` only support drop partitions by expressions, we should add new api to support drop partitions by names. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-28205) Implement direct sql for get_partitions_ps_with_auth api
[ https://issues.apache.org/jira/browse/HIVE-28205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-28205 started by Wechar. - > Implement direct sql for get_partitions_ps_with_auth api > > > Key: HIVE-28205 > URL: https://issues.apache.org/jira/browse/HIVE-28205 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28205) Implement direct sql for get_partitions_ps_with_auth api
Wechar created HIVE-28205: - Summary: Implement direct sql for get_partitions_ps_with_auth api Key: HIVE-28205 URL: https://issues.apache.org/jira/browse/HIVE-28205 Project: Hive Issue Type: Improvement Components: Hive Reporter: Wechar Assignee: Wechar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28200) Improve get_partitions_by_filter/expr when partition limit enabled
Wechar created HIVE-28200: - Summary: Improve get_partitions_by_filter/expr when partition limit enabled Key: HIVE-28200 URL: https://issues.apache.org/jira/browse/HIVE-28200 Project: Hive Issue Type: Improvement Components: Hive Reporter: Wechar Assignee: Wechar When {{hive.metastore.limit.partition.request}} is configured, HMS would get the matching partition counts before get the real partition objects. The count could be a slow query if the input filter or expr is too complex. In this case, such slow filter will be executed in both counting partition numbers and fetching real partition objects, which harms the performance and backend DBMS. We can make an improvement by getting matched partition names firstly, and then check limit through the size of partition names, and finally get the partitions by the partition names. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28096) Fix the mismatched function names in HMSHandler audit log
[ https://issues.apache.org/jira/browse/HIVE-28096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-28096: -- Summary: Fix the mismatched function names in HMSHandler audit log (was: Fix the mismatched function names in HMSHandler ) > Fix the mismatched function names in HMSHandler audit log > - > > Key: HIVE-28096 > URL: https://issues.apache.org/jira/browse/HIVE-28096 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0-alpha-2 >Reporter: Wechar >Assignee: Wechar >Priority: Major > > Some function names in startFunction and endFunction are inconsistent, which > would cause the counter and api cost metrics wrong. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28096) Fix the mismatched function names in HMSHandler
Wechar created HIVE-28096: - Summary: Fix the mismatched function names in HMSHandler Key: HIVE-28096 URL: https://issues.apache.org/jira/browse/HIVE-28096 Project: Hive Issue Type: Bug Components: Hive Affects Versions: 4.0.0-alpha-2 Reporter: Wechar Assignee: Wechar Some function names in startFunction and endFunction are inconsistent, which would cause the counter and api cost metrics wrong. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28062) Optimize get_partitions_by_names in direct sql
[ https://issues.apache.org/jira/browse/HIVE-28062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-28062: -- Summary: Optimize get_partitions_by_names in direct sql (was: Optimize get_partitions_by_names direct sql) > Optimize get_partitions_by_names in direct sql > -- > > Key: HIVE-28062 > URL: https://issues.apache.org/jira/browse/HIVE-28062 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0-beta-1 >Reporter: Wechar >Assignee: Wechar >Priority: Major > > Currently {{MetaStoreDirectSql#getPartitionsViaPartNames}} would first get > partition id through partition name in each batch, which was unnecessary > because we can get all fields through partition name directly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28062) Optimize get_partitions_by_names direct sql
Wechar created HIVE-28062: - Summary: Optimize get_partitions_by_names direct sql Key: HIVE-28062 URL: https://issues.apache.org/jira/browse/HIVE-28062 Project: Hive Issue Type: Improvement Components: Hive Affects Versions: 4.0.0-beta-1 Reporter: Wechar Assignee: Wechar Currently {{MetaStoreDirectSql#getPartitionsViaPartNames}} would first get partition id through partition name in each batch, which was unnecessary because we can get all fields through partition name directly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28038) Disable fallback to jdo for DeadlineException
Wechar created HIVE-28038: - Summary: Disable fallback to jdo for DeadlineException Key: HIVE-28038 URL: https://issues.apache.org/jira/browse/HIVE-28038 Project: Hive Issue Type: Improvement Components: Hive Affects Versions: 4.0.0-beta-1 Reporter: Wechar Assignee: Wechar When {{DeadlineException}} occurs, the call should be terminated immediately instead of falling back to jdo query, because some jdo query does not take use of {{{}Deadline.checkTimeout(){}}}. {*}How to Reproduce{*}: Step 1: start a metastore server with metastore config: {code:xml} hive.metastore.client.socket.timeout 60s {code} Step 2: Create a table with large partitions Step 3: set hive client socket timeout same as server, and call drop_table to drop this table {code:java} conf.set("hive.metastore.client.socket.timeout", "60s"); conf.set("hive.metastore.failure.retries", "0"); {code} The direct sql query would throw DeadlineException while dropping partitions in batch, but it would fall back to jdo query to drop partitions for each batch, we can find clues from the server log: {code:bash} 2024-01-26T08:52:14,877 INFO [Metastore-Handler-Pool: Thread-54]: HiveMetaStore.audit (HMSHandler.java:logAuditEvent(207)) - ugi=root ip=10.89.0.1cmd=source:10.89.0.1 get_all_functions 2024-01-26T08:52:14,879 INFO [Metastore-Handler-Pool: Thread-54]: metastore.HMSHandler (HMSHandler.java:newRawStoreForConf(600)) - Opening raw store with implementation class: org.apache.hadoop.hive.metastore.ObjectStore 2024-01-26T08:52:14,879 INFO [Metastore-Handler-Pool: Thread-54]: metastore.PersistenceManagerProvider (PersistenceManagerProvider.java:correctAutoStartMechanism(577)) - Configuration datanucleus.autoStartMechanismMode is not set . Defaulting to 'ignored' 2024-01-26T08:52:14,879 INFO [Metastore-Handler-Pool: Thread-54]: metastore.ObjectStore (ObjectStore.java:shutdown(568)) - RawStore: org.apache.hadoop.hive.metastore.ObjectStore@53c9fbcd, with PersistenceManager: null will be shu tdown 2024-01-26T08:52:14,881 INFO [Metastore-Handler-Pool: Thread-54]: metastore.ObjectStore (ObjectStore.java:initialize(430)) - RawStore: org.apache.hadoop.hive.metastore.ObjectStore@53c9fbcd, with PersistenceManager: org.datanucleu s.api.jdo.JDOPersistenceManager@586cd877 created in the thread with id: 54 2024-01-26T08:52:14,890 INFO [Metastore-Handler-Pool: Thread-54]: metastore.HMSHandler (HMSHandler.java:getMSForConf(587)) - Created RawStore: org.apache.hadoop.hive.metastore.ObjectStore@53c9fbcd 2024-01-26T08:52:14,913 INFO [Metastore-Handler-Pool: Thread-54]: HiveMetaStore.audit (HMSHandler.java:logAuditEvent(207)) - ugi=root ip=10.89.0.1cmd=source:10.89.0.1 get_table : tbl=hive.default.test_tbl 2024-01-26T08:52:15,069 INFO [Metastore-Handler-Pool: Thread-54]: metastore.HMSHandler (HMSHandler.java:getTableInternal(3582)) - Skipping translation for processor with null 2024-01-26T08:52:15,094 INFO [Metastore-Handler-Pool: Thread-54]: HiveMetaStore.audit (HMSHandler.java:logAuditEvent(207)) - ugi=root ip=10.89.0.1cmd=source:10.89.0.1 drop_table : tbl=hive.default.test_tbl 2024-01-26T08:52:54,593 INFO [Metastore Scheduled Worker 1]: metastore.HMSHandler (HMSHandler.java:newRawStoreForConf(600)) - Opening raw store with implementation class: org.apache.hadoop.hive.metastore.ObjectStore 2024-01-26T08:52:54,594 INFO [Metastore Scheduled Worker 1]: metastore.PersistenceManagerProvider (PersistenceManagerProvider.java:correctAutoStartMechanism(577)) - Configuration datanucleus.autoStartMechanismMode is not set. Def aulting to 'ignored' 2024-01-26T08:52:54,594 INFO [Metastore Scheduled Worker 1]: metastore.ObjectStore (ObjectStore.java:shutdown(568)) - RawStore: org.apache.hadoop.hive.metastore.ObjectStore@63e26ab7, with PersistenceManager: null will be shutdown 2024-01-26T08:52:54,595 INFO [Metastore Scheduled Worker 1]: metastore.ObjectStore (ObjectStore.java:initialize(430)) - RawStore: org.apache.hadoop.hive.metastore.ObjectStore@63e26ab7, with PersistenceManager: org.datanucleus.api .jdo.JDOPersistenceManager@19d73712 created in the thread with id: 48 2024-01-26T08:52:54,601 INFO [Metastore Scheduled Worker 1]: metastore.HMSHandler (HMSHandler.java:getMSForConf(587)) - Created RawStore: org.apache.hadoop.hive.metastore.ObjectStore@63e26ab7 2024-01-26T08:53:15,125 WARN [Metastore-Handler-Pool: Thread-54]: metastore.ObjectStore (ObjectStore.java:handleDirectSqlError(4455)) - Falling back to ORM path due to direct SQL failure (this is not an error): Timeout when execu ting method: drop_table_with_environment_context; 60021ms exceeds 6ms at org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.newMetaException(MetaStoreUtils.java:218) at org.apache.hadoop.hive.metastore.utils.MetaStoreUtils. newMetaException(MetaStoreUtil
[jira] [Created] (HIVE-28024) Support sources profile for hive metastore modules
Wechar created HIVE-28024: - Summary: Support sources profile for hive metastore modules Key: HIVE-28024 URL: https://issues.apache.org/jira/browse/HIVE-28024 Project: Hive Issue Type: Improvement Components: Hive Affects Versions: 4.0.0-beta-1 Reporter: Wechar Assignee: Wechar Maven profile sources in Hive supports to generate source jar for all submodules. Since Hive Metastore modules are separated from hive parent module, it would be convenient to add this profile in standalone-metastore module. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-28014) to_unix_timestamp udf produces inconsistent results in different jdk versions
[ https://issues.apache.org/jira/browse/HIVE-28014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17808601#comment-17808601 ] Wechar commented on HIVE-28014: --- I have downgraded the jdk8 version in wecharyu/hive-dev-box:executor to avoid the CI failed temporally. It needs more clues to determine which side is not correct. > to_unix_timestamp udf produces inconsistent results in different jdk versions > - > > Key: HIVE-28014 > URL: https://issues.apache.org/jira/browse/HIVE-28014 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0-beta-1 >Reporter: Wechar >Assignee: Wechar >Priority: Major > > In HIVE-27999 we update the CI docker image which upgrades jdk8 from > {*}1.8.0_262-b19{*} to *1.8.0_392-b08*. This upgrade cause 3 timestamp > related tests failed: > *1. Testing / split-02 / PostProcess / > testTimestampToString[zoneId=Europe/Paris, timestamp=2417-03-26T02:08:43] – > org.apache.hadoop.hive.metastore.utils.TestMetaStoreUtils* > {code:bash} > Error > expected:<2417-03-26 0[2]:08:43> but was:<2417-03-26 0[3]:08:43> > Stacktrace > org.junit.ComparisonFailure: expected:<2417-03-26 0[2]:08:43> but > was:<2417-03-26 0[3]:08:43> > at > org.apache.hadoop.hive.metastore.utils.TestMetaStoreUtils.testTimestampToString(TestMetaStoreUtils.java:85) > {code} > *2. Testing / split-01 / PostProcess / testCliDriver[udf5] – > org.apache.hadoop.hive.cli.split24.TestMiniLlapLocalCliDriver* > {code:bash} > Error > Client Execution succeeded but contained differences (error code = 1) after > executing udf5.q > 263c263 > < 1400-11-08 07:35:34 > --- > > 1400-11-08 07:35:24 > 272c272 > < 1800-11-08 07:35:34 > --- > > 1800-11-08 07:35:24 > 434c434 > < 1399-12-31 23:35:34 > --- > > 1399-12-31 23:35:24 > 443c443 > < 1799-12-31 23:35:34 > --- > > 1799-12-31 23:35:24 > 452c452 > < 1899-12-31 23:35:34 > --- > > 1899-12-31 23:35:24 > {code} > *3. Testing / split-19 / PostProcess / testStringArg2 – > org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFToUnixTimestamp* > {code:bash} > Stacktrace > org.junit.ComparisonFailure: expected:<-17984790[40]0> but > was:<-17984790[39]0> > at org.junit.Assert.assertEquals(Assert.java:117) > at org.junit.Assert.assertEquals(Assert.java:146) > at > org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFToUnixTimestamp.runAndVerify(TestGenericUDFToUnixTimestamp.java:70) > at > org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFToUnixTimestamp.testStringArg2(TestGenericUDFToUnixTimestamp.java:167) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > {code} > It maybe a jdk bug and fixed in the new release, because we could get the > same result from Spark: > {code:sql} > spark-sql> select to_unix_timestamp(to_timestamp("1400-02-01 00:00:00 ICT", > "-MM-dd HH:mm:ss z"), "US/Pacific"); > -17984790390 > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28014) to_unix_timestamp udf produces inconsistent results in different jdk versions
Wechar created HIVE-28014: - Summary: to_unix_timestamp udf produces inconsistent results in different jdk versions Key: HIVE-28014 URL: https://issues.apache.org/jira/browse/HIVE-28014 Project: Hive Issue Type: Bug Components: Hive Affects Versions: 4.0.0-beta-1 Reporter: Wechar In HIVE-27999 we update the CI docker image which upgrades jdk8 from {*}1.8.0_262-b19{*} to *1.8.0_392-b08*. This upgrade cause 3 timestamp related tests failed: *1. Testing / split-02 / PostProcess / testTimestampToString[zoneId=Europe/Paris, timestamp=2417-03-26T02:08:43] – org.apache.hadoop.hive.metastore.utils.TestMetaStoreUtils* {code:bash} Error expected:<2417-03-26 0[2]:08:43> but was:<2417-03-26 0[3]:08:43> Stacktrace org.junit.ComparisonFailure: expected:<2417-03-26 0[2]:08:43> but was:<2417-03-26 0[3]:08:43> at org.apache.hadoop.hive.metastore.utils.TestMetaStoreUtils.testTimestampToString(TestMetaStoreUtils.java:85) {code} *2. Testing / split-01 / PostProcess / testCliDriver[udf5] – org.apache.hadoop.hive.cli.split24.TestMiniLlapLocalCliDriver* {code:bash} Error Client Execution succeeded but contained differences (error code = 1) after executing udf5.q 263c263 < 1400-11-08 07:35:34 --- > 1400-11-08 07:35:24 272c272 < 1800-11-08 07:35:34 --- > 1800-11-08 07:35:24 434c434 < 1399-12-31 23:35:34 --- > 1399-12-31 23:35:24 443c443 < 1799-12-31 23:35:34 --- > 1799-12-31 23:35:24 452c452 < 1899-12-31 23:35:34 --- > 1899-12-31 23:35:24 {code} *3. Testing / split-19 / PostProcess / testStringArg2 – org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFToUnixTimestamp* {code:bash} Stacktrace org.junit.ComparisonFailure: expected:<-17984790[40]0> but was:<-17984790[39]0> at org.junit.Assert.assertEquals(Assert.java:117) at org.junit.Assert.assertEquals(Assert.java:146) at org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFToUnixTimestamp.runAndVerify(TestGenericUDFToUnixTimestamp.java:70) at org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFToUnixTimestamp.testStringArg2(TestGenericUDFToUnixTimestamp.java:167) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) {code} It maybe a jdk bug and fixed in the new release, because we could get the same result from Spark: {code:sql} spark-sql> select to_unix_timestamp(to_timestamp("1400-02-01 00:00:00 ICT", "-MM-dd HH:mm:ss z"), "US/Pacific"); -17984790390 {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-28014) to_unix_timestamp udf produces inconsistent results in different jdk versions
[ https://issues.apache.org/jira/browse/HIVE-28014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar reassigned HIVE-28014: - Assignee: Wechar > to_unix_timestamp udf produces inconsistent results in different jdk versions > - > > Key: HIVE-28014 > URL: https://issues.apache.org/jira/browse/HIVE-28014 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0-beta-1 >Reporter: Wechar >Assignee: Wechar >Priority: Major > > In HIVE-27999 we update the CI docker image which upgrades jdk8 from > {*}1.8.0_262-b19{*} to *1.8.0_392-b08*. This upgrade cause 3 timestamp > related tests failed: > *1. Testing / split-02 / PostProcess / > testTimestampToString[zoneId=Europe/Paris, timestamp=2417-03-26T02:08:43] – > org.apache.hadoop.hive.metastore.utils.TestMetaStoreUtils* > {code:bash} > Error > expected:<2417-03-26 0[2]:08:43> but was:<2417-03-26 0[3]:08:43> > Stacktrace > org.junit.ComparisonFailure: expected:<2417-03-26 0[2]:08:43> but > was:<2417-03-26 0[3]:08:43> > at > org.apache.hadoop.hive.metastore.utils.TestMetaStoreUtils.testTimestampToString(TestMetaStoreUtils.java:85) > {code} > *2. Testing / split-01 / PostProcess / testCliDriver[udf5] – > org.apache.hadoop.hive.cli.split24.TestMiniLlapLocalCliDriver* > {code:bash} > Error > Client Execution succeeded but contained differences (error code = 1) after > executing udf5.q > 263c263 > < 1400-11-08 07:35:34 > --- > > 1400-11-08 07:35:24 > 272c272 > < 1800-11-08 07:35:34 > --- > > 1800-11-08 07:35:24 > 434c434 > < 1399-12-31 23:35:34 > --- > > 1399-12-31 23:35:24 > 443c443 > < 1799-12-31 23:35:34 > --- > > 1799-12-31 23:35:24 > 452c452 > < 1899-12-31 23:35:34 > --- > > 1899-12-31 23:35:24 > {code} > *3. Testing / split-19 / PostProcess / testStringArg2 – > org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFToUnixTimestamp* > {code:bash} > Stacktrace > org.junit.ComparisonFailure: expected:<-17984790[40]0> but > was:<-17984790[39]0> > at org.junit.Assert.assertEquals(Assert.java:117) > at org.junit.Assert.assertEquals(Assert.java:146) > at > org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFToUnixTimestamp.runAndVerify(TestGenericUDFToUnixTimestamp.java:70) > at > org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFToUnixTimestamp.testStringArg2(TestGenericUDFToUnixTimestamp.java:167) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > {code} > It maybe a jdk bug and fixed in the new release, because we could get the > same result from Spark: > {code:sql} > spark-sql> select to_unix_timestamp(to_timestamp("1400-02-01 00:00:00 ICT", > "-MM-dd HH:mm:ss z"), "US/Pacific"); > -17984790390 > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27999) Upgrade to Java 17 to run sonar analysis
[ https://issues.apache.org/jira/browse/HIVE-27999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-27999: -- Description: http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4905/4/pipeline/596/ CI failed by sonar plugin: {code:bash} [2024-01-15T03:58:10.100Z] [INFO] BUILD FAILURE [2024-01-15T03:58:10.100Z] [INFO] [2024-01-15T03:58:10.100Z] [INFO] Total time: 01:07 min [2024-01-15T03:58:10.100Z] [INFO] Finished at: 2024-01-15T03:58:09Z [2024-01-15T03:58:10.100Z] [INFO] [2024-01-15T03:58:10.100Z] [ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.9.1.2184:sonar (default-cli) on project hive: [2024-01-15T03:58:10.100Z] [ERROR] [2024-01-15T03:58:10.100Z] [ERROR] The version of Java (11.0.8) used to run this analysis is deprecated, and SonarCloud no longer supports it. Please upgrade to Java 17 or later. [2024-01-15T03:58:10.100Z] [ERROR] As a temporary measure, you can set the property 'sonar.scanner.force-deprecated-java-version' to 'true' to continue using Java 11.0.8 [2024-01-15T03:58:10.100Z] [ERROR] This workaround will only be effective until January 28, 2024. After this date, all scans using the deprecated Java 11 will fail. [2024-01-15T03:58:10.100Z] [ERROR] -> [Help 1] [2024-01-15T03:58:10.100Z] [ERROR] [2024-01-15T03:58:10.100Z] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [2024-01-15T03:58:10.100Z] [ERROR] Re-run Maven using the -X switch to enable full debug logging. {code} was: CI failed by sonar plugin: {code:bash} [2024-01-15T03:58:10.100Z] [INFO] BUILD FAILURE [2024-01-15T03:58:10.100Z] [INFO] [2024-01-15T03:58:10.100Z] [INFO] Total time: 01:07 min [2024-01-15T03:58:10.100Z] [INFO] Finished at: 2024-01-15T03:58:09Z [2024-01-15T03:58:10.100Z] [INFO] [2024-01-15T03:58:10.100Z] [ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.9.1.2184:sonar (default-cli) on project hive: [2024-01-15T03:58:10.100Z] [ERROR] [2024-01-15T03:58:10.100Z] [ERROR] The version of Java (11.0.8) used to run this analysis is deprecated, and SonarCloud no longer supports it. Please upgrade to Java 17 or later. [2024-01-15T03:58:10.100Z] [ERROR] As a temporary measure, you can set the property 'sonar.scanner.force-deprecated-java-version' to 'true' to continue using Java 11.0.8 [2024-01-15T03:58:10.100Z] [ERROR] This workaround will only be effective until January 28, 2024. After this date, all scans using the deprecated Java 11 will fail. [2024-01-15T03:58:10.100Z] [ERROR] -> [Help 1] [2024-01-15T03:58:10.100Z] [ERROR] [2024-01-15T03:58:10.100Z] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [2024-01-15T03:58:10.100Z] [ERROR] Re-run Maven using the -X switch to enable full debug logging. {code} > Upgrade to Java 17 to run sonar analysis > > > Key: HIVE-27999 > URL: https://issues.apache.org/jira/browse/HIVE-27999 > Project: Hive > Issue Type: Bug >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > > http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4905/4/pipeline/596/ > CI failed by sonar plugin: > {code:bash} > [2024-01-15T03:58:10.100Z] [INFO] BUILD FAILURE > [2024-01-15T03:58:10.100Z] [INFO] > > [2024-01-15T03:58:10.100Z] [INFO] Total time: 01:07 min > [2024-01-15T03:58:10.100Z] [INFO] Finished at: 2024-01-15T03:58:09Z > [2024-01-15T03:58:10.100Z] [INFO] > > [2024-01-15T03:58:10.100Z] [ERROR] Failed to execute goal > org.sonarsource.scanner.maven:sonar-maven-plugin:3.9.1.2184:sonar > (default-cli) on project hive: > [2024-01-15T03:58:10.100Z] [ERROR] > [2024-01-15T03:58:10.100Z] [ERROR] The version of Java (11.0.8) used to run > this analysis is deprecated, and SonarCloud no longer supports it. Please > upgrade to Java 17 or later. > [2024-01-15T03:58:10.100Z] [ERROR] As a temporary measure, you can set the > property 'sonar.scanner.force-deprecated-java-version' to 'true' to continue > using Java 11.0.8 > [2024-01-15T03:58:10.100Z] [ERROR] This workaround will only be effective > until January 28, 2024. After this date, all scans using the deprecated Java > 11 will fail. > [2024-01-15T03:58:10.100Z] [ERROR] -> [Help 1] > [2024-01-15T03:58:10.100Z] [ERROR] > [2024-01
[jira] [Created] (HIVE-27999) Upgrade to Java 17 to run sonar analysis
Wechar created HIVE-27999: - Summary: Upgrade to Java 17 to run sonar analysis Key: HIVE-27999 URL: https://issues.apache.org/jira/browse/HIVE-27999 Project: Hive Issue Type: Bug Reporter: Wechar Assignee: Wechar CI failed by sonar plugin: {code:bash} [2024-01-15T03:58:10.100Z] [INFO] BUILD FAILURE [2024-01-15T03:58:10.100Z] [INFO] [2024-01-15T03:58:10.100Z] [INFO] Total time: 01:07 min [2024-01-15T03:58:10.100Z] [INFO] Finished at: 2024-01-15T03:58:09Z [2024-01-15T03:58:10.100Z] [INFO] [2024-01-15T03:58:10.100Z] [ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.9.1.2184:sonar (default-cli) on project hive: [2024-01-15T03:58:10.100Z] [ERROR] [2024-01-15T03:58:10.100Z] [ERROR] The version of Java (11.0.8) used to run this analysis is deprecated, and SonarCloud no longer supports it. Please upgrade to Java 17 or later. [2024-01-15T03:58:10.100Z] [ERROR] As a temporary measure, you can set the property 'sonar.scanner.force-deprecated-java-version' to 'true' to continue using Java 11.0.8 [2024-01-15T03:58:10.100Z] [ERROR] This workaround will only be effective until January 28, 2024. After this date, all scans using the deprecated Java 11 will fail. [2024-01-15T03:58:10.100Z] [ERROR] -> [Help 1] [2024-01-15T03:58:10.100Z] [ERROR] [2024-01-15T03:58:10.100Z] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [2024-01-15T03:58:10.100Z] [ERROR] Re-run Maven using the -X switch to enable full debug logging. {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27857) Do not check write permission while dropping external table or partition
[ https://issues.apache.org/jira/browse/HIVE-27857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar resolved HIVE-27857. --- Fix Version/s: 4.0.0 Resolution: Fixed > Do not check write permission while dropping external table or partition > > > Key: HIVE-27857 > URL: https://issues.apache.org/jira/browse/HIVE-27857 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > When drop table for external table, we actually not need delete the table > path. But now HMS will check write permission if > {{AUTHORIZATION_STORAGE_AUTH_CHECKS}} enabled no matter the table is external > or not. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27775) DirectSQL and JDO results are different when fetching partitions by timestamp in DST shift
[ https://issues.apache.org/jira/browse/HIVE-27775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17800453#comment-17800453 ] Wechar commented on HIVE-27775: --- +1. We also noticed this bug that jdo query would return empty result for timestamp filter in the benchmark test of HIVE-27827. > DirectSQL and JDO results are different when fetching partitions by timestamp > in DST shift > -- > > Key: HIVE-27775 > URL: https://issues.apache.org/jira/browse/HIVE-27775 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 4.0.0-beta-1 >Reporter: Stamatis Zampetakis >Assignee: Zhihua Deng >Priority: Critical > Labels: pull-request-available > > DirectSQL and JDO results are different when fetching partitions by timestamp > in DST shift. > {code:sql} > --! qt:timezone:Europe/Paris > CREATE EXTERNAL TABLE payments (card string) PARTITIONED BY(txn_datetime > TIMESTAMP) STORED AS ORC; > INSERT into payments VALUES('---', '2023-03-26 02:30:00'); > SELECT * FROM payments WHERE txn_datetime = '2023-03-26 02:30:00'; > {code} > The '2023-03-26 02:30:00' is a timestamp that in Europe/Paris timezone falls > exactly in the middle of the DST shift. In this particular timezone this date > time never really exists since we are jumping directly from 02:00:00 to > 03:00:00. However, the TIMESTAMP data type in Hive is timezone agnostic > (https://cwiki.apache.org/confluence/display/Hive/Different+TIMESTAMP+types) > so it is a perfectly valid timestamp that can be inserted in a table and we > must be able to recover it back. > For the SELECT query above, partition pruning kicks in and calls the > ObjectStore#getPartitionsByExpr method in order to fetch the respective > partitions matching the timestamp from HMS. > The tests however reveal that DirectSQL and JDO paths are not returning the > same results leading to an exception when VerifyingObjectStore is used. > According to the error below DirectSQL is able to recover one partition from > HMS (expected) while JDO/ORM returns empty (not expected). > {noformat} > 2023-10-06T03:51:19,406 ERROR [80252df4-3fdc-4971-badf-ad67ce8567c7 main] > metastore.VerifyingObjectStore: Lists are not the same size: SQL 1, ORM 0 > 2023-10-06T03:51:19,409 ERROR [80252df4-3fdc-4971-badf-ad67ce8567c7 main] > metastore.RetryingHMSHandler: MetaException(message:Lists are not the same > size: SQL 1, ORM 0) > at > org.apache.hadoop.hive.metastore.VerifyingObjectStore.verifyLists(VerifyingObjectStore.java:148) > at > org.apache.hadoop.hive.metastore.VerifyingObjectStore.getPartitionsByExpr(VerifyingObjectStore.java:88) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) > at com.sun.proxy.$Proxy57.getPartitionsByExpr(Unknown Source) > at > org.apache.hadoop.hive.metastore.HMSHandler.get_partitions_spec_by_expr(HMSHandler.java:7330) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:98) > at > org.apache.hadoop.hive.metastore.AbstractHMSHandlerProxy.invoke(AbstractHMSHandlerProxy.java:82) > at com.sun.proxy.$Proxy59.get_partitions_spec_by_expr(Unknown Source) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionsSpecByExprInternal(HiveMetaStoreClient.java:2472) > at > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreClientWithLocalCache.getPartitionsSpecByExprInternal(HiveMetaStoreClientWithLocalCache.java:396) > at > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getPartitionsSpecByExprInternal(SessionHiveMetaStoreClient.java:2279) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitionsSpecByExpr(HiveMetaStoreClient.java:2484) > at > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.listPartitionsSpecByExpr(SessionHiveMetaStoreClient.java:1346) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[jira] [Created] (HIVE-27966) Disable flaky testFetchResultsOfLogWithOrientation
Wechar created HIVE-27966: - Summary: Disable flaky testFetchResultsOfLogWithOrientation Key: HIVE-27966 URL: https://issues.apache.org/jira/browse/HIVE-27966 Project: Hive Issue Type: Bug Components: Hive Reporter: Wechar Assignee: Wechar {{org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithMr#testFetchResultsOfLogWithOrientation}} test is flaky and can easily result into CI failed. Here are some affected pipelines: http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4959/3/tests http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4755/15/tests http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4966/1/tests http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4517/25/tests -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27386) Avoid duplicate audit log in cleanupHandlerContext
[ https://issues.apache.org/jira/browse/HIVE-27386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar resolved HIVE-27386. --- Fix Version/s: 4.0.0-beta-1 Resolution: Fixed > Avoid duplicate audit log in cleanupHandlerContext > -- > > Key: HIVE-27386 > URL: https://issues.apache.org/jira/browse/HIVE-27386 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-beta-1 > > Attachments: screenshot-1.png > > > {{HMSHandler#cleanupHandlerContext}} will be called twice if > {{IMetaStoreClient#close}} was called. In this case, where will create two > same audit log: Done cleaning up thread local RawStore. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27958) Refactor DirectSqlUpdatePart class
Wechar created HIVE-27958: - Summary: Refactor DirectSqlUpdatePart class Key: HIVE-27958 URL: https://issues.apache.org/jira/browse/HIVE-27958 Project: Hive Issue Type: Improvement Components: Hive Reporter: Wechar Assignee: Wechar Followed HIVE-27530 , refactor the {{{}DirectSqlUpdatePart.java{}}}: # Move lock related operations into a common class, details in [comment|https://github.com/apache/hive/pull/4517#discussion_r1410553559]. # Use try-for-resources instead of finally close, details in [comment|https://github.com/apache/hive/pull/4517#discussion_r1410559604]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27914) Fix the missing partitions judgement in drop_partitions_req
Wechar created HIVE-27914: - Summary: Fix the missing partitions judgement in drop_partitions_req Key: HIVE-27914 URL: https://issues.apache.org/jira/browse/HIVE-27914 Project: Hive Issue Type: Bug Reporter: Wechar Assignee: Wechar When call {{drop_partitions_req}} by partition expressions, HMS determine if there are missing partitions by the fetched partitions number, which is not correct. For example, assuming that we have a table {{tbl}} with partition names: {code:bash} dt=20231129/hr=10 dt=20231129/hr=11 {code} Then we try to drop partitions by sql: {code:sql} alter table tbl drop partition (dt='20231129'), partition (dt='20231130'); {code} It should throw `NoSuchObjectException` because filter {{dt='20231130'}} can not match any partitions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27913) Wrap NoSuchObjectException into HiveException in Hive.dropDatabase and Hive.dropConnector
Wechar created HIVE-27913: - Summary: Wrap NoSuchObjectException into HiveException in Hive.dropDatabase and Hive.dropConnector Key: HIVE-27913 URL: https://issues.apache.org/jira/browse/HIVE-27913 Project: Hive Issue Type: Improvement Components: Hive Reporter: Wechar Assignee: Wechar Align the drop_database and drop_dataconnector behavior with drop_table, which wrap NoSuchObjectException into HiveException when the object does not exist and input ifNotExists is false. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27857) Do not check write permission while dropping external table or partition
Wechar created HIVE-27857: - Summary: Do not check write permission while dropping external table or partition Key: HIVE-27857 URL: https://issues.apache.org/jira/browse/HIVE-27857 Project: Hive Issue Type: Bug Components: Hive Reporter: Wechar Assignee: Wechar When drop table for external table, we actually not need delete the table path. But now HMS will check write permission if {{AUTHORIZATION_STORAGE_AUTH_CHECKS}} enabled no matter the table is external or not. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27827) Improve performance of direct SQL implement for getPartitionsByFilter
Wechar created HIVE-27827: - Summary: Improve performance of direct SQL implement for getPartitionsByFilter Key: HIVE-27827 URL: https://issues.apache.org/jira/browse/HIVE-27827 Project: Hive Issue Type: Improvement Reporter: Wechar Assignee: Wechar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27762) ObjectStore GetHelper do not need run jdo query if direct sql exception is unrecoverable
Wechar created HIVE-27762: - Summary: ObjectStore GetHelper do not need run jdo query if direct sql exception is unrecoverable Key: HIVE-27762 URL: https://issues.apache.org/jira/browse/HIVE-27762 Project: Hive Issue Type: Improvement Components: Hive Reporter: Wechar Assignee: Wechar Currently {{GetHelper}} will call {{getJdoResult()}} if {{getSqlResult()}} throws exception, it can be avoid if the exception is unrecoverable to improve performance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27725) Remove redundant columns in TAB_COL_STATS and PART_COL_STATS
Wechar created HIVE-27725: - Summary: Remove redundant columns in TAB_COL_STATS and PART_COL_STATS Key: HIVE-27725 URL: https://issues.apache.org/jira/browse/HIVE-27725 Project: Hive Issue Type: Improvement Components: Hive Affects Versions: 4.0.0-beta-1 Reporter: Wechar Assignee: Wechar {{TAB_COL_STATS}} table includes {{CAT_NAME}}, {{DB_NAME}} and {{TABLE_NAME}}, which can be fetched by join {{TBLS}} and {{DBS}} tables on {{TBL_ID}} and {{DB_ID}} columns. {{PART_COL_STATS}} table includes {{CAT_NAME}}, {{DB_NAME}}, {{TABLE_NAME}} and {{PARTITION_NAME}}, which can be fetched by join {{PARTITIONS}}, {{TBLS}} and {{DBS}} tables on {{PART_ID}}, {{TBL_ID}} and {{DB_ID}}. In addition, current HMS get table statistics without join other table, while delete table statistics with join {{TBLS}}. This inconsistency will result exception if in a corner case where some table column statistics were left when drop table, then the user recreate the table with same name and database name but will get another {{TBL_ID}}, in this case user will get the old table column statistics incorrectly. And if user try delete stats fetched by get api, the {{NoSuchObjectException}} will be thrown. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27676) Reuse the add_partitions logic for add_partition in ObjectStore
Wechar created HIVE-27676: - Summary: Reuse the add_partitions logic for add_partition in ObjectStore Key: HIVE-27676 URL: https://issues.apache.org/jira/browse/HIVE-27676 Project: Hive Issue Type: Improvement Components: Hive Affects Versions: 4.0.0-beta-1 Reporter: Wechar Assignee: Wechar HIVE-26035 implements direct SQL for {{add_partitions}} to improve performance, we can also reuse this logic for {{{}add_partition{}}} with following benefits: * Get the performance improvement in direct SQL * Code cleaner, reduce the duplicate code. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27530) Implement direct SQL for alter partitions to improve performance
Wechar created HIVE-27530: - Summary: Implement direct SQL for alter partitions to improve performance Key: HIVE-27530 URL: https://issues.apache.org/jira/browse/HIVE-27530 Project: Hive Issue Type: Improvement Reporter: Wechar Assignee: Wechar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27500) Normalize add_partitions related api in HMS
[ https://issues.apache.org/jira/browse/HIVE-27500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-27500: -- Summary: Normalize add_partitions related api in HMS (was: Optimize add_partitions related api in HMS) > Normalize add_partitions related api in HMS > --- > > Key: HIVE-27500 > URL: https://issues.apache.org/jira/browse/HIVE-27500 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > > {{HMSHandler}} has three APIs for add partitions: {{add_partitions_req}}, > {{add_partitions}} and {{add_partitions_pspec}}. Currently both > {{add_partitions_req}} and {{add_partitions}} APIs invoke > {{add_partitions_core}} function which was improved by direct sql in > HIVE-26035. > We can also invoke {{add_partitions_core}} function within > {{add_partitions_pspec}} API for performance improvement. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27500) Optimize add_partitions related api in HMS
Wechar created HIVE-27500: - Summary: Optimize add_partitions related api in HMS Key: HIVE-27500 URL: https://issues.apache.org/jira/browse/HIVE-27500 Project: Hive Issue Type: Improvement Components: Hive Reporter: Wechar Assignee: Wechar {{HMSHandler}} has three APIs for add partitions: {{add_partitions_req}}, {{add_partitions}} and {{add_partitions_pspec}}. Currently both {{add_partitions_req}} and {{add_partitions}} APIs invoke {{add_partitions_core}} function which was improved by direct sql in HIVE-26035. We can also invoke {{add_partitions_core}} function within {{add_partitions_pspec}} API for performance improvement. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27469) HMSHandler lost root cause of MetaStorePreEventListener
[ https://issues.apache.org/jira/browse/HIVE-27469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-27469: -- Attachment: screenshot-1.png > HMSHandler lost root cause of MetaStorePreEventListener > --- > > Key: HIVE-27469 > URL: https://issues.apache.org/jira/browse/HIVE-27469 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > Attachments: screenshot-1.png > > > Currently {{HMSHandler}} will convert {{NoSuchObjectException}} and > {{InvalidOperationException}} to {{MetaException}}, but it will lose the root > cause. The related code as follows: > {code:java} > // HMSHandler.java#L3956 > private void firePreEvent(PreEventContext event) throws MetaException { > for (MetaStorePreEventListener listener : preListeners) { > try { > listener.onEvent(event); > } catch (NoSuchObjectException e) { > throw new MetaException(e.getMessage()); > } catch (InvalidOperationException e) { > throw new MetaException(e.getMessage()); > } > } > } > {code} > In this patch, we want to add the root cause and it can help troubleshooting > through HMS log. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27469) HMSHandler lost root cause of MetaStorePreEventListener
Wechar created HIVE-27469: - Summary: HMSHandler lost root cause of MetaStorePreEventListener Key: HIVE-27469 URL: https://issues.apache.org/jira/browse/HIVE-27469 Project: Hive Issue Type: Improvement Components: Hive Reporter: Wechar Assignee: Wechar Currently {{HMSHandler}} will convert {{NoSuchObjectException}} and {{InvalidOperationException}} to {{MetaException}}, but it will lose the root cause. The related code as follows: {code:java} // HMSHandler.java#L3956 private void firePreEvent(PreEventContext event) throws MetaException { for (MetaStorePreEventListener listener : preListeners) { try { listener.onEvent(event); } catch (NoSuchObjectException e) { throw new MetaException(e.getMessage()); } catch (InvalidOperationException e) { throw new MetaException(e.getMessage()); } } } {code} In this patch, we want to add the root cause and it can help troubleshooting through HMS log. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27461) HiveMetaStoreAuthorizer lost the root cause of checkPrivileges
Wechar created HIVE-27461: - Summary: HiveMetaStoreAuthorizer lost the root cause of checkPrivileges Key: HIVE-27461 URL: https://issues.apache.org/jira/browse/HIVE-27461 Project: Hive Issue Type: Improvement Components: Hive Reporter: Wechar Assignee: Wechar We discovered that the {{HiveMetaStoreAuthorizer}} fails to retain the underlying cause of {{checkPrivileges}}, making it difficult for us to troubleshoot authorization failures effectively. {code:bash} 2023-06-21T18:45:03,695 ERROR [pool-10-thread-1]: metastore.HiveMetaStoreAuthorizer (HiveMetaStoreAuthorizer.java:onEvent(111)) - HiveMeta StoreAuthorizer.onEvent(): failed org.apache.hadoop.hive.metastore.api.MetaException: Permission denied: Principal [name=weiqiang.yu, type=USER] does not have following pri vileges for operation QUERY [[SELECT] on Object [type=DATABASE, name=default]] at org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.checkPrivileges(HiveMetaStoreAuthoriz er.java:232) ~[hive-exec-3.1.2-sdi-022.jar:3.1.2-sdi-022] at org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.onEvent(HiveMetaStoreAuthorizer.java: 108) [hive-exec-3.1.2-sdi-022.jar:3.1.2-sdi-022] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.firePreEvent(HiveMetaStore.java:3469) [hive-exec-3.1.2-sdi-022.jar:3. 1.2-sdi-022] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database(HiveMetaStore.java:1463) [hive-exec-3.1.2-sdi-022.jar:3. 1.2-sdi-022] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:158) [hive-exec-3.1.2-sdi-022.jar:3. 1.2-sdi-022] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:119) [hive-exec-3.1.2-sdi-022.jar:3.1.2-sdi- 022] at com.sun.proxy.$Proxy29.get_database(Unknown Source) [?:?] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_d {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27460) HiveMetastoreClient should not wrap TTransportException in MetaException
Wechar created HIVE-27460: - Summary: HiveMetastoreClient should not wrap TTransportException in MetaException Key: HIVE-27460 URL: https://issues.apache.org/jira/browse/HIVE-27460 Project: Hive Issue Type: Improvement Components: Hive Reporter: Wechar Assignee: Wechar When instantiate hive metastore client, we will wrap the {{TTransportException}} within {{{}MetaException{}}}, and will try to connect to another metastore thrift uri if encountered {{{}MetaException{}}}. As discussed in HIVE-27097, client side only concerns about {{TTransportException}}, which should also apply to the connection building stage. So we can distinguish {{TTransportException}} from {{MetaException}} and stop connect immediately when encountered {{MetaException}}. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27417) Remove Filter.g and unnecessary antlr3 dependency
Wechar created HIVE-27417: - Summary: Remove Filter.g and unnecessary antlr3 dependency Key: HIVE-27417 URL: https://issues.apache.org/jira/browse/HIVE-27417 Project: Hive Issue Type: Sub-task Components: Hive Reporter: Wechar Assignee: Wechar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27386) Avoid duplicate audit log in cleanupHandlerContext
[ https://issues.apache.org/jira/browse/HIVE-27386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-27386: -- Summary: Avoid duplicate audit log in cleanupHandlerContext (was: Remove duplicate audit log in cleanupHandlerContext) > Avoid duplicate audit log in cleanupHandlerContext > -- > > Key: HIVE-27386 > URL: https://issues.apache.org/jira/browse/HIVE-27386 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > Attachments: screenshot-1.png > > > {{HMSHandler#cleanupHandlerContext}} will be called twice if > {{IMetaStoreClient#close}} was called. In this case, where will create two > same audit log: Done cleaning up thread local RawStore. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27386) Remove duplicate audit log in cleanupHandlerContext
[ https://issues.apache.org/jira/browse/HIVE-27386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-27386: -- Attachment: screenshot-1.png > Remove duplicate audit log in cleanupHandlerContext > --- > > Key: HIVE-27386 > URL: https://issues.apache.org/jira/browse/HIVE-27386 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > Attachments: screenshot-1.png > > > {{HMSHandler#cleanupHandlerContext}} will be called twice if > {{IMetaStoreClient#close}} was called. In this case, where will create two > same audit log: Done cleaning up thread local RawStore. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27386) Remove duplicate audit log in cleanupHandlerContext
Wechar created HIVE-27386: - Summary: Remove duplicate audit log in cleanupHandlerContext Key: HIVE-27386 URL: https://issues.apache.org/jira/browse/HIVE-27386 Project: Hive Issue Type: Improvement Components: Hive Reporter: Wechar Assignee: Wechar {{HMSHandler#cleanupHandlerContext}} will be called twice if {{IMetaStoreClient#close}} was called. In this case, where will create two same audit log: Done cleaning up thread local RawStore. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27362) Migrate Filter.g to Antlr4
Wechar created HIVE-27362: - Summary: Migrate Filter.g to Antlr4 Key: HIVE-27362 URL: https://issues.apache.org/jira/browse/HIVE-27362 Project: Hive Issue Type: Sub-task Components: Hive Reporter: Wechar Assignee: Wechar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27284) Make HMSHandler proxy pluggable
Wechar created HIVE-27284: - Summary: Make HMSHandler proxy pluggable Key: HIVE-27284 URL: https://issues.apache.org/jira/browse/HIVE-27284 Project: Hive Issue Type: Improvement Components: Hive Affects Versions: 4.0.0-alpha-2 Reporter: Wechar Assignee: Wechar Currently HMS use the only proxy implementation of HMSHandler i.e {{RetryingHMSHandler}}, resulting in some code hacks in {{HMSHandler}}. For example when test HMS timeout, we add additional static fields {{testTimeoutEnabled}} and {{testTimeoutValue}}, and add sleep code in {{create_database}} method, which is not elegant and flexible. So we introduce a new conf {{metastore.hmshandler.proxy}} to configure proxy class for HMSHandler, it will be more convenient to extend the new proxy of HMSHandler and can separate test code from HMSHandler. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27266) Retrieve only partNames if not need drop data in HMSHandler.dropPartitionsAndGetLocations
[ https://issues.apache.org/jira/browse/HIVE-27266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-27266: -- Description: Followed HIVE-19783, we only need partNames instead of partName and location pairs if we do not need check location. We add a new benchmark *dropTableMetadataWithPartitions* to delete only metadata rather than the real table data. Test results like: * Before the patch: {code:bash} Operation Mean Med Min Max Err% dropTableMetaOnlyWithPartitions.10 23.7021.8719.3631.7314.48 dropTableMetaOnlyWithPartitions.100 54.4254.1545.9276.688.891 dropTableMetaOnlyWithPartitions.1000 462.5456.1321.0654.315.96 {code} * After the patch: {code:bash} Operation Mean Med Min Max Err% dropTableMetaOnlyWithPartitions.10 21.4921.2419.3027.906.661 dropTableMetaOnlyWithPartitions.100 51.5148.3044.8685.2316.91 dropTableMetaOnlyWithPartitions.1000 415.4407.2308.8595.214.28 {code} was:Followed HIVE-19783, we only need partNames instead of partName and location pairs if we do not need check location. > Retrieve only partNames if not need drop data in > HMSHandler.dropPartitionsAndGetLocations > - > > Key: HIVE-27266 > URL: https://issues.apache.org/jira/browse/HIVE-27266 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0-alpha-2 >Reporter: Wechar >Assignee: Wechar >Priority: Major > > Followed HIVE-19783, we only need partNames instead of partName and location > pairs if we do not need check location. > We add a new benchmark *dropTableMetadataWithPartitions* to delete only > metadata rather than the real table data. > Test results like: > * Before the patch: > {code:bash} > Operation Mean Med Min Max Err% > dropTableMetaOnlyWithPartitions.10 23.7021.8719.3631.7314.48 > dropTableMetaOnlyWithPartitions.100 54.4254.1545.9276.688.891 > dropTableMetaOnlyWithPartitions.1000 462.5456.1321.0654.315.96 > {code} > * After the patch: > {code:bash} > Operation Mean Med Min Max Err% > dropTableMetaOnlyWithPartitions.10 21.4921.2419.3027.906.661 > dropTableMetaOnlyWithPartitions.100 51.5148.3044.8685.2316.91 > dropTableMetaOnlyWithPartitions.1000 415.4407.2308.8595.214.28 > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27266) Retrieve only partNames if not need drop data in HMSHandler.dropPartitionsAndGetLocations
Wechar created HIVE-27266: - Summary: Retrieve only partNames if not need drop data in HMSHandler.dropPartitionsAndGetLocations Key: HIVE-27266 URL: https://issues.apache.org/jira/browse/HIVE-27266 Project: Hive Issue Type: Improvement Components: Hive Affects Versions: 4.0.0-alpha-2 Reporter: Wechar Assignee: Wechar Followed HIVE-19783, we only need partNames instead of partName and location pairs if we do not need check location. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27172) Add the HMS client connection timeout config
[ https://issues.apache.org/jira/browse/HIVE-27172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-27172: -- Issue Type: Task (was: New Feature) > Add the HMS client connection timeout config > > > Key: HIVE-27172 > URL: https://issues.apache.org/jira/browse/HIVE-27172 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > > Currently {{HiveMetastoreClient}} use {{CLIENT_SOCKET_TIMEOUT}} as both > socket timeout and connection timeout, it's not convenient for users to set a > smaller connection timeout. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27172) Add the HMS client connection timeout config
[ https://issues.apache.org/jira/browse/HIVE-27172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar reassigned HIVE-27172: - > Add the HMS client connection timeout config > > > Key: HIVE-27172 > URL: https://issues.apache.org/jira/browse/HIVE-27172 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > > Currently {{HiveMetastoreClient}} use {{CLIENT_SOCKET_TIMEOUT}} as both > socket timeout and connection timeout, it's not convenient for users to set a > smaller connection timeout. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27150) Drop single partition can also support direct sql
[ https://issues.apache.org/jira/browse/HIVE-27150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar reassigned HIVE-27150: - > Drop single partition can also support direct sql > - > > Key: HIVE-27150 > URL: https://issues.apache.org/jira/browse/HIVE-27150 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > > *Background:* > [HIVE-6980|https://issues.apache.org/jira/browse/HIVE-6980] supports direct > sql for drop_partitions, we can reuse this huge improvement in drop_partition. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27097) Improve the retry strategy for Metastore client and server
[ https://issues.apache.org/jira/browse/HIVE-27097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-27097: -- Description: *Background* Hive provides *{{RetryingMetaStoreClient}}* and *{{RetryingHMSHandler}}* to do retry when thrift request failed: * RetryingMetaStoreClient will retry for *thrift related exception* and some *MetaException* * RetryingHMSHandler will retry for all {*}JDOException{*} or *NucleusException*. *Motivation* Current retry mechanism will lead to many unnecessary retries in both client and server. To simplify the process, we introduce following retry mechanism: * Client side only concerns the error of communication, i.e., {*}TTransportException{*}. * Server side can skip some exceptions which always turn to fail even with retry, like {*}SQLIntegrityConstraintViolationException{*}. was: *Background* Hive provides *{{RetryingMetaStoreClient}}* and *{{RetryingHMSHandler}}* to do retry when thrift request failed: * RetryingMetaStoreClient will retry for *thrift related exception* and some *MetaException* * RetryingHMSHandler will retry for all *JDOException*. *Motivation* Current retry mechanism will lead to many unnecessary retries in both client and server. To simplify the process, we introduce following retry mechanism: * Client side only concerns the error of communication, i.e., *TTransportException*. * Server side can skip some exceptions which always turn to fail even with retry, like *SQLIntegrityConstraintViolationException*. > Improve the retry strategy for Metastore client and server > -- > > Key: HIVE-27097 > URL: https://issues.apache.org/jira/browse/HIVE-27097 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0-alpha-2 >Reporter: Wechar >Assignee: Wechar >Priority: Major > > *Background* > Hive provides *{{RetryingMetaStoreClient}}* and *{{RetryingHMSHandler}}* to > do retry when thrift request failed: > * RetryingMetaStoreClient will retry for *thrift related exception* and some > *MetaException* > * RetryingHMSHandler will retry for all {*}JDOException{*} or > *NucleusException*. > *Motivation* > Current retry mechanism will lead to many unnecessary retries in both client > and server. To simplify the process, we introduce following retry mechanism: > * Client side only concerns the error of communication, i.e., > {*}TTransportException{*}. > * Server side can skip some exceptions which always turn to fail even with > retry, like {*}SQLIntegrityConstraintViolationException{*}. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27097) Improve the retry strategy for Metastore client and server
[ https://issues.apache.org/jira/browse/HIVE-27097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar reassigned HIVE-27097: - > Improve the retry strategy for Metastore client and server > -- > > Key: HIVE-27097 > URL: https://issues.apache.org/jira/browse/HIVE-27097 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0-alpha-2 >Reporter: Wechar >Assignee: Wechar >Priority: Major > > *Background* > Hive provides *{{RetryingMetaStoreClient}}* and *{{RetryingHMSHandler}}* to > do retry when thrift request failed: > * RetryingMetaStoreClient will retry for *thrift related exception* and some > *MetaException* > * RetryingHMSHandler will retry for all *JDOException*. > *Motivation* > Current retry mechanism will lead to many unnecessary retries in both client > and server. To simplify the process, we introduce following retry mechanism: > * Client side only concerns the error of communication, i.e., > *TTransportException*. > * Server side can skip some exceptions which always turn to fail even with > retry, like *SQLIntegrityConstraintViolationException*. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27093) Fix NPE in initialize() of Partition class
[ https://issues.apache.org/jira/browse/HIVE-27093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar reassigned HIVE-27093: - > Fix NPE in initialize() of Partition class > -- > > Key: HIVE-27093 > URL: https://issues.apache.org/jira/browse/HIVE-27093 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0-alpha-2 >Reporter: Wechar >Assignee: Wechar >Priority: Major > Attachments: image-2023-02-19-02-00-43-537.png > > > We will get a *NullPointerException* when get_partitions from HMS if the > partition SD does not exists. > !image-2023-02-19-02-00-43-537.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26935) Expose root cause of MetaException in RetryingHMSHandler
[ https://issues.apache.org/jira/browse/HIVE-26935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683633#comment-17683633 ] Wechar commented on HIVE-26935: --- Thanks for your review and suggestions [~zabetak]! > Expose root cause of MetaException in RetryingHMSHandler > > > Key: HIVE-26935 > URL: https://issues.apache.org/jira/browse/HIVE-26935 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0-alpha-2 >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > MetaException is generated by thrift, and only {{message}} filed will be > transport to client, we should expose the root cause in message to the > clients with following advantages: > * More friendly for user troubleshooting > * Some root cause is unrecoverable, exposing it can disable the unnecessary > retry. > *How to Reproduce:* > - Step 1: Disable direct sql for HMS for our test case. > - Step 2: Add an illegal {{PART_COL_STATS}} for a partition, > - Step 3: Try to {{drop table}} with Spark. > The exception in Hive metastore is: > {code:sh} > 2023-01-11T17:13:51,259 ERROR [Metastore-Handler-Pool: Thread-39]: > metastore.ObjectStore (ObjectStore.java:run(4369)) - > javax.jdo.JDOUserException: One or more instances could not be deleted > at > org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:625) > ~[datanucleus-api-jdo-5.2.8.jar:?] > at > org.datanucleus.api.jdo.JDOQuery.deletePersistentInternal(JDOQuery.java:530) > ~[datanucleus-api-jdo-5.2.8.jar:?] > at > org.datanucleus.api.jdo.JDOQuery.deletePersistentAll(JDOQuery.java:499) > ~[datanucleus-api-jdo-5.2.8.jar:?] > at > org.apache.hadoop.hive.metastore.QueryWrapper.deletePersistentAll(QueryWrapper.java:108) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionsNoTxn(ObjectStore.java:4207) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore.access$1000(ObjectStore.java:285) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore$7.run(ObjectStore.java:3086) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:74) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionsViaJdo(ObjectStore.java:3074) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore.access$400(ObjectStore.java:285) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore$6.getJdoResult(ObjectStore.java:3058) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore$6.getJdoResult(ObjectStore.java:3050) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:4362) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionsInternal(ObjectStore.java:3061) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore.dropPartitions(ObjectStore.java:3040) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[?:1.8.0_332] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[?:1.8.0_332] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:1.8.0_332] > at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_332] > at > org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at com.sun.proxy.$Proxy24.dropPartitions(Unknown Source) ~[?:?] > at > org.apache.hadoop.hive.metastore.HMSHandler.dropPartitionsAndGetLocations(HMSHandler.java:3186) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.HMSHandler.drop_table_core(HMSHandler.java:2963) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.HMSHandler.drop_table_with_environment_context(HMSHandler.java:3211) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.HMSHandler.drop_table_wit
[jira] [Assigned] (HIVE-26935) Expose root cause of MetaException to client sides
[ https://issues.apache.org/jira/browse/HIVE-26935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar reassigned HIVE-26935: - > Expose root cause of MetaException to client sides > -- > > Key: HIVE-26935 > URL: https://issues.apache.org/jira/browse/HIVE-26935 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0-alpha-2 >Reporter: Wechar >Assignee: Wechar >Priority: Major > > MetaException is generated by thrift, and only {{message}} filed will be > transport to client, we should expose the root cause in message to the > clients with following advantages: > * More friendly for user troubleshooting > * Some root cause is unrecoverable, exposing it can disable the unnecessary > retry. > *How to Reproduce:* > - Step 1: Disable direct sql for HMS for our test case. > - Step 2: Add an illegal {{PART_COL_STATS}} for a partition, > - Step 3: Try to {{drop table}} with Spark. > The exception in Hive metastore is: > {code:sh} > 2023-01-11T17:13:51,259 ERROR [Metastore-Handler-Pool: Thread-39]: > metastore.ObjectStore (ObjectStore.java:run(4369)) - > javax.jdo.JDOUserException: One or more instances could not be deleted > at > org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:625) > ~[datanucleus-api-jdo-5.2.8.jar:?] > at > org.datanucleus.api.jdo.JDOQuery.deletePersistentInternal(JDOQuery.java:530) > ~[datanucleus-api-jdo-5.2.8.jar:?] > at > org.datanucleus.api.jdo.JDOQuery.deletePersistentAll(JDOQuery.java:499) > ~[datanucleus-api-jdo-5.2.8.jar:?] > at > org.apache.hadoop.hive.metastore.QueryWrapper.deletePersistentAll(QueryWrapper.java:108) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionsNoTxn(ObjectStore.java:4207) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore.access$1000(ObjectStore.java:285) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore$7.run(ObjectStore.java:3086) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:74) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionsViaJdo(ObjectStore.java:3074) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore.access$400(ObjectStore.java:285) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore$6.getJdoResult(ObjectStore.java:3058) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore$6.getJdoResult(ObjectStore.java:3050) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:4362) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionsInternal(ObjectStore.java:3061) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.ObjectStore.dropPartitions(ObjectStore.java:3040) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[?:1.8.0_332] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[?:1.8.0_332] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:1.8.0_332] > at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_332] > at > org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at com.sun.proxy.$Proxy24.dropPartitions(Unknown Source) ~[?:?] > at > org.apache.hadoop.hive.metastore.HMSHandler.dropPartitionsAndGetLocations(HMSHandler.java:3186) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.HMSHandler.drop_table_core(HMSHandler.java:2963) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.HMSHandler.drop_table_with_environment_context(HMSHandler.java:3211) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at > org.apache.hadoop.hive.metastore.HMSHandler.drop_table_with_environment_context(HMSHandler.java:3199) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[?:1.8.0_332] > at > sun.reflect.NativeMethodAccessorImpl.invoke(Na
[jira] [Assigned] (HIVE-26661) Support partition filter for char and varchar types on Hive metastore
[ https://issues.apache.org/jira/browse/HIVE-26661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar reassigned HIVE-26661: - > Support partition filter for char and varchar types on Hive metastore > - > > Key: HIVE-26661 > URL: https://issues.apache.org/jira/browse/HIVE-26661 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0-alpha-1 >Reporter: Wechar >Assignee: Wechar >Priority: Minor > > HMS stores partition value as string in backend database, so it supports > "equal filter" for string (include date type) and integer types, and other > filter (for example greater than > ) for only string type. > The char and varchar types can also considered as string, and we can support > the partition filter for them on HMS. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26401) Refine the log of add_partitions if the partition already exists
[ https://issues.apache.org/jira/browse/HIVE-26401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar reassigned HIVE-26401: - > Refine the log of add_partitions if the partition already exists > > > Key: HIVE-26401 > URL: https://issues.apache.org/jira/browse/HIVE-26401 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0-alpha-1 >Reporter: Wechar >Assignee: Wechar >Priority: Minor > Fix For: 4.0.0-alpha-2 > > > Currently {{*add_partitions_xxx*}} will log the complete information of a > partition if it already exists, see in > [HMSHandler.java#L4320|https://github.com/apache/hive/blob/e3751ab545370f9b252d0b4a07bc315037541a95/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java#L4320]: > {code:java} > if (!shouldAdd) { > LOG.info("Not adding partition {} as it already exists", part); > return false; > } > {code} > It will print a long message including the columns of this partition, we > think it is unnecessary based on the following two points: > {color:red}1. The long message is redundant.{color} > We can get enough information from just > *cat_name.db_name.tbl_name[part_col1=part_val1/part_col2=part_val2...]* > {color:red}2. The long message is not friendly to save and query.{color} > This log message will take up a large log space especially when the user need > to execute *MSCK REPAIR TABLE* operation regularly because the old partition > must be already existed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26386) Exceptions thrown in ObjectStore should contain all causes
[ https://issues.apache.org/jira/browse/HIVE-26386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-26386: -- Description: {{*RetryingHMSHandler*}} will retry the failed operations if the exception is caused by {{*JDOException*}} or {{{}*NucleusException*{}}}, the logic is in [RetryingHMSHandler.java#L185:|https://github.com/apache/hive/blob/723d52270488b8dbc108db1a9925d1c569415039/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RetryingHMSHandler.java#L185] {code:java} } else if (e.getCause() instanceof MetaException && e.getCause().getCause() != null) { if (e.getCause().getCause() instanceof javax.jdo.JDOException || e.getCause().getCause() instanceof NucleusException) { // The JDOException or the Nucleus Exception may be wrapped further in a MetaException caughtException = e.getCause().getCause(); {code} But in {{{}*ObjectStore.java*{}}}, we found some exceptions thrown as {{{}*MetaException*{}}} are not wrapped by the causes, which cause some operations failed by {{JDOException}} can not do retry. was: {{*RetryingHMSHandler*}} will retry the failed operations if the exception is caused by {{*JDOException*}} or {{{}*NucleusException*{}}}, the logic is in [RetryingHMSHandler.java#L185:|https://github.com/apache/hive/blob/723d52270488b8dbc108db1a9925d1c569415039/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RetryingHMSHandler.java#L185] {code:java} } else if (e.getCause() instanceof MetaException && e.getCause().getCause() != null) { if (e.getCause().getCause() instanceof javax.jdo.JDOException || e.getCause().getCause() instanceof NucleusException) { // The JDOException or the Nucleus Exception may be wrapped further in a MetaException caughtException = e.getCause().getCause(); {code} But in {{{}*ObjectStore.java*{}}}, we found some exceptions thrown as {{{}*MetaException*{} are not wrapped by the causes, which cause some operations failed by {{JDOException}} can not do retry. > Exceptions thrown in ObjectStore should contain all causes > -- > > Key: HIVE-26386 > URL: https://issues.apache.org/jira/browse/HIVE-26386 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0-alpha-1 >Reporter: Wechar >Assignee: Wechar >Priority: Major > Fix For: 4.0.0-alpha-2 > > > {{*RetryingHMSHandler*}} will retry the failed operations if the exception is > caused by {{*JDOException*}} or {{{}*NucleusException*{}}}, the logic is in > [RetryingHMSHandler.java#L185:|https://github.com/apache/hive/blob/723d52270488b8dbc108db1a9925d1c569415039/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RetryingHMSHandler.java#L185] > {code:java} > } else if (e.getCause() instanceof MetaException && e.getCause().getCause() > != null) { > if (e.getCause().getCause() instanceof javax.jdo.JDOException || > e.getCause().getCause() instanceof NucleusException) { > // The JDOException or the Nucleus Exception may be wrapped > further in a MetaException > caughtException = e.getCause().getCause(); > {code} > But in {{{}*ObjectStore.java*{}}}, we found some exceptions thrown as > {{{}*MetaException*{}}} are not wrapped by the causes, which cause some > operations failed by {{JDOException}} can not do retry. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26386) Exceptions thrown in ObjectStore should contain all causes
[ https://issues.apache.org/jira/browse/HIVE-26386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar reassigned HIVE-26386: - > Exceptions thrown in ObjectStore should contain all causes > -- > > Key: HIVE-26386 > URL: https://issues.apache.org/jira/browse/HIVE-26386 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0-alpha-1 >Reporter: Wechar >Assignee: Wechar >Priority: Major > Fix For: 4.0.0-alpha-2 > > > {{*RetryingHMSHandler*}} will retry the failed operations if the exception is > caused by {{*JDOException*}} or {{{}*NucleusException*{}}}, the logic is in > [RetryingHMSHandler.java#L185:|https://github.com/apache/hive/blob/723d52270488b8dbc108db1a9925d1c569415039/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RetryingHMSHandler.java#L185] > {code:java} > } else if (e.getCause() instanceof MetaException && e.getCause().getCause() > != null) { > if (e.getCause().getCause() instanceof javax.jdo.JDOException || > e.getCause().getCause() instanceof NucleusException) { > // The JDOException or the Nucleus Exception may be wrapped > further in a MetaException > caughtException = e.getCause().getCause(); > {code} > But in {{{}*ObjectStore.java*{}}}, we found some exceptions thrown as > {{{}*MetaException*{} are not wrapped by the causes, which cause some > operations failed by {{JDOException}} can not do retry. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26355) Column compare should be case insensitive for name
[ https://issues.apache.org/jira/browse/HIVE-26355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-26355: -- Summary: Column compare should be case insensitive for name (was: Column compare should be case insensitive) > Column compare should be case insensitive for name > -- > > Key: HIVE-26355 > URL: https://issues.apache.org/jira/browse/HIVE-26355 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0-alpha-1 >Reporter: Wechar >Assignee: Wechar >Priority: Major > Fix For: 4.0.0-alpha-2 > > > Hive stores all name related value as lower case, such as db_name, tbl_name, > col_name etc. But the compare of {{FieldSchema}} does not ignore the case of > name, which may cause incorrect result of compare. > *Bug Description:* > Some computing engines are case sensitive for column name. For example, Spark > will add a table property to save the column fields when creating a table, > and will replace column fields with this property when fetching table fields. > When calling {{*ALTER TABLE ... ADD COLUMNS*}}, the compare of fields between > old table and new table will be not expected, and the ADD COLUMNS operation > will be cascaded to PARTITIONS, which is unnecessary and time consuming if > the table has many partitions. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (HIVE-26355) Column compare should be case insensitive
[ https://issues.apache.org/jira/browse/HIVE-26355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar reassigned HIVE-26355: - > Column compare should be case insensitive > - > > Key: HIVE-26355 > URL: https://issues.apache.org/jira/browse/HIVE-26355 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0-alpha-1 >Reporter: Wechar >Assignee: Wechar >Priority: Major > Fix For: 4.0.0-alpha-2 > > > Hive stores all name related value as lower case, such as db_name, tbl_name, > col_name etc. But the compare of {{FieldSchema}} does not ignore the case of > name, which may cause incorrect result of compare. > *Bug Description:* > Some computing engines are case sensitive for column name. For example, Spark > will add a table property to save the column fields when creating a table, > and will replace column fields with this property when fetching table fields. > When calling {{*ALTER TABLE ... ADD COLUMNS*}}, the compare of fields between > old table and new table will be not expected, and the ADD COLUMNS operation > will be cascaded to PARTITIONS, which is unnecessary and time consuming if > the table has many partitions. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HIVE-26314) Support alter function in Hive DDL
[ https://issues.apache.org/jira/browse/HIVE-26314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-26314: -- Description: Hive SQL does not support {{*ALTER FUNCTION*}} yet, we can refer to the {{*CREATE [OR REPLACE] FUNCTION*}} of [Spark|https://spark.apache.org/docs/3.1.2/sql-ref-syntax-ddl-create-function.html] to implement the alter function . {code:sql} CREATE [ TEMPORARY ] FUNCTION [ OR REPLACE ] [IF NOT EXISTS ] [db_name.]function_name AS class_name [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ]; {code} * *OR REPLACE* If specified, the resources for the function are reloaded. This is mainly useful to pick up any changes made to the implementation of the function. This parameter is mutually exclusive to {{*IF NOT EXISTS*}} and can not be specified together. was: Hive SQL does not support {{*ALTER FUNCTION*}} yet, we can refer to the {{*CREATE [OR REPLACE] FUNCTION*}} of [Spark|https://spark.apache.org/docs/3.1.2/sql-ref-syntax-ddl-create-function.html] to implement the alter function . {code:sql} CREATE [ OR REPLACE ] [ TEMPORARY ] FUNCTION [IF NOT EXISTS ] [db_name.]function_name AS class_name [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ]; {code} * *OR REPLACE* If specified, the resources for the function are reloaded. This is mainly useful to pick up any changes made to the implementation of the function. This parameter is mutually exclusive to {{*IF NOT EXISTS*}} and can not be specified together. > Support alter function in Hive DDL > -- > > Key: HIVE-26314 > URL: https://issues.apache.org/jira/browse/HIVE-26314 > Project: Hive > Issue Type: Task > Components: Hive >Affects Versions: 4.0.0-alpha-1 >Reporter: Wechar >Assignee: Wechar >Priority: Minor > Fix For: 4.0.0-alpha-2 > > > Hive SQL does not support {{*ALTER FUNCTION*}} yet, we can refer to the > {{*CREATE [OR REPLACE] FUNCTION*}} of > [Spark|https://spark.apache.org/docs/3.1.2/sql-ref-syntax-ddl-create-function.html] > to implement the alter function . > {code:sql} > CREATE [ TEMPORARY ] FUNCTION [ OR REPLACE ] [IF NOT EXISTS ] > [db_name.]function_name AS class_name > [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ]; > {code} > * *OR REPLACE* > If specified, the resources for the function are reloaded. This is mainly > useful to pick up any changes made to the implementation of the function. > This parameter is mutually exclusive to {{*IF NOT EXISTS*}} and can not be > specified together. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HIVE-26314) Support alter function in Hive DDL
[ https://issues.apache.org/jira/browse/HIVE-26314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-26314: -- Description: Hive SQL does not support {{*ALTER FUNCTION*}} yet, we can refer to the {{*CREATE [OR REPLACE] FUNCTION*}} of [Spark|https://spark.apache.org/docs/3.1.2/sql-ref-syntax-ddl-create-function.html] to implement the alter function . {code:sql} CREATE [ OR REPLACE ] [ TEMPORARY ] FUNCTION [IF NOT EXISTS ] [db_name.]function_name AS class_name [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ]; {code} * *OR REPLACE* If specified, the resources for the function are reloaded. This is mainly useful to pick up any changes made to the implementation of the function. This parameter is mutually exclusive to {{*IF NOT EXISTS*}} and can not be specified together. was: Hive SQL does not support {{*ALTER FUNCTION*}} yet, we can refer to the {{*CREATE [OR REPLACE] FUNCTION*}} of [Spark|https://spark.apache.org/docs/3.1.2/sql-ref-syntax-ddl-create-function.html] to implement the alter function . {code:sql} CREATE [ OR REPLACE ] [ TEMPORARY ] FUNCTION [IF NOT EXISTS ] [db_name.]function_name AS class_name [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ]; {code} * *OR REPLACE* If specified, the resources for the function are reloaded. This is mainly useful to pick up any changes made to the implementation of the function. This parameter is mutually exclusive to {{IF NOT EXISTS}} and can not be specified together. > Support alter function in Hive DDL > -- > > Key: HIVE-26314 > URL: https://issues.apache.org/jira/browse/HIVE-26314 > Project: Hive > Issue Type: Task > Components: Hive >Affects Versions: 4.0.0-alpha-1 >Reporter: Wechar >Assignee: Wechar >Priority: Minor > Fix For: 4.0.0-alpha-2 > > > Hive SQL does not support {{*ALTER FUNCTION*}} yet, we can refer to the > {{*CREATE [OR REPLACE] FUNCTION*}} of > [Spark|https://spark.apache.org/docs/3.1.2/sql-ref-syntax-ddl-create-function.html] > to implement the alter function . > {code:sql} > CREATE [ OR REPLACE ] [ TEMPORARY ] FUNCTION [IF NOT EXISTS ] > [db_name.]function_name AS class_name > [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ]; > {code} > * *OR REPLACE* > If specified, the resources for the function are reloaded. This is mainly > useful to pick up any changes made to the implementation of the function. > This parameter is mutually exclusive to {{*IF NOT EXISTS*}} and can not be > specified together. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HIVE-26314) Support alter function in Hive DDL
[ https://issues.apache.org/jira/browse/HIVE-26314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-26314: -- Description: Hive SQL does not support {{*ALTER FUNCTION*}} yet, we can refer to the {{*CREATE [OR REPLACE] FUNCTION*}} of [Spark|https://spark.apache.org/docs/3.1.2/sql-ref-syntax-ddl-create-function.html] to implement the alter function . {code:sql} CREATE [ OR REPLACE ] [ TEMPORARY ] FUNCTION [IF NOT EXISTS ] [db_name.]function_name AS class_name [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ]; {code} * *OR REPLACE* If specified, the resources for the function are reloaded. This is mainly useful to pick up any changes made to the implementation of the function. This parameter is mutually exclusive to {{IF NOT EXISTS}} and can not be specified together. was:Hive SQL does not support {{*ALTER FUNCTION*}} yet, we can refer to the {{*CREATE [OR REPLACE] FUNCTION*}} of [Spark|https://spark.apache.org/docs/3.1.2/sql-ref-syntax-ddl-create-function.html] to implement the alter function . > Support alter function in Hive DDL > -- > > Key: HIVE-26314 > URL: https://issues.apache.org/jira/browse/HIVE-26314 > Project: Hive > Issue Type: Task > Components: Hive >Affects Versions: 4.0.0-alpha-1 >Reporter: Wechar >Assignee: Wechar >Priority: Minor > Fix For: 4.0.0-alpha-2 > > > Hive SQL does not support {{*ALTER FUNCTION*}} yet, we can refer to the > {{*CREATE [OR REPLACE] FUNCTION*}} of > [Spark|https://spark.apache.org/docs/3.1.2/sql-ref-syntax-ddl-create-function.html] > to implement the alter function . > {code:sql} > CREATE [ OR REPLACE ] [ TEMPORARY ] FUNCTION [IF NOT EXISTS ] > [db_name.]function_name AS class_name > [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ]; > {code} > * *OR REPLACE* > If specified, the resources for the function are reloaded. This is mainly > useful to pick up any changes made to the implementation of the function. > This parameter is mutually exclusive to {{IF NOT EXISTS}} and can not be > specified together. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (HIVE-26314) Support alter function in Hive DDL
[ https://issues.apache.org/jira/browse/HIVE-26314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar reassigned HIVE-26314: - > Support alter function in Hive DDL > -- > > Key: HIVE-26314 > URL: https://issues.apache.org/jira/browse/HIVE-26314 > Project: Hive > Issue Type: Task > Components: Hive >Affects Versions: 4.0.0-alpha-1 >Reporter: Wechar >Assignee: Wechar >Priority: Minor > Fix For: 4.0.0-alpha-2 > > > Hive SQL does not support {{*ALTER FUNCTION*}} yet, we can refer to the > {{*CREATE [OR REPLACE] FUNCTION*}} of > [Spark|https://spark.apache.org/docs/3.1.2/sql-ref-syntax-ddl-create-function.html] > to implement the alter function . -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (HIVE-26259) Alter Function does not update resource uris
[ https://issues.apache.org/jira/browse/HIVE-26259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar reassigned HIVE-26259: - > Alter Function does not update resource uris > > > Key: HIVE-26259 > URL: https://issues.apache.org/jira/browse/HIVE-26259 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2 >Reporter: Wechar >Assignee: Wechar >Priority: Major > Fix For: 4.0.0-alpha-2 > > > *Bug Description:* > The jar of Hive permanent UDF can be loaded based on the resource uris, but > we encountered an issue after changing the resource uris through spark-sql: > {code:sql} > CREATE OR REPLACE FUNCTION test_db.test_udf AS 'com.xxx.xxx' > USING JAR 'hdfs://path/to/jar'; > {code} > Then when we use the UDF `test_db.test_udf`, an error occured like this: > {code:sh} > Error in query: Can not load class 'com.xxx.xxx' when registering the > function 'test_db.test_udf'... > {code} > *Root Cause:* > Hive metastore does not update resource uris while executing > `alter_function()`, which should be included and will not make any side > effect. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HIVE-26227) Add support of catalog related statements for Hive ql
[ https://issues.apache.org/jira/browse/HIVE-26227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17539577#comment-17539577 ] Wechar commented on HIVE-26227: --- Not quite yet. In our scene, HMS is in charge of handling catalogs as the metadata center, the other computing engines will only use catalogs while querying. Simply put, we extend the original `db_name.tbl_name` to `cat_name.db_name.tbl_name` in computing engines to support data from different systems or sources. So we do not plan to manage catalogs in other components now. > Add support of catalog related statements for Hive ql > - > > Key: HIVE-26227 > URL: https://issues.apache.org/jira/browse/HIVE-26227 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 20m > Remaining Estimate: 0h > > Catalog concept is proposed to Hive 3.0 to allow different systems to connect > to different catalogs in the metastore. But so far we can not query catalog > through Hive ql, this task aims to implement the ddl statements related to > catalog. > *Create Catalog* > {code:sql} > CREATE CATALOG [IF NOT EXISTS] catalog_name > LOCATION hdfs_path > [COMMENT catalog_comment]; > {code} > LOCATION is required for creating a new catalog now. > *Alter Catalog* > {code:sql} > ALTER CATALOG catalog_name SET LOCATION hdfs_path; > {code} > Only location metadata can be altered for catalog. > *Drop Catalog* > {code:sql} > DROP CATALOG [IF EXISTS] catalog_name; > {code} > DROP CATALOG is always RESTRICT, which means DROP CATALOG will fail if there > are non-default databases in the catalog. > *Show Catalogs* > {code:sql} > SHOW CATALOGS [LIKE 'identifier_with_wildcards']; > {code} > SHOW CATALOGS lists all of the catalogs defined in the metastore. > The optional LIKE clause allows the list of catalogs to be filtered using a > regular expression. > *Describe Catalog* > {code:sql} > DESC[RIBE] CATALOG [EXTENDED] cat_name; > {code} > DESCRIBE CATALOG shows the name of the catalog, its comment (if one has been > set), and its root location on the filesystem. > EXTENDED also shows the create time. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HIVE-26227) Add support of catalog related statements for Hive ql
[ https://issues.apache.org/jira/browse/HIVE-26227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17536689#comment-17536689 ] Wechar commented on HIVE-26227: --- Sure [~zabetak]. We plan to provide a unified metadata management service through the Hive metastore, which means the metadata of various systems are stored in Hive metastore and divided by catalog. Currently we want to manage the metadata from Hive, Hbase, Kafka, Jdbc, etc, and computing engines like Hive, Spark, Presto, Flink can join data from different systems based on the metadata in Hive metastore. > Add support of catalog related statements for Hive ql > - > > Key: HIVE-26227 > URL: https://issues.apache.org/jira/browse/HIVE-26227 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 10m > Remaining Estimate: 0h > > Catalog concept is proposed to Hive 3.0 to allow different systems to connect > to different catalogs in the metastore. But so far we can not query catalog > through Hive ql, this task aims to implement the ddl statements related to > catalog. > *Create Catalog* > {code:sql} > CREATE CATALOG [IF NOT EXISTS] catalog_name > LOCATION hdfs_path > [COMMENT catalog_comment]; > {code} > LOCATION is required for creating a new catalog now. > *Alter Catalog* > {code:sql} > ALTER CATALOG catalog_name SET LOCATION hdfs_path; > {code} > Only location metadata can be altered for catalog. > *Drop Catalog* > {code:sql} > DROP CATALOG [IF EXISTS] catalog_name; > {code} > DROP CATALOG is always RESTRICT, which means DROP CATALOG will fail if there > are non-default databases in the catalog. > *Show Catalogs* > {code:sql} > SHOW CATALOGS [LIKE 'identifier_with_wildcards']; > {code} > SHOW CATALOGS lists all of the catalogs defined in the metastore. > The optional LIKE clause allows the list of catalogs to be filtered using a > regular expression. > *Describe Catalog* > {code:sql} > DESC[RIBE] CATALOG [EXTENDED] cat_name; > {code} > DESCRIBE CATALOG shows the name of the catalog, its comment (if one has been > set), and its root location on the filesystem. > EXTENDED also shows the create time. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HIVE-26227) Add support of catalog related statements for Hive ql
[ https://issues.apache.org/jira/browse/HIVE-26227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-26227: -- Description: Catalog concept is proposed to Hive 3.0 to allow different systems to connect to different catalogs in the metastore. But so far we can not query catalog through Hive ql, this task aims to implement the ddl statements related to catalog. *Create Catalog* {code:sql} CREATE CATALOG [IF NOT EXISTS] catalog_name LOCATION hdfs_path [COMMENT catalog_comment]; {code} LOCATION is required for creating a new catalog now. *Alter Catalog* {code:sql} ALTER CATALOG catalog_name SET LOCATION hdfs_path; {code} Only location metadata can be altered for catalog. *Drop Catalog* {code:sql} DROP CATALOG [IF EXISTS] catalog_name; {code} DROP CATALOG is always RESTRICT, which means DROP CATALOG will fail if there are non-default databases in the catalog. *Show Catalogs* {code:sql} SHOW CATALOGS [LIKE 'identifier_with_wildcards']; {code} SHOW CATALOGS lists all of the catalogs defined in the metastore. The optional LIKE clause allows the list of catalogs to be filtered using a regular expression. *Describe Catalog* {code:sql} DESC[RIBE] CATALOG [EXTENDED] cat_name; {code} DESCRIBE CATALOG shows the name of the catalog, its comment (if one has been set), and its root location on the filesystem. EXTENDED also shows the create time. was: Catalog concept is proposed to Hive 3.0 to allow different systems to connect to different catalogs in the metastore. But so far we can not query catalog through Hive ql, this task aims to implement the ddl statements related to catalog. *Create Catalog* {code:sql} CREATE CATALOG [IF NOT EXISTS] catalog_name LOCATION hdfs_path [COMMENT catalog_comment]; {code} LOCATION is required for creating a new catalog now. *Alter Catalog* {code:sql} ALTER CATALOG catalog_name SET LOCATION hdfs_path; {code} Only location metadata can be altered for catalog. *Drop Catalog* {code:sql} DROP CATALOG [IF EXISTS] catalog_name; {code} DROP CATALOG is always RESTRICT, which means DROP CATALOG will fail if there are non-default databases in the catalog. *Show Catalog* {code:sql} SHOW CATALOGS [LIKE 'identifier_with_wildcards']; {code} SHOW CATALOGS lists all of the catalogs defined in the metastore. The optional LIKE clause allows the list of catalogs to be filtered using a regular expression. *Describe Catalog* {code:sql} DESC[RIBE] CATALOG [EXTENDED] cat_name; {code} DESCRIBE CATALOG shows the name of the catalog, its comment (if one has been set), and its root location on the filesystem. EXTENDED also shows the create time. > Add support of catalog related statements for Hive ql > - > > Key: HIVE-26227 > URL: https://issues.apache.org/jira/browse/HIVE-26227 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Minor > Fix For: 4.0.0-alpha-2 > > > Catalog concept is proposed to Hive 3.0 to allow different systems to connect > to different catalogs in the metastore. But so far we can not query catalog > through Hive ql, this task aims to implement the ddl statements related to > catalog. > *Create Catalog* > {code:sql} > CREATE CATALOG [IF NOT EXISTS] catalog_name > LOCATION hdfs_path > [COMMENT catalog_comment]; > {code} > LOCATION is required for creating a new catalog now. > *Alter Catalog* > {code:sql} > ALTER CATALOG catalog_name SET LOCATION hdfs_path; > {code} > Only location metadata can be altered for catalog. > *Drop Catalog* > {code:sql} > DROP CATALOG [IF EXISTS] catalog_name; > {code} > DROP CATALOG is always RESTRICT, which means DROP CATALOG will fail if there > are non-default databases in the catalog. > *Show Catalogs* > {code:sql} > SHOW CATALOGS [LIKE 'identifier_with_wildcards']; > {code} > SHOW CATALOGS lists all of the catalogs defined in the metastore. > The optional LIKE clause allows the list of catalogs to be filtered using a > regular expression. > *Describe Catalog* > {code:sql} > DESC[RIBE] CATALOG [EXTENDED] cat_name; > {code} > DESCRIBE CATALOG shows the name of the catalog, its comment (if one has been > set), and its root location on the filesystem. > EXTENDED also shows the create time. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (HIVE-26227) Add support of catalog related statements for Hive ql
[ https://issues.apache.org/jira/browse/HIVE-26227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar reassigned HIVE-26227: - > Add support of catalog related statements for Hive ql > - > > Key: HIVE-26227 > URL: https://issues.apache.org/jira/browse/HIVE-26227 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Minor > Fix For: 4.0.0-alpha-2 > > > Catalog concept is proposed to Hive 3.0 to allow different systems to connect > to different catalogs in the metastore. But so far we can not query catalog > through Hive ql, this task aims to implement the ddl statements related to > catalog. > *Create Catalog* > {code:sql} > CREATE CATALOG [IF NOT EXISTS] catalog_name > LOCATION hdfs_path > [COMMENT catalog_comment]; > {code} > LOCATION is required for creating a new catalog now. > *Alter Catalog* > {code:sql} > ALTER CATALOG catalog_name SET LOCATION hdfs_path; > {code} > Only location metadata can be altered for catalog. > *Drop Catalog* > {code:sql} > DROP CATALOG [IF EXISTS] catalog_name; > {code} > DROP CATALOG is always RESTRICT, which means DROP CATALOG will fail if there > are non-default databases in the catalog. > *Show Catalog* > {code:sql} > SHOW CATALOGS [LIKE 'identifier_with_wildcards']; > {code} > SHOW CATALOGS lists all of the catalogs defined in the metastore. > The optional LIKE clause allows the list of catalogs to be filtered using a > regular expression. > *Describe Catalog* > {code:sql} > DESC[RIBE] CATALOG [EXTENDED] cat_name; > {code} > DESCRIBE CATALOG shows the name of the catalog, its comment (if one has been > set), and its root location on the filesystem. > EXTENDED also shows the create time. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (HIVE-26205) Remove the incorrect org.slf4j dependency in kafka-handler
[ https://issues.apache.org/jira/browse/HIVE-26205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar reassigned HIVE-26205: - > Remove the incorrect org.slf4j dependency in kafka-handler > -- > > Key: HIVE-26205 > URL: https://issues.apache.org/jira/browse/HIVE-26205 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0-alpha-2 >Reporter: Wechar >Assignee: Wechar >Priority: Major > Fix For: 4.0.0-alpha-2 > > > Get a compile error while executing: > {code:bash} > mvn clean install -DskipTests > {code} > The error message is: > {code:bash} > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile > (default-compile) on project kafka-handler: Compilation failure: Compilation > failure: > [ERROR] > /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaStorageHandler.java:[53,17] > package org.slf4j does not exist > [ERROR] > /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaStorageHandler.java:[54,17] > package org.slf4j does not exist > [ERROR] > /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaStorageHandler.java:[73,24] > cannot find symbol > [ERROR] symbol: class Logger > [ERROR] location: class org.apache.hadoop.hive.kafka.KafkaStorageHandler > [ERROR] > /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/VectorizedKafkaRecordReader.java:[37,17] > package org.slf4j does not exist > [ERROR] > /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/VectorizedKafkaRecordReader.java:[47,24] > cannot find symbol > [ERROR] symbol: class Logger > [ERROR] location: class > org.apache.hadoop.hive.kafka.VectorizedKafkaRecordReader > [ERROR] > /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaJsonSerDe.java:[63,17] > package org.slf4j does not exist > [ERROR] > /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/SimpleKafkaWriter.java:[35,17] > package org.slf4j does not exist > [ERROR] > /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/SimpleKafkaWriter.java:[50,24] > cannot find symbol > [ERROR] symbol: class Logger > [ERROR] location: class org.apache.hadoop.hive.kafka.SimpleKafkaWriter > [ERROR] > /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaOutputFormat.java:[34,17] > package org.slf4j does not exist > [ERROR] > /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaOutputFormat.java:[43,24] > cannot find symbol > [ERROR] symbol: class Logger > [ERROR] location: class org.apache.hadoop.hive.kafka.KafkaOutputFormat > [ERROR] > /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/RetryUtils.java:[24,17] > package org.slf4j does not exist > [ERROR] > /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/RetryUtils.java:[34,24] > cannot find symbol > [ERROR] symbol: class Logger > [ERROR] location: class org.apache.hadoop.hive.kafka.RetryUtils > [ERROR] > /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaScanTrimmer.java:[51,17] > package org.slf4j does not exist > [ERROR] > /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaScanTrimmer.java:[65,24] > cannot find symbol > [ERROR] symbol: class Logger > [ERROR] location: class org.apache.hadoop.hive.kafka.KafkaScanTrimmer > [ERROR] > /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/TransactionalKafkaWriter.java:[45,17] > package org.slf4j does not exist > [ERROR] > /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/TransactionalKafkaWriter.java:[65,24] > cannot find symbol > [ERROR] symbol: class Logger > [ERROR] location: class > org.apache.hadoop.hive.kafka.TransactionalKafkaWriter > [ERROR] > /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/HiveKafkaProducer.java:[37,17] > package org.slf4j does not exist > [ERROR] > /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/HiveKafkaProducer.java:[59,24] > cannot find symbol > [ERROR] symbol: class Logger > [ERROR] location: class org.apache.hadoop.hive.kafka.HiveKafkaProducer > [ERROR] > /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaRecordIterator.java:[30,17] > package org.slf4j does not exist > [ERROR] > /ldap_home/w
[jira] [Updated] (HIVE-26159) hive cli is unavailable from hive command
[ https://issues.apache.org/jira/browse/HIVE-26159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-26159: -- Description: Hive cli is a convenient tool to connect to hive metastore service, but now hive cli can not start even if we use *--service cli* option, it should be a bug of ticket HIVE-24348. *Steps to reproduce:* {code:bash} hive@hive:/root$ /usr/share/hive/bin/hive --service cli --hiveconf hive.metastore.uris=thrift://hive:9084 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Beeline version 4.0.0-alpha-2-SNAPSHOT by Apache Hive beeline> {code} was: Hive cli is a convenient tool to connect to hive metastore service, but now hive cli can not start even if we use *--service cli* option, it should be a bug of ticket [HIVE-24348|https://issues.apache.org/jira/browse/HIVE-24348]. *Step to reproduce:* {code:bash} hive@hive:/root$ /usr/share/hive/bin/hive --service cli --hiveconf hive.metastore.uris=thrift://hive:9084 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Beeline version 4.0.0-alpha-2-SNAPSHOT by Apache Hive beeline> {code} > hive cli is unavailable from hive command > - > > Key: HIVE-26159 > URL: https://issues.apache.org/jira/browse/HIVE-26159 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0-alpha-1 >Reporter: Wechar >Assignee: Wechar >Priority: Major > Fix For: 4.0.0 > > > Hive cli is a convenient tool to connect to hive metastore service, but now > hive cli can not start even if we use *--service cli* option, it should be a > bug of ticket HIVE-24348. > *Steps to reproduce:* > {code:bash} > hive@hive:/root$ /usr/share/hive/bin/hive --service cli --hiveconf > hive.metastore.uris=thrift://hive:9084 > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/opt/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/opt/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] > Beeline version 4.0.0-alpha-2-SNAPSHOT by Apache Hive > beeline> > {code} -- Thi
[jira] [Assigned] (HIVE-26159) hive cli is unavailable from hive command
[ https://issues.apache.org/jira/browse/HIVE-26159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar reassigned HIVE-26159: - > hive cli is unavailable from hive command > - > > Key: HIVE-26159 > URL: https://issues.apache.org/jira/browse/HIVE-26159 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0-alpha-1 >Reporter: Wechar >Assignee: Wechar >Priority: Major > Fix For: 4.0.0 > > > Hive cli is a convenient tool to connect to hive metastore service, but now > hive cli can not start even if we use *--service cli* option, it should be a > bug of ticket [HIVE-24348|https://issues.apache.org/jira/browse/HIVE-24348]. > *Step to reproduce:* > {code:bash} > hive@hive:/root$ /usr/share/hive/bin/hive --service cli --hiveconf > hive.metastore.uris=thrift://hive:9084 > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/opt/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/opt/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] > Beeline version 4.0.0-alpha-2-SNAPSHOT by Apache Hive > beeline> > {code} -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (HIVE-26148) Keep MetaStoreFilterHook interface compatibility after introducing catalogs
[ https://issues.apache.org/jira/browse/HIVE-26148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar reassigned HIVE-26148: - > Keep MetaStoreFilterHook interface compatibility after introducing catalogs > --- > > Key: HIVE-26148 > URL: https://issues.apache.org/jira/browse/HIVE-26148 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.0.0 >Reporter: Wechar >Assignee: Wechar >Priority: Minor > Fix For: 4.0.0-alpha-1 > > > Hive 3.0 introduce catalog concept, when we upgrade hive dependency version > from 2.3 to 3.x, we found some interfaces of *MetaStoreFilterHook* are not > compatible: > {code:bash} > git show ba8a99e115 -- > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreFilterHook.java > {code} > {code:bash} > --- > a/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreFilterHook.java > +++ > b/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreFilterHook.java >/** > * Filter given list of tables > - * @param dbName > - * @param tableList > + * @param catName catalog name > + * @param dbName database name > + * @param tableList list of table returned by the metastore > * @return List of filtered table names > */ > - public List filterTableNames(String dbName, List > tableList) throws MetaException; > + List filterTableNames(String catName, String dbName, List > tableList) > + throws MetaException; > {code} > We can remain the previous interfaces and use the default catalog to > implement. -- This message was sent by Atlassian Jira (v8.20.1#820001)