[jira] [Resolved] (HIVE-28311) Backward compatibility of java.sql.Date and java.sql.Timestamp in hive-serde
[
https://issues.apache.org/jira/browse/HIVE-28311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar resolved HIVE-28311.
---
Fix Version/s: Not Applicable
Resolution: Won't Fix
> Backward compatibility of java.sql.Date and java.sql.Timestamp in hive-serde
>
>
> Key: HIVE-28311
> URL: https://issues.apache.org/jira/browse/HIVE-28311
> Project: Hive
> Issue Type: Bug
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Fix For: Not Applicable
>
>
> HIVE-20007 introduced {{org.apache.hadoop.hive.common.type.Date}} and
> {{org.apache.hadoop.hive.common.type.Timestamp}} to replace {{java.sql.Date}}
> and {{{}java.sql.Timestamp{}}}.
> It's a huge improvements but it also produce incompatibility issues for
> clients without this update.
> {code:bash}
> Caused by: java.lang.ClassCastException: java.sql.Timestamp cannot be cast to
> org.apache.hadoop.hive.common.type.Timestamp
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaTimestampObjectInspector.getPrimitiveWritableObject(JavaTimestampObjectInspector.java:33)
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getTimestamp(PrimitiveObjectInspectorUtils.java:1232)
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TimestampConverter.convert(PrimitiveObjectInspectorConverter.java:291)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getConstantObjectInspector(ObjectInspectorUtils.java:1397)
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc.getWritableObjectInspector(ExprNodeConstantDesc.java:93)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeConstantEvaluator.(ExprNodeConstantEvaluator.java:41)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:49)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.(ExprNodeGenericFuncEvaluator.java:101)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:58)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:43)
> at
> org.apache.hadoop.hive.ql.optimizer.ppr.PartExprEvalUtils.prepareExpr(PartExprEvalUtils.java:118)
> at
> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prunePartitionNames(PartitionPruner.java:551)
> at
> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionExpressionForMetastore.filterPartitionsByExpr(PartitionExpressionForMetastore.java:73)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionNamesPrunedByExprNoTxn(ObjectStore.java:3606)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$1000(ObjectStore.java:241)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$16.getJdoResult(ObjectStore.java:4157)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$16.getJdoResult(ObjectStore.java:4124)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3913)
> ... 30 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-28311) Backward compatibility of java.sql.Date and java.sql.Timestamp in hive-serde
[
https://issues.apache.org/jira/browse/HIVE-28311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17860348#comment-17860348
]
Wechar edited comment on HIVE-28311 at 6/27/24 6:32 AM:
[~zabetak] I found this issue in hive client 2.3.8 and server 3.1.2, and this
patch can solve this issue.
But the master has upgrade to Kryo5 in HIVE-22944, then hive client 2.x and 3.x
serialization bytes can not be deserialized successfully in HMS 4.x. Luckily
after applying patch HIVE-22944 in hive 2.x, it works well. I think we can
upgrade kryo in low version clients to solve this issue.
And we can close this ticket now.
was (Author: wechar):
[~zabetak] I found this issue in hive client 2.3.8 and server 3.1.2, and this
patch can solve this issue.
But the master has upgrade to Kryo5 in HIVE-22944, then hive client 2.x and 3.x
serialization bytes can not be deserialized successfully in HMS 4.x. Luckily
after applying patch HIVE-22944 in hive 2.x, it works well. I think we can
upgrade kryo in low version clients to solve this issue.
> Backward compatibility of java.sql.Date and java.sql.Timestamp in hive-serde
>
>
> Key: HIVE-28311
> URL: https://issues.apache.org/jira/browse/HIVE-28311
> Project: Hive
> Issue Type: Bug
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
>
> HIVE-20007 introduced {{org.apache.hadoop.hive.common.type.Date}} and
> {{org.apache.hadoop.hive.common.type.Timestamp}} to replace {{java.sql.Date}}
> and {{{}java.sql.Timestamp{}}}.
> It's a huge improvements but it also produce incompatibility issues for
> clients without this update.
> {code:bash}
> Caused by: java.lang.ClassCastException: java.sql.Timestamp cannot be cast to
> org.apache.hadoop.hive.common.type.Timestamp
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaTimestampObjectInspector.getPrimitiveWritableObject(JavaTimestampObjectInspector.java:33)
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getTimestamp(PrimitiveObjectInspectorUtils.java:1232)
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TimestampConverter.convert(PrimitiveObjectInspectorConverter.java:291)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getConstantObjectInspector(ObjectInspectorUtils.java:1397)
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc.getWritableObjectInspector(ExprNodeConstantDesc.java:93)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeConstantEvaluator.(ExprNodeConstantEvaluator.java:41)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:49)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.(ExprNodeGenericFuncEvaluator.java:101)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:58)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:43)
> at
> org.apache.hadoop.hive.ql.optimizer.ppr.PartExprEvalUtils.prepareExpr(PartExprEvalUtils.java:118)
> at
> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prunePartitionNames(PartitionPruner.java:551)
> at
> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionExpressionForMetastore.filterPartitionsByExpr(PartitionExpressionForMetastore.java:73)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionNamesPrunedByExprNoTxn(ObjectStore.java:3606)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$1000(ObjectStore.java:241)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$16.getJdoResult(ObjectStore.java:4157)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$16.getJdoResult(ObjectStore.java:4124)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3913)
> ... 30 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-28311) Backward compatibility of java.sql.Date and java.sql.Timestamp in hive-serde
[
https://issues.apache.org/jira/browse/HIVE-28311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17860348#comment-17860348
]
Wechar commented on HIVE-28311:
---
[~zabetak] I found this issue in hive client 2.3.8 and server 3.1.2, and this
patch can solve this issue.
But the master has upgrade to Kryo5 in HIVE-22944, then hive client 2.x and 3.x
serialization bytes can not be deserialized successfully in HMS 4.x. Luckily
after applying patch HIVE-22944 in hive 2.x, it works well. I think we can
upgrade kryo in low version clients to solve this issue.
> Backward compatibility of java.sql.Date and java.sql.Timestamp in hive-serde
>
>
> Key: HIVE-28311
> URL: https://issues.apache.org/jira/browse/HIVE-28311
> Project: Hive
> Issue Type: Bug
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
>
> HIVE-20007 introduced {{org.apache.hadoop.hive.common.type.Date}} and
> {{org.apache.hadoop.hive.common.type.Timestamp}} to replace {{java.sql.Date}}
> and {{{}java.sql.Timestamp{}}}.
> It's a huge improvements but it also produce incompatibility issues for
> clients without this update.
> {code:bash}
> Caused by: java.lang.ClassCastException: java.sql.Timestamp cannot be cast to
> org.apache.hadoop.hive.common.type.Timestamp
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaTimestampObjectInspector.getPrimitiveWritableObject(JavaTimestampObjectInspector.java:33)
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getTimestamp(PrimitiveObjectInspectorUtils.java:1232)
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TimestampConverter.convert(PrimitiveObjectInspectorConverter.java:291)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getConstantObjectInspector(ObjectInspectorUtils.java:1397)
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc.getWritableObjectInspector(ExprNodeConstantDesc.java:93)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeConstantEvaluator.(ExprNodeConstantEvaluator.java:41)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:49)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.(ExprNodeGenericFuncEvaluator.java:101)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:58)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:43)
> at
> org.apache.hadoop.hive.ql.optimizer.ppr.PartExprEvalUtils.prepareExpr(PartExprEvalUtils.java:118)
> at
> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prunePartitionNames(PartitionPruner.java:551)
> at
> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionExpressionForMetastore.filterPartitionsByExpr(PartitionExpressionForMetastore.java:73)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionNamesPrunedByExprNoTxn(ObjectStore.java:3606)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$1000(ObjectStore.java:241)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$16.getJdoResult(ObjectStore.java:4157)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$16.getJdoResult(ObjectStore.java:4124)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3913)
> ... 30 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-28338) Client connection count is not correct in HiveMetaStore#close
Wechar created HIVE-28338:
-
Summary: Client connection count is not correct in
HiveMetaStore#close
Key: HIVE-28338
URL: https://issues.apache.org/jira/browse/HIVE-28338
Project: Hive
Issue Type: Bug
Components: Hive
Reporter: Wechar
Assignee: Wechar
HIVE-24349 introduced a bug in {{HiveMetaStoreClient}} for embedded metastore,
where the log would print negative connection counts.
*Root Cause*
Connection count is only used in remote metastore, we do not need decrease
connection counts when transport is null.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28307) Fast return when database exists and IF NOT EXISTS clause is enabled
[
https://issues.apache.org/jira/browse/HIVE-28307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-28307:
--
Description:
When use beeline for testing purpose:
{code:bash}
beeline -u jdbc:hive2://
{code}
{{CREATE TABLE IF NOT EXISTS}} will not print {{{}AlreadyExistsException{}}},
but {{CREATE DATABASE IF NOT EXISTS}} will.
The error log may be confused because the sql actually execute successfully.
The root cause is that {{SemanticAnalyzer}} will fast-return if table exists:
[https://github.com/apache/hive/blob/98d9d22398370f817fe64449368671b978fff096/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L14163-L14174]
was:
When use beeline for testing purpose:
{code:bash}
beeline -u jdbc:hive2://
{code}
{{CREATE TABLE IF NOT EXISTS}} will not print {{AlreadyExistsException}}, but
{{CREATE DATABASE IF NOT EXISTS}} will.
The root cause is that {{SemanticAnalyzer}} will fast-return if table exists:
https://github.com/apache/hive/blob/98d9d22398370f817fe64449368671b978fff096/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L14163-L14174
> Fast return when database exists and IF NOT EXISTS clause is enabled
>
>
> Key: HIVE-28307
> URL: https://issues.apache.org/jira/browse/HIVE-28307
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Attachments: create_database.png, create_table.png
>
>
> When use beeline for testing purpose:
> {code:bash}
> beeline -u jdbc:hive2://
> {code}
> {{CREATE TABLE IF NOT EXISTS}} will not print {{{}AlreadyExistsException{}}},
> but {{CREATE DATABASE IF NOT EXISTS}} will.
> The error log may be confused because the sql actually execute successfully.
>
> The root cause is that {{SemanticAnalyzer}} will fast-return if table exists:
> [https://github.com/apache/hive/blob/98d9d22398370f817fe64449368671b978fff096/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L14163-L14174]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28307) Fast return when database exists and IF NOT EXISTS clause is enabled
[
https://issues.apache.org/jira/browse/HIVE-28307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-28307:
--
Attachment: create_table.png
> Fast return when database exists and IF NOT EXISTS clause is enabled
>
>
> Key: HIVE-28307
> URL: https://issues.apache.org/jira/browse/HIVE-28307
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Attachments: create_database.png, create_table.png
>
>
> When use beeline for testing purpose:
> {code:bash}
> beeline -u jdbc:hive2://
> {code}
> {{CREATE TABLE IF NOT EXISTS}} will not print {{AlreadyExistsException}}, but
> {{CREATE DATABASE IF NOT EXISTS}} will.
> The root cause is that {{SemanticAnalyzer}} will fast-return if table exists:
> https://github.com/apache/hive/blob/98d9d22398370f817fe64449368671b978fff096/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L14163-L14174
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28307) Fast return when database exists and IF NOT EXISTS clause is enabled
[
https://issues.apache.org/jira/browse/HIVE-28307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-28307:
--
Attachment: create_database.png
> Fast return when database exists and IF NOT EXISTS clause is enabled
>
>
> Key: HIVE-28307
> URL: https://issues.apache.org/jira/browse/HIVE-28307
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Attachments: create_database.png
>
>
> When use beeline for testing purpose:
> {code:bash}
> beeline -u jdbc:hive2://
> {code}
> {{CREATE TABLE IF NOT EXISTS}} will not print {{AlreadyExistsException}}, but
> {{CREATE DATABASE IF NOT EXISTS}} will.
> The root cause is that {{SemanticAnalyzer}} will fast-return if table exists:
> https://github.com/apache/hive/blob/98d9d22398370f817fe64449368671b978fff096/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L14163-L14174
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28307) Fast return when database exists and IF NOT EXISTS clause is enabled
[
https://issues.apache.org/jira/browse/HIVE-28307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-28307:
--
Summary: Fast return when database exists and IF NOT EXISTS clause is
enabled (was: Enable IF NOT EXISTS in CREATE DATABASE statements.)
> Fast return when database exists and IF NOT EXISTS clause is enabled
>
>
> Key: HIVE-28307
> URL: https://issues.apache.org/jira/browse/HIVE-28307
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
>
> When use beeline for testing purpose:
> {code:bash}
> beeline -u jdbc:hive2://
> {code}
> {{CREATE TABLE IF NOT EXISTS}} will not print {{AlreadyExistsException}}, but
> {{CREATE DATABASE IF NOT EXISTS}} will.
> The root cause is that {{SemanticAnalyzer}} will fast-return if table exists:
> https://github.com/apache/hive/blob/98d9d22398370f817fe64449368671b978fff096/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L14163-L14174
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28307) Enable IF NOT EXISTS in CREATE DATABASE statements.
[
https://issues.apache.org/jira/browse/HIVE-28307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-28307:
--
Description:
When use beeline for testing purpose:
{code:bash}
beeline -u jdbc:hive2://
{code}
{{CREATE TABLE IF NOT EXISTS}} will not print {{AlreadyExistsException}}, but
{{CREATE DATABASE IF NOT EXISTS}} will.
The root cause is that {{SemanticAnalyzer}} will fast-return if table exists:
https://github.com/apache/hive/blob/98d9d22398370f817fe64449368671b978fff096/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L14163-L14174
was:If a database already exists, we should not throw exception if we try to
create it with {{IF NOT EXISTS}} clause.
> Enable IF NOT EXISTS in CREATE DATABASE statements.
> ---
>
> Key: HIVE-28307
> URL: https://issues.apache.org/jira/browse/HIVE-28307
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
>
> When use beeline for testing purpose:
> {code:bash}
> beeline -u jdbc:hive2://
> {code}
> {{CREATE TABLE IF NOT EXISTS}} will not print {{AlreadyExistsException}}, but
> {{CREATE DATABASE IF NOT EXISTS}} will.
> The root cause is that {{SemanticAnalyzer}} will fast-return if table exists:
> https://github.com/apache/hive/blob/98d9d22398370f817fe64449368671b978fff096/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L14163-L14174
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28311) Backward compatibility of java.sql.Date and java.sql.Timestamp in hive-serde
[
https://issues.apache.org/jira/browse/HIVE-28311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-28311:
--
Description:
HIVE-20007 introduced {{org.apache.hadoop.hive.common.type.Date}} and
{{org.apache.hadoop.hive.common.type.Timestamp}} to replace {{java.sql.Date}}
and {{{}java.sql.Timestamp{}}}.
It's a huge improvements but it also produce incompatibility issues for clients
without this update.
{code:bash}
Caused by: java.lang.ClassCastException: java.sql.Timestamp cannot be cast to
org.apache.hadoop.hive.common.type.Timestamp
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaTimestampObjectInspector.getPrimitiveWritableObject(JavaTimestampObjectInspector.java:33)
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getTimestamp(PrimitiveObjectInspectorUtils.java:1232)
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TimestampConverter.convert(PrimitiveObjectInspectorConverter.java:291)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getConstantObjectInspector(ObjectInspectorUtils.java:1397)
at
org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc.getWritableObjectInspector(ExprNodeConstantDesc.java:93)
at
org.apache.hadoop.hive.ql.exec.ExprNodeConstantEvaluator.(ExprNodeConstantEvaluator.java:41)
at
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:49)
at
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.(ExprNodeGenericFuncEvaluator.java:101)
at
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:58)
at
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:43)
at
org.apache.hadoop.hive.ql.optimizer.ppr.PartExprEvalUtils.prepareExpr(PartExprEvalUtils.java:118)
at
org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prunePartitionNames(PartitionPruner.java:551)
at
org.apache.hadoop.hive.ql.optimizer.ppr.PartitionExpressionForMetastore.filterPartitionsByExpr(PartitionExpressionForMetastore.java:73)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionNamesPrunedByExprNoTxn(ObjectStore.java:3606)
at
org.apache.hadoop.hive.metastore.ObjectStore.access$1000(ObjectStore.java:241)
at
org.apache.hadoop.hive.metastore.ObjectStore$16.getJdoResult(ObjectStore.java:4157)
at
org.apache.hadoop.hive.metastore.ObjectStore$16.getJdoResult(ObjectStore.java:4124)
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3913)
... 30 more
{code}
> Backward compatibility of java.sql.Date and java.sql.Timestamp in hive-serde
>
>
> Key: HIVE-28311
> URL: https://issues.apache.org/jira/browse/HIVE-28311
> Project: Hive
> Issue Type: Bug
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
>
> HIVE-20007 introduced {{org.apache.hadoop.hive.common.type.Date}} and
> {{org.apache.hadoop.hive.common.type.Timestamp}} to replace {{java.sql.Date}}
> and {{{}java.sql.Timestamp{}}}.
> It's a huge improvements but it also produce incompatibility issues for
> clients without this update.
> {code:bash}
> Caused by: java.lang.ClassCastException: java.sql.Timestamp cannot be cast to
> org.apache.hadoop.hive.common.type.Timestamp
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaTimestampObjectInspector.getPrimitiveWritableObject(JavaTimestampObjectInspector.java:33)
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getTimestamp(PrimitiveObjectInspectorUtils.java:1232)
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TimestampConverter.convert(PrimitiveObjectInspectorConverter.java:291)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getConstantObjectInspector(ObjectInspectorUtils.java:1397)
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc.getWritableObjectInspector(ExprNodeConstantDesc.java:93)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeConstantEvaluator.(ExprNodeConstantEvaluator.java:41)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:49)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.(ExprNodeGenericFuncEvaluator.java:101)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:58)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.get(ExprNodeEvaluatorFactory.java:43)
> at
>
[jira] [Created] (HIVE-28311) Backward compatibility of java.sql.Date and java.sql.Timestamp in hive-serde
Wechar created HIVE-28311: - Summary: Backward compatibility of java.sql.Date and java.sql.Timestamp in hive-serde Key: HIVE-28311 URL: https://issues.apache.org/jira/browse/HIVE-28311 Project: Hive Issue Type: Bug Reporter: Wechar Assignee: Wechar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28307) Enable IF NOT EXISTS in CREATE DATABASE statements.
[
https://issues.apache.org/jira/browse/HIVE-28307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-28307:
--
Description: If a database already exists, we should not throw exception if
we try to create it with {{IF NOT EXISTS}} clause.
> Enable IF NOT EXISTS in CREATE DATABASE statements.
> ---
>
> Key: HIVE-28307
> URL: https://issues.apache.org/jira/browse/HIVE-28307
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
>
> If a database already exists, we should not throw exception if we try to
> create it with {{IF NOT EXISTS}} clause.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-28307) Enable IF NOT EXISTS in CREATE DATABASE statements.
Wechar created HIVE-28307: - Summary: Enable IF NOT EXISTS in CREATE DATABASE statements. Key: HIVE-28307 URL: https://issues.apache.org/jira/browse/HIVE-28307 Project: Hive Issue Type: Bug Components: Hive Reporter: Wechar Assignee: Wechar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-28297) Unify table pattern definition in Hive MetaStore
[
https://issues.apache.org/jira/browse/HIVE-28297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-28297 started by Wechar.
-
> Unify table pattern definition in Hive MetaStore
>
>
> Key: HIVE-28297
> URL: https://issues.apache.org/jira/browse/HIVE-28297
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
>
> *Current Situation*
> There are two cases for table pattern in Hive:
> - Case 1: Wildcards ('*' and '|)
> It's replaced by SQL type like expressions with '%' and '_' since Hive 4.0.0,
> details in HIVE-23359.
> DDL documents:
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Show
> - Case 2: Java regular expression
> It's declared in hive code:
> [https://github.com/apache/hive/blob/45867be6cb5308566e4cf16c7b4cf8081085b58c/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1882-L1894],
> and also match the jdo {{matches(String pattern)}} method:
> [https://www.datanucleus.org/products/accessplatform/jdo/query.html]
> *Issues*
> The different definitions cause the misuse of table pattern, in some codes it
> use * to represent match everything, while it use .* in other codes.
> The misuse would cause the wrong result because both client and server side
> of HMS need a trick handle for '*' with code like:
> {code:java}
> patterns.split("\\|")
> pattern.replaceAll("\\*", ".*"))
> {code}
> *Action*
> We should honor the definition in HMS of Case 2, because Case 1 is just the
> frontend of SQL, and it has been replaced.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28297) Unify table pattern definition in Hive MetaStore
[
https://issues.apache.org/jira/browse/HIVE-28297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-28297:
--
Description:
*Current Situation*
There are two cases for table pattern in Hive:
- Case 1: Wildcards ('*' and '|)
It's replaced by SQL type like expressions with '%' and '_' since Hive 4.0.0,
details in HIVE-23359.
DDL documents:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Show
- Case 2: Java regular expression
It's declared in hive code:
[https://github.com/apache/hive/blob/45867be6cb5308566e4cf16c7b4cf8081085b58c/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1882-L1894],
and also match the jdo {{matches(String pattern)}} method:
[https://www.datanucleus.org/products/accessplatform/jdo/query.html]
*Issues*
The different definitions cause the misuse of table pattern, in some codes it
use * to represent match everything, while it use .* in other codes.
The misuse would cause the wrong result because both client and server side of
HMS need a trick handle for '*' with code like:
{code:java}
patterns.split("\\|")
pattern.replaceAll("\\*", ".*"))
{code}
*Action*
We should honor the definition in HMS of Case 2, because Case 1 is just the
frontend of SQL, and it has been replaced.
was:
*Current Situation*
There are two cases for table pattern in Hive:
- Case 1: Wildcards ('*' and '|)
It's replaced by SQL type like expressions with '%' and '_' since Hive 4.0.0,
details in HIVE-23359
- Case 2: Java regular expression
It's declared in hive code:
[https://github.com/apache/hive/blob/45867be6cb5308566e4cf16c7b4cf8081085b58c/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1882-L1894],
and also match the jdo {{matches(String pattern)}} method:
[https://www.datanucleus.org/products/accessplatform/jdo/query.html]
*Issues*
The different definitions cause the misuse of table pattern, in some codes it
use * to represent match everything, while it use .* in other codes.
The misuse would cause the wrong result because both client and server side of
HMS need a trick handle for '*' with code like:
{code:java}
patterns.split("\\|")
pattern.replaceAll("\\*", ".*"))
{code}
*Action*
We should honor the definition in HMS of Case 2, because Case 1 is just the
frontend of SQL, and it has been replaced.
> Unify table pattern definition in Hive MetaStore
>
>
> Key: HIVE-28297
> URL: https://issues.apache.org/jira/browse/HIVE-28297
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
>
> *Current Situation*
> There are two cases for table pattern in Hive:
> - Case 1: Wildcards ('*' and '|)
> It's replaced by SQL type like expressions with '%' and '_' since Hive 4.0.0,
> details in HIVE-23359.
> DDL documents:
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Show
> - Case 2: Java regular expression
> It's declared in hive code:
> [https://github.com/apache/hive/blob/45867be6cb5308566e4cf16c7b4cf8081085b58c/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1882-L1894],
> and also match the jdo {{matches(String pattern)}} method:
> [https://www.datanucleus.org/products/accessplatform/jdo/query.html]
> *Issues*
> The different definitions cause the misuse of table pattern, in some codes it
> use * to represent match everything, while it use .* in other codes.
> The misuse would cause the wrong result because both client and server side
> of HMS need a trick handle for '*' with code like:
> {code:java}
> patterns.split("\\|")
> pattern.replaceAll("\\*", ".*"))
> {code}
> *Action*
> We should honor the definition in HMS of Case 2, because Case 1 is just the
> frontend of SQL, and it has been replaced.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-28297) Unify table pattern definition in Hive MetaStore
Wechar created HIVE-28297:
-
Summary: Unify table pattern definition in Hive MetaStore
Key: HIVE-28297
URL: https://issues.apache.org/jira/browse/HIVE-28297
Project: Hive
Issue Type: Improvement
Components: Hive
Reporter: Wechar
Assignee: Wechar
*Current Situation*
There are two cases for table pattern in Hive:
- Case 1: Wildcards ('*' and '|)
It's replaced by SQL type like expressions with '%' and '_' since Hive 4.0.0,
details in HIVE-23359
- Case 2: Java regular expression
It's declared in hive code:
[https://github.com/apache/hive/blob/45867be6cb5308566e4cf16c7b4cf8081085b58c/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1882-L1894],
and also match the jdo {{matches(String pattern)}} method:
[https://www.datanucleus.org/products/accessplatform/jdo/query.html]
*Issues*
The different definitions cause the misuse of table pattern, in some codes it
use * to represent match everything, while it use .* in other codes.
The misuse would cause the wrong result because both client and server side of
HMS need a trick handle for '*' with code like:
{code:java}
patterns.split("\\|")
pattern.replaceAll("\\*", ".*"))
{code}
*Action*
We should honor the definition in HMS of Case 2, because Case 1 is just the
frontend of SQL, and it has been replaced.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28292) Optimize SHOW TABLES|VIEWS statements
[
https://issues.apache.org/jira/browse/HIVE-28292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-28292:
--
Description:
There are some improvement points in {{SHOW TABLES}} statement:
1. pattern is not pushed to metastore server
2. get table objects even if only table name and table type is needed.
3. index {{UNIQUETABLE(TBL_NAME, DB_ID)}} in {{TBLS}} table is not efficient
enough compared to index {{(DB_ID, TBL_NAME)}} when querying tables by name
pattern.
> Optimize SHOW TABLES|VIEWS statements
> -
>
> Key: HIVE-28292
> URL: https://issues.apache.org/jira/browse/HIVE-28292
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
>
> There are some improvement points in {{SHOW TABLES}} statement:
> 1. pattern is not pushed to metastore server
> 2. get table objects even if only table name and table type is needed.
> 3. index {{UNIQUETABLE(TBL_NAME, DB_ID)}} in {{TBLS}} table is not efficient
> enough compared to index {{(DB_ID, TBL_NAME)}} when querying tables by name
> pattern.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work started] (HIVE-28292) Optimize SHOW TABLES|VIEWS statements
[
https://issues.apache.org/jira/browse/HIVE-28292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-28292 started by Wechar.
-
> Optimize SHOW TABLES|VIEWS statements
> -
>
> Key: HIVE-28292
> URL: https://issues.apache.org/jira/browse/HIVE-28292
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
>
> There are some improvement points in {{SHOW TABLES}} statement:
> 1. pattern is not pushed to metastore server
> 2. get table objects even if only table name and table type is needed.
> 3. index {{UNIQUETABLE(TBL_NAME, DB_ID)}} in {{TBLS}} table is not efficient
> enough compared to index {{(DB_ID, TBL_NAME)}} when querying tables by name
> pattern.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-28292) Optimize SHOW TABLES|VIEWS statements
Wechar created HIVE-28292: - Summary: Optimize SHOW TABLES|VIEWS statements Key: HIVE-28292 URL: https://issues.apache.org/jira/browse/HIVE-28292 Project: Hive Issue Type: Improvement Components: Hive Reporter: Wechar Assignee: Wechar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-28279) Output the database name for SHOW EXTENDED TABLES statement
[
https://issues.apache.org/jira/browse/HIVE-28279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar resolved HIVE-28279.
---
Fix Version/s: Not Applicable
Resolution: Won't Do
> Output the database name for SHOW EXTENDED TABLES statement
> ---
>
> Key: HIVE-28279
> URL: https://issues.apache.org/jira/browse/HIVE-28279
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Fix For: Not Applicable
>
>
> HIVE-21301 introduced {{SHOW EXTENDED TABLES}} statement which output table
> name and table type while listing tables in a database.
> In this patch, we aim to add a new output filed for database name with
> following reasons:
> 1. database name in {{SHOW EXTENDED TABLES}} statement is optional, output
> the database is informal in this case.
> 2. when statistic table names and database names by this statement for list
> of databases, the output result including database name is much more helpful.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work stopped] (HIVE-28279) Output the database name for SHOW EXTENDED TABLES statement
[
https://issues.apache.org/jira/browse/HIVE-28279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-28279 stopped by Wechar.
-
> Output the database name for SHOW EXTENDED TABLES statement
> ---
>
> Key: HIVE-28279
> URL: https://issues.apache.org/jira/browse/HIVE-28279
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
>
> HIVE-21301 introduced {{SHOW EXTENDED TABLES}} statement which output table
> name and table type while listing tables in a database.
> In this patch, we aim to add a new output filed for database name with
> following reasons:
> 1. database name in {{SHOW EXTENDED TABLES}} statement is optional, output
> the database is informal in this case.
> 2. when statistic table names and database names by this statement for list
> of databases, the output result including database name is much more helpful.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work started] (HIVE-28279) Output the database name for SHOW EXTENDED TABLES statement
[
https://issues.apache.org/jira/browse/HIVE-28279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-28279 started by Wechar.
-
> Output the database name for SHOW EXTENDED TABLES statement
> ---
>
> Key: HIVE-28279
> URL: https://issues.apache.org/jira/browse/HIVE-28279
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
>
> HIVE-21301 introduced {{SHOW EXTENDED TABLES}} statement which output table
> name and table type while listing tables in a database.
> In this patch, we aim to add a new output filed for database name with
> following reasons:
> 1. database name in {{SHOW EXTENDED TABLES}} statement is optional, output
> the database is informal in this case.
> 2. when statistic table names and database names by this statement for list
> of databases, the output result including database name is much more helpful.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-28279) Output the database name for SHOW EXTENDED TABLES statement
Wechar created HIVE-28279:
-
Summary: Output the database name for SHOW EXTENDED TABLES
statement
Key: HIVE-28279
URL: https://issues.apache.org/jira/browse/HIVE-28279
Project: Hive
Issue Type: Task
Components: Hive
Reporter: Wechar
Assignee: Wechar
HIVE-21301 introduced {{SHOW EXTENDED TABLES}} statement which output table
name and table type while listing tables in a database.
In this patch, we aim to add a new output filed for database name with
following reasons:
1. database name in {{SHOW EXTENDED TABLES}} statement is optional, output the
database is informal in this case.
2. when statistic table names and database names by this statement for list of
databases, the output result including database name is much more helpful.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-28270) Fix missing partition paths bug on drop_database
Wechar created HIVE-28270:
-
Summary: Fix missing partition paths bug on drop_database
Key: HIVE-28270
URL: https://issues.apache.org/jira/browse/HIVE-28270
Project: Hive
Issue Type: Bug
Components: Hive
Reporter: Wechar
Assignee: Wechar
In {{HMSHandler#drop_database_core}}, it needs to collect all partition paths
that were not in the subdirectory of the table path, but now it only fetch the
last batch of paths.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-28239) Fix bug on HMSHandler#checkLimitNumberOfPartitions
Wechar created HIVE-28239:
-
Summary: Fix bug on HMSHandler#checkLimitNumberOfPartitions
Key: HIVE-28239
URL: https://issues.apache.org/jira/browse/HIVE-28239
Project: Hive
Issue Type: Bug
Components: Hive
Reporter: Wechar
Assignee: Wechar
{{HMSHandler#checkLimitNumberOfPartitions}} should not compare request size,
which can cause the incorrect limit check.
Assume that HMS configure {{metastore.limit.partition.request}} as 100, the
client calls {{get_partitions_by_filter}} with maxParts as 101, and the
matching partition number is 50, in this case the HMS server should not throw
MetaException by partition limit check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-28024) Support sources profile for hive metastore modules
[ https://issues.apache.org/jira/browse/HIVE-28024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar resolved HIVE-28024. --- Fix Version/s: 4.0.0 Resolution: Won't Fix > Support sources profile for hive metastore modules > -- > > Key: HIVE-28024 > URL: https://issues.apache.org/jira/browse/HIVE-28024 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0-beta-1 >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Maven profile sources in Hive supports to generate source jar for all > submodules. Since Hive Metastore modules are separated from hive parent > module, it would be convenient to add this profile in standalone-metastore > module. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-28024) Support sources profile for hive metastore modules
[ https://issues.apache.org/jira/browse/HIVE-28024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842050#comment-17842050 ] Wechar commented on HIVE-28024: --- Thanks [~zabetak], close this jira now. > Support sources profile for hive metastore modules > -- > > Key: HIVE-28024 > URL: https://issues.apache.org/jira/browse/HIVE-28024 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0-beta-1 >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > > Maven profile sources in Hive supports to generate source jar for all > submodules. Since Hive Metastore modules are separated from hive parent > module, it would be convenient to add this profile in standalone-metastore > module. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-28024) Support sources profile for hive metastore modules
[ https://issues.apache.org/jira/browse/HIVE-28024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841307#comment-17841307 ] Wechar commented on HIVE-28024: --- Because the root pom.xml in Hive defines the [sources|https://github.com/apache/hive/blob/4614ce72a7f366674d89a3a78f687e419400cb89/pom.xml#L1940] profile explicitly, so I think it's also needed in metastore modules. But since apache-release profile is available, do we still need sources profile in Hive? > Support sources profile for hive metastore modules > -- > > Key: HIVE-28024 > URL: https://issues.apache.org/jira/browse/HIVE-28024 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0-beta-1 >Reporter: Wechar >Assignee: Wechar >Priority: Major > Labels: pull-request-available > > Maven profile sources in Hive supports to generate source jar for all > submodules. Since Hive Metastore modules are separated from hive parent > module, it would be convenient to add this profile in standalone-metastore > module. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28219) Support drop partitions by names in IMetaStoreClient
Wechar created HIVE-28219:
-
Summary: Support drop partitions by names in IMetaStoreClient
Key: HIVE-28219
URL: https://issues.apache.org/jira/browse/HIVE-28219
Project: Hive
Issue Type: New Feature
Components: Hive
Reporter: Wechar
Assignee: Wechar
In thrift api definition, HMS support to drop partitions by both partition
names and expressions:
{code:bash}
// hive_metastore.thrift
DropPartitionsResult drop_partitions_req(1: DropPartitionsRequest req)
throws(1:NoSuchObjectException o1, 2:MetaException o2)
// Request type for drop_partitions_req
struct DropPartitionsRequest {
// ...
3: required RequestPartsSpec parts,
// ...
}
union RequestPartsSpec {
1: list names;
2: list exprs;
}
{code}
But current api in `IMetaStoreClient` only support drop partitions by
expressions, we should add new api to support drop partitions by names.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work started] (HIVE-28205) Implement direct sql for get_partitions_ps_with_auth api
[ https://issues.apache.org/jira/browse/HIVE-28205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-28205 started by Wechar. - > Implement direct sql for get_partitions_ps_with_auth api > > > Key: HIVE-28205 > URL: https://issues.apache.org/jira/browse/HIVE-28205 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28205) Implement direct sql for get_partitions_ps_with_auth api
Wechar created HIVE-28205: - Summary: Implement direct sql for get_partitions_ps_with_auth api Key: HIVE-28205 URL: https://issues.apache.org/jira/browse/HIVE-28205 Project: Hive Issue Type: Improvement Components: Hive Reporter: Wechar Assignee: Wechar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28200) Improve get_partitions_by_filter/expr when partition limit enabled
Wechar created HIVE-28200:
-
Summary: Improve get_partitions_by_filter/expr when partition
limit enabled
Key: HIVE-28200
URL: https://issues.apache.org/jira/browse/HIVE-28200
Project: Hive
Issue Type: Improvement
Components: Hive
Reporter: Wechar
Assignee: Wechar
When {{hive.metastore.limit.partition.request}} is configured, HMS would get
the matching partition counts before get the real partition objects. The count
could be a slow query if the input filter or expr is too complex.
In this case, such slow filter will be executed in both counting partition
numbers and fetching real partition objects, which harms the performance and
backend DBMS.
We can make an improvement by getting matched partition names firstly, and then
check limit through the size of partition names, and finally get the partitions
by the partition names.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28096) Fix the mismatched function names in HMSHandler audit log
[ https://issues.apache.org/jira/browse/HIVE-28096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar updated HIVE-28096: -- Summary: Fix the mismatched function names in HMSHandler audit log (was: Fix the mismatched function names in HMSHandler ) > Fix the mismatched function names in HMSHandler audit log > - > > Key: HIVE-28096 > URL: https://issues.apache.org/jira/browse/HIVE-28096 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0-alpha-2 >Reporter: Wechar >Assignee: Wechar >Priority: Major > > Some function names in startFunction and endFunction are inconsistent, which > would cause the counter and api cost metrics wrong. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28096) Fix the mismatched function names in HMSHandler
Wechar created HIVE-28096: - Summary: Fix the mismatched function names in HMSHandler Key: HIVE-28096 URL: https://issues.apache.org/jira/browse/HIVE-28096 Project: Hive Issue Type: Bug Components: Hive Affects Versions: 4.0.0-alpha-2 Reporter: Wechar Assignee: Wechar Some function names in startFunction and endFunction are inconsistent, which would cause the counter and api cost metrics wrong. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28062) Optimize get_partitions_by_names in direct sql
[
https://issues.apache.org/jira/browse/HIVE-28062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-28062:
--
Summary: Optimize get_partitions_by_names in direct sql (was: Optimize
get_partitions_by_names direct sql)
> Optimize get_partitions_by_names in direct sql
> --
>
> Key: HIVE-28062
> URL: https://issues.apache.org/jira/browse/HIVE-28062
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0-beta-1
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
>
> Currently {{MetaStoreDirectSql#getPartitionsViaPartNames}} would first get
> partition id through partition name in each batch, which was unnecessary
> because we can get all fields through partition name directly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-28062) Optimize get_partitions_by_names direct sql
Wechar created HIVE-28062:
-
Summary: Optimize get_partitions_by_names direct sql
Key: HIVE-28062
URL: https://issues.apache.org/jira/browse/HIVE-28062
Project: Hive
Issue Type: Improvement
Components: Hive
Affects Versions: 4.0.0-beta-1
Reporter: Wechar
Assignee: Wechar
Currently {{MetaStoreDirectSql#getPartitionsViaPartNames}} would first get
partition id through partition name in each batch, which was unnecessary
because we can get all fields through partition name directly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-28038) Disable fallback to jdo for DeadlineException
Wechar created HIVE-28038:
-
Summary: Disable fallback to jdo for DeadlineException
Key: HIVE-28038
URL: https://issues.apache.org/jira/browse/HIVE-28038
Project: Hive
Issue Type: Improvement
Components: Hive
Affects Versions: 4.0.0-beta-1
Reporter: Wechar
Assignee: Wechar
When {{DeadlineException}} occurs, the call should be terminated immediately
instead of falling back to jdo query, because some jdo query does not take use
of {{{}Deadline.checkTimeout(){}}}.
{*}How to Reproduce{*}:
Step 1: start a metastore server with metastore config:
{code:xml}
hive.metastore.client.socket.timeout
60s
{code}
Step 2: Create a table with large partitions
Step 3: set hive client socket timeout same as server, and call drop_table to
drop this table
{code:java}
conf.set("hive.metastore.client.socket.timeout", "60s");
conf.set("hive.metastore.failure.retries", "0");
{code}
The direct sql query would throw DeadlineException while dropping partitions in
batch, but it would fall back to jdo query to drop partitions for each batch,
we can find clues from the server log:
{code:bash}
2024-01-26T08:52:14,877 INFO [Metastore-Handler-Pool: Thread-54]:
HiveMetaStore.audit (HMSHandler.java:logAuditEvent(207)) - ugi=root
ip=10.89.0.1cmd=source:10.89.0.1 get_all_functions
2024-01-26T08:52:14,879 INFO [Metastore-Handler-Pool: Thread-54]:
metastore.HMSHandler (HMSHandler.java:newRawStoreForConf(600)) - Opening raw
store with implementation class: org.apache.hadoop.hive.metastore.ObjectStore
2024-01-26T08:52:14,879 INFO [Metastore-Handler-Pool: Thread-54]:
metastore.PersistenceManagerProvider
(PersistenceManagerProvider.java:correctAutoStartMechanism(577)) -
Configuration datanucleus.autoStartMechanismMode is not set
. Defaulting to 'ignored'
2024-01-26T08:52:14,879 INFO [Metastore-Handler-Pool: Thread-54]:
metastore.ObjectStore (ObjectStore.java:shutdown(568)) - RawStore:
org.apache.hadoop.hive.metastore.ObjectStore@53c9fbcd, with PersistenceManager:
null will be shu
tdown
2024-01-26T08:52:14,881 INFO [Metastore-Handler-Pool: Thread-54]:
metastore.ObjectStore (ObjectStore.java:initialize(430)) - RawStore:
org.apache.hadoop.hive.metastore.ObjectStore@53c9fbcd, with PersistenceManager:
org.datanucleu
s.api.jdo.JDOPersistenceManager@586cd877 created in the thread with id: 54
2024-01-26T08:52:14,890 INFO [Metastore-Handler-Pool: Thread-54]:
metastore.HMSHandler (HMSHandler.java:getMSForConf(587)) - Created RawStore:
org.apache.hadoop.hive.metastore.ObjectStore@53c9fbcd
2024-01-26T08:52:14,913 INFO [Metastore-Handler-Pool: Thread-54]:
HiveMetaStore.audit (HMSHandler.java:logAuditEvent(207)) - ugi=root
ip=10.89.0.1cmd=source:10.89.0.1 get_table : tbl=hive.default.test_tbl
2024-01-26T08:52:15,069 INFO [Metastore-Handler-Pool: Thread-54]:
metastore.HMSHandler (HMSHandler.java:getTableInternal(3582)) - Skipping
translation for processor with null
2024-01-26T08:52:15,094 INFO [Metastore-Handler-Pool: Thread-54]:
HiveMetaStore.audit (HMSHandler.java:logAuditEvent(207)) - ugi=root
ip=10.89.0.1cmd=source:10.89.0.1 drop_table : tbl=hive.default.test_tbl
2024-01-26T08:52:54,593 INFO [Metastore Scheduled Worker 1]:
metastore.HMSHandler (HMSHandler.java:newRawStoreForConf(600)) - Opening raw
store with implementation class: org.apache.hadoop.hive.metastore.ObjectStore
2024-01-26T08:52:54,594 INFO [Metastore Scheduled Worker 1]:
metastore.PersistenceManagerProvider
(PersistenceManagerProvider.java:correctAutoStartMechanism(577)) -
Configuration datanucleus.autoStartMechanismMode is not set. Def
aulting to 'ignored'
2024-01-26T08:52:54,594 INFO [Metastore Scheduled Worker 1]:
metastore.ObjectStore (ObjectStore.java:shutdown(568)) - RawStore:
org.apache.hadoop.hive.metastore.ObjectStore@63e26ab7, with PersistenceManager:
null will be shutdown
2024-01-26T08:52:54,595 INFO [Metastore Scheduled Worker 1]:
metastore.ObjectStore (ObjectStore.java:initialize(430)) - RawStore:
org.apache.hadoop.hive.metastore.ObjectStore@63e26ab7, with PersistenceManager:
org.datanucleus.api
.jdo.JDOPersistenceManager@19d73712 created in the thread with id: 48
2024-01-26T08:52:54,601 INFO [Metastore Scheduled Worker 1]:
metastore.HMSHandler (HMSHandler.java:getMSForConf(587)) - Created RawStore:
org.apache.hadoop.hive.metastore.ObjectStore@63e26ab7
2024-01-26T08:53:15,125 WARN [Metastore-Handler-Pool: Thread-54]:
metastore.ObjectStore (ObjectStore.java:handleDirectSqlError(4455)) - Falling
back to ORM path due to direct SQL failure (this is not an error): Timeout when
execu
ting method: drop_table_with_environment_context; 60021ms exceeds 6ms at
org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.newMetaException(MetaStoreUtils.java:218)
at org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.
[jira] [Created] (HIVE-28024) Support sources profile for hive metastore modules
Wechar created HIVE-28024: - Summary: Support sources profile for hive metastore modules Key: HIVE-28024 URL: https://issues.apache.org/jira/browse/HIVE-28024 Project: Hive Issue Type: Improvement Components: Hive Affects Versions: 4.0.0-beta-1 Reporter: Wechar Assignee: Wechar Maven profile sources in Hive supports to generate source jar for all submodules. Since Hive Metastore modules are separated from hive parent module, it would be convenient to add this profile in standalone-metastore module. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-28014) to_unix_timestamp udf produces inconsistent results in different jdk versions
[
https://issues.apache.org/jira/browse/HIVE-28014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17808601#comment-17808601
]
Wechar commented on HIVE-28014:
---
I have downgraded the jdk8 version in wecharyu/hive-dev-box:executor to avoid
the CI failed temporally. It needs more clues to determine which side is not
correct.
> to_unix_timestamp udf produces inconsistent results in different jdk versions
> -
>
> Key: HIVE-28014
> URL: https://issues.apache.org/jira/browse/HIVE-28014
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0-beta-1
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
>
> In HIVE-27999 we update the CI docker image which upgrades jdk8 from
> {*}1.8.0_262-b19{*} to *1.8.0_392-b08*. This upgrade cause 3 timestamp
> related tests failed:
> *1. Testing / split-02 / PostProcess /
> testTimestampToString[zoneId=Europe/Paris, timestamp=2417-03-26T02:08:43] –
> org.apache.hadoop.hive.metastore.utils.TestMetaStoreUtils*
> {code:bash}
> Error
> expected:<2417-03-26 0[2]:08:43> but was:<2417-03-26 0[3]:08:43>
> Stacktrace
> org.junit.ComparisonFailure: expected:<2417-03-26 0[2]:08:43> but
> was:<2417-03-26 0[3]:08:43>
> at
> org.apache.hadoop.hive.metastore.utils.TestMetaStoreUtils.testTimestampToString(TestMetaStoreUtils.java:85)
> {code}
> *2. Testing / split-01 / PostProcess / testCliDriver[udf5] –
> org.apache.hadoop.hive.cli.split24.TestMiniLlapLocalCliDriver*
> {code:bash}
> Error
> Client Execution succeeded but contained differences (error code = 1) after
> executing udf5.q
> 263c263
> < 1400-11-08 07:35:34
> ---
> > 1400-11-08 07:35:24
> 272c272
> < 1800-11-08 07:35:34
> ---
> > 1800-11-08 07:35:24
> 434c434
> < 1399-12-31 23:35:34
> ---
> > 1399-12-31 23:35:24
> 443c443
> < 1799-12-31 23:35:34
> ---
> > 1799-12-31 23:35:24
> 452c452
> < 1899-12-31 23:35:34
> ---
> > 1899-12-31 23:35:24
> {code}
> *3. Testing / split-19 / PostProcess / testStringArg2 –
> org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFToUnixTimestamp*
> {code:bash}
> Stacktrace
> org.junit.ComparisonFailure: expected:<-17984790[40]0> but
> was:<-17984790[39]0>
> at org.junit.Assert.assertEquals(Assert.java:117)
> at org.junit.Assert.assertEquals(Assert.java:146)
> at
> org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFToUnixTimestamp.runAndVerify(TestGenericUDFToUnixTimestamp.java:70)
> at
> org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFToUnixTimestamp.testStringArg2(TestGenericUDFToUnixTimestamp.java:167)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> {code}
> It maybe a jdk bug and fixed in the new release, because we could get the
> same result from Spark:
> {code:sql}
> spark-sql> select to_unix_timestamp(to_timestamp("1400-02-01 00:00:00 ICT",
> "-MM-dd HH:mm:ss z"), "US/Pacific");
> -17984790390
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-28014) to_unix_timestamp udf produces inconsistent results in different jdk versions
Wechar created HIVE-28014:
-
Summary: to_unix_timestamp udf produces inconsistent results in
different jdk versions
Key: HIVE-28014
URL: https://issues.apache.org/jira/browse/HIVE-28014
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 4.0.0-beta-1
Reporter: Wechar
In HIVE-27999 we update the CI docker image which upgrades jdk8 from
{*}1.8.0_262-b19{*} to *1.8.0_392-b08*. This upgrade cause 3 timestamp related
tests failed:
*1. Testing / split-02 / PostProcess /
testTimestampToString[zoneId=Europe/Paris, timestamp=2417-03-26T02:08:43] –
org.apache.hadoop.hive.metastore.utils.TestMetaStoreUtils*
{code:bash}
Error
expected:<2417-03-26 0[2]:08:43> but was:<2417-03-26 0[3]:08:43>
Stacktrace
org.junit.ComparisonFailure: expected:<2417-03-26 0[2]:08:43> but
was:<2417-03-26 0[3]:08:43>
at
org.apache.hadoop.hive.metastore.utils.TestMetaStoreUtils.testTimestampToString(TestMetaStoreUtils.java:85)
{code}
*2. Testing / split-01 / PostProcess / testCliDriver[udf5] –
org.apache.hadoop.hive.cli.split24.TestMiniLlapLocalCliDriver*
{code:bash}
Error
Client Execution succeeded but contained differences (error code = 1) after
executing udf5.q
263c263
< 1400-11-08 07:35:34
---
> 1400-11-08 07:35:24
272c272
< 1800-11-08 07:35:34
---
> 1800-11-08 07:35:24
434c434
< 1399-12-31 23:35:34
---
> 1399-12-31 23:35:24
443c443
< 1799-12-31 23:35:34
---
> 1799-12-31 23:35:24
452c452
< 1899-12-31 23:35:34
---
> 1899-12-31 23:35:24
{code}
*3. Testing / split-19 / PostProcess / testStringArg2 –
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFToUnixTimestamp*
{code:bash}
Stacktrace
org.junit.ComparisonFailure: expected:<-17984790[40]0> but was:<-17984790[39]0>
at org.junit.Assert.assertEquals(Assert.java:117)
at org.junit.Assert.assertEquals(Assert.java:146)
at
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFToUnixTimestamp.runAndVerify(TestGenericUDFToUnixTimestamp.java:70)
at
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFToUnixTimestamp.testStringArg2(TestGenericUDFToUnixTimestamp.java:167)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
{code}
It maybe a jdk bug and fixed in the new release, because we could get the same
result from Spark:
{code:sql}
spark-sql> select to_unix_timestamp(to_timestamp("1400-02-01 00:00:00 ICT",
"-MM-dd HH:mm:ss z"), "US/Pacific");
-17984790390
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-28014) to_unix_timestamp udf produces inconsistent results in different jdk versions
[
https://issues.apache.org/jira/browse/HIVE-28014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar reassigned HIVE-28014:
-
Assignee: Wechar
> to_unix_timestamp udf produces inconsistent results in different jdk versions
> -
>
> Key: HIVE-28014
> URL: https://issues.apache.org/jira/browse/HIVE-28014
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0-beta-1
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
>
> In HIVE-27999 we update the CI docker image which upgrades jdk8 from
> {*}1.8.0_262-b19{*} to *1.8.0_392-b08*. This upgrade cause 3 timestamp
> related tests failed:
> *1. Testing / split-02 / PostProcess /
> testTimestampToString[zoneId=Europe/Paris, timestamp=2417-03-26T02:08:43] –
> org.apache.hadoop.hive.metastore.utils.TestMetaStoreUtils*
> {code:bash}
> Error
> expected:<2417-03-26 0[2]:08:43> but was:<2417-03-26 0[3]:08:43>
> Stacktrace
> org.junit.ComparisonFailure: expected:<2417-03-26 0[2]:08:43> but
> was:<2417-03-26 0[3]:08:43>
> at
> org.apache.hadoop.hive.metastore.utils.TestMetaStoreUtils.testTimestampToString(TestMetaStoreUtils.java:85)
> {code}
> *2. Testing / split-01 / PostProcess / testCliDriver[udf5] –
> org.apache.hadoop.hive.cli.split24.TestMiniLlapLocalCliDriver*
> {code:bash}
> Error
> Client Execution succeeded but contained differences (error code = 1) after
> executing udf5.q
> 263c263
> < 1400-11-08 07:35:34
> ---
> > 1400-11-08 07:35:24
> 272c272
> < 1800-11-08 07:35:34
> ---
> > 1800-11-08 07:35:24
> 434c434
> < 1399-12-31 23:35:34
> ---
> > 1399-12-31 23:35:24
> 443c443
> < 1799-12-31 23:35:34
> ---
> > 1799-12-31 23:35:24
> 452c452
> < 1899-12-31 23:35:34
> ---
> > 1899-12-31 23:35:24
> {code}
> *3. Testing / split-19 / PostProcess / testStringArg2 –
> org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFToUnixTimestamp*
> {code:bash}
> Stacktrace
> org.junit.ComparisonFailure: expected:<-17984790[40]0> but
> was:<-17984790[39]0>
> at org.junit.Assert.assertEquals(Assert.java:117)
> at org.junit.Assert.assertEquals(Assert.java:146)
> at
> org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFToUnixTimestamp.runAndVerify(TestGenericUDFToUnixTimestamp.java:70)
> at
> org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFToUnixTimestamp.testStringArg2(TestGenericUDFToUnixTimestamp.java:167)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> {code}
> It maybe a jdk bug and fixed in the new release, because we could get the
> same result from Spark:
> {code:sql}
> spark-sql> select to_unix_timestamp(to_timestamp("1400-02-01 00:00:00 ICT",
> "-MM-dd HH:mm:ss z"), "US/Pacific");
> -17984790390
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27999) Upgrade to Java 17 to run sonar analysis
[
https://issues.apache.org/jira/browse/HIVE-27999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-27999:
--
Description:
http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4905/4/pipeline/596/
CI failed by sonar plugin:
{code:bash}
[2024-01-15T03:58:10.100Z] [INFO] BUILD FAILURE
[2024-01-15T03:58:10.100Z] [INFO]
[2024-01-15T03:58:10.100Z] [INFO] Total time: 01:07 min
[2024-01-15T03:58:10.100Z] [INFO] Finished at: 2024-01-15T03:58:09Z
[2024-01-15T03:58:10.100Z] [INFO]
[2024-01-15T03:58:10.100Z] [ERROR] Failed to execute goal
org.sonarsource.scanner.maven:sonar-maven-plugin:3.9.1.2184:sonar (default-cli)
on project hive:
[2024-01-15T03:58:10.100Z] [ERROR]
[2024-01-15T03:58:10.100Z] [ERROR] The version of Java (11.0.8) used to run
this analysis is deprecated, and SonarCloud no longer supports it. Please
upgrade to Java 17 or later.
[2024-01-15T03:58:10.100Z] [ERROR] As a temporary measure, you can set the
property 'sonar.scanner.force-deprecated-java-version' to 'true' to continue
using Java 11.0.8
[2024-01-15T03:58:10.100Z] [ERROR] This workaround will only be effective until
January 28, 2024. After this date, all scans using the deprecated Java 11 will
fail.
[2024-01-15T03:58:10.100Z] [ERROR] -> [Help 1]
[2024-01-15T03:58:10.100Z] [ERROR]
[2024-01-15T03:58:10.100Z] [ERROR] To see the full stack trace of the errors,
re-run Maven with the -e switch.
[2024-01-15T03:58:10.100Z] [ERROR] Re-run Maven using the -X switch to enable
full debug logging.
{code}
was:
CI failed by sonar plugin:
{code:bash}
[2024-01-15T03:58:10.100Z] [INFO] BUILD FAILURE
[2024-01-15T03:58:10.100Z] [INFO]
[2024-01-15T03:58:10.100Z] [INFO] Total time: 01:07 min
[2024-01-15T03:58:10.100Z] [INFO] Finished at: 2024-01-15T03:58:09Z
[2024-01-15T03:58:10.100Z] [INFO]
[2024-01-15T03:58:10.100Z] [ERROR] Failed to execute goal
org.sonarsource.scanner.maven:sonar-maven-plugin:3.9.1.2184:sonar (default-cli)
on project hive:
[2024-01-15T03:58:10.100Z] [ERROR]
[2024-01-15T03:58:10.100Z] [ERROR] The version of Java (11.0.8) used to run
this analysis is deprecated, and SonarCloud no longer supports it. Please
upgrade to Java 17 or later.
[2024-01-15T03:58:10.100Z] [ERROR] As a temporary measure, you can set the
property 'sonar.scanner.force-deprecated-java-version' to 'true' to continue
using Java 11.0.8
[2024-01-15T03:58:10.100Z] [ERROR] This workaround will only be effective until
January 28, 2024. After this date, all scans using the deprecated Java 11 will
fail.
[2024-01-15T03:58:10.100Z] [ERROR] -> [Help 1]
[2024-01-15T03:58:10.100Z] [ERROR]
[2024-01-15T03:58:10.100Z] [ERROR] To see the full stack trace of the errors,
re-run Maven with the -e switch.
[2024-01-15T03:58:10.100Z] [ERROR] Re-run Maven using the -X switch to enable
full debug logging.
{code}
> Upgrade to Java 17 to run sonar analysis
>
>
> Key: HIVE-27999
> URL: https://issues.apache.org/jira/browse/HIVE-27999
> Project: Hive
> Issue Type: Bug
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
>
> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4905/4/pipeline/596/
> CI failed by sonar plugin:
> {code:bash}
> [2024-01-15T03:58:10.100Z] [INFO] BUILD FAILURE
> [2024-01-15T03:58:10.100Z] [INFO]
>
> [2024-01-15T03:58:10.100Z] [INFO] Total time: 01:07 min
> [2024-01-15T03:58:10.100Z] [INFO] Finished at: 2024-01-15T03:58:09Z
> [2024-01-15T03:58:10.100Z] [INFO]
>
> [2024-01-15T03:58:10.100Z] [ERROR] Failed to execute goal
> org.sonarsource.scanner.maven:sonar-maven-plugin:3.9.1.2184:sonar
> (default-cli) on project hive:
> [2024-01-15T03:58:10.100Z] [ERROR]
> [2024-01-15T03:58:10.100Z] [ERROR] The version of Java (11.0.8) used to run
> this analysis is deprecated, and SonarCloud no longer supports it. Please
> upgrade to Java 17 or later.
> [2024-01-15T03:58:10.100Z] [ERROR] As a temporary measure, you can set the
> property 'sonar.scanner.force-deprecated-java-version' to 'true' to continue
> using Java 11.0.8
> [2024-01-15T03:58:10.100Z] [ERROR] This workaround will only be effective
> until January 28, 2024. After this date, all scans using the deprecated Java
> 11 will fail.
> [2024-01-15T03:58:10.100Z] [ERROR] -> [Help 1]
> [2024-01-15T03:58:10.100Z] [ERROR]
>
[jira] [Created] (HIVE-27999) Upgrade to Java 17 to run sonar analysis
Wechar created HIVE-27999:
-
Summary: Upgrade to Java 17 to run sonar analysis
Key: HIVE-27999
URL: https://issues.apache.org/jira/browse/HIVE-27999
Project: Hive
Issue Type: Bug
Reporter: Wechar
Assignee: Wechar
CI failed by sonar plugin:
{code:bash}
[2024-01-15T03:58:10.100Z] [INFO] BUILD FAILURE
[2024-01-15T03:58:10.100Z] [INFO]
[2024-01-15T03:58:10.100Z] [INFO] Total time: 01:07 min
[2024-01-15T03:58:10.100Z] [INFO] Finished at: 2024-01-15T03:58:09Z
[2024-01-15T03:58:10.100Z] [INFO]
[2024-01-15T03:58:10.100Z] [ERROR] Failed to execute goal
org.sonarsource.scanner.maven:sonar-maven-plugin:3.9.1.2184:sonar (default-cli)
on project hive:
[2024-01-15T03:58:10.100Z] [ERROR]
[2024-01-15T03:58:10.100Z] [ERROR] The version of Java (11.0.8) used to run
this analysis is deprecated, and SonarCloud no longer supports it. Please
upgrade to Java 17 or later.
[2024-01-15T03:58:10.100Z] [ERROR] As a temporary measure, you can set the
property 'sonar.scanner.force-deprecated-java-version' to 'true' to continue
using Java 11.0.8
[2024-01-15T03:58:10.100Z] [ERROR] This workaround will only be effective until
January 28, 2024. After this date, all scans using the deprecated Java 11 will
fail.
[2024-01-15T03:58:10.100Z] [ERROR] -> [Help 1]
[2024-01-15T03:58:10.100Z] [ERROR]
[2024-01-15T03:58:10.100Z] [ERROR] To see the full stack trace of the errors,
re-run Maven with the -e switch.
[2024-01-15T03:58:10.100Z] [ERROR] Re-run Maven using the -X switch to enable
full debug logging.
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27857) Do not check write permission while dropping external table or partition
[
https://issues.apache.org/jira/browse/HIVE-27857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar resolved HIVE-27857.
---
Fix Version/s: 4.0.0
Resolution: Fixed
> Do not check write permission while dropping external table or partition
>
>
> Key: HIVE-27857
> URL: https://issues.apache.org/jira/browse/HIVE-27857
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> When drop table for external table, we actually not need delete the table
> path. But now HMS will check write permission if
> {{AUTHORIZATION_STORAGE_AUTH_CHECKS}} enabled no matter the table is external
> or not.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27775) DirectSQL and JDO results are different when fetching partitions by timestamp in DST shift
[
https://issues.apache.org/jira/browse/HIVE-27775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17800453#comment-17800453
]
Wechar commented on HIVE-27775:
---
+1. We also noticed this bug that jdo query would return empty result for
timestamp filter in the benchmark test of HIVE-27827.
> DirectSQL and JDO results are different when fetching partitions by timestamp
> in DST shift
> --
>
> Key: HIVE-27775
> URL: https://issues.apache.org/jira/browse/HIVE-27775
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0-beta-1
>Reporter: Stamatis Zampetakis
>Assignee: Zhihua Deng
>Priority: Critical
> Labels: pull-request-available
>
> DirectSQL and JDO results are different when fetching partitions by timestamp
> in DST shift.
> {code:sql}
> --! qt:timezone:Europe/Paris
> CREATE EXTERNAL TABLE payments (card string) PARTITIONED BY(txn_datetime
> TIMESTAMP) STORED AS ORC;
> INSERT into payments VALUES('---', '2023-03-26 02:30:00');
> SELECT * FROM payments WHERE txn_datetime = '2023-03-26 02:30:00';
> {code}
> The '2023-03-26 02:30:00' is a timestamp that in Europe/Paris timezone falls
> exactly in the middle of the DST shift. In this particular timezone this date
> time never really exists since we are jumping directly from 02:00:00 to
> 03:00:00. However, the TIMESTAMP data type in Hive is timezone agnostic
> (https://cwiki.apache.org/confluence/display/Hive/Different+TIMESTAMP+types)
> so it is a perfectly valid timestamp that can be inserted in a table and we
> must be able to recover it back.
> For the SELECT query above, partition pruning kicks in and calls the
> ObjectStore#getPartitionsByExpr method in order to fetch the respective
> partitions matching the timestamp from HMS.
> The tests however reveal that DirectSQL and JDO paths are not returning the
> same results leading to an exception when VerifyingObjectStore is used.
> According to the error below DirectSQL is able to recover one partition from
> HMS (expected) while JDO/ORM returns empty (not expected).
> {noformat}
> 2023-10-06T03:51:19,406 ERROR [80252df4-3fdc-4971-badf-ad67ce8567c7 main]
> metastore.VerifyingObjectStore: Lists are not the same size: SQL 1, ORM 0
> 2023-10-06T03:51:19,409 ERROR [80252df4-3fdc-4971-badf-ad67ce8567c7 main]
> metastore.RetryingHMSHandler: MetaException(message:Lists are not the same
> size: SQL 1, ORM 0)
> at
> org.apache.hadoop.hive.metastore.VerifyingObjectStore.verifyLists(VerifyingObjectStore.java:148)
> at
> org.apache.hadoop.hive.metastore.VerifyingObjectStore.getPartitionsByExpr(VerifyingObjectStore.java:88)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy57.getPartitionsByExpr(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HMSHandler.get_partitions_spec_by_expr(HMSHandler.java:7330)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:98)
> at
> org.apache.hadoop.hive.metastore.AbstractHMSHandlerProxy.invoke(AbstractHMSHandlerProxy.java:82)
> at com.sun.proxy.$Proxy59.get_partitions_spec_by_expr(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionsSpecByExprInternal(HiveMetaStoreClient.java:2472)
> at
> org.apache.hadoop.hive.ql.metadata.HiveMetaStoreClientWithLocalCache.getPartitionsSpecByExprInternal(HiveMetaStoreClientWithLocalCache.java:396)
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getPartitionsSpecByExprInternal(SessionHiveMetaStoreClient.java:2279)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitionsSpecByExpr(HiveMetaStoreClient.java:2484)
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.listPartitionsSpecByExpr(SessionHiveMetaStoreClient.java:1346)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
[jira] [Created] (HIVE-27966) Disable flaky testFetchResultsOfLogWithOrientation
Wechar created HIVE-27966:
-
Summary: Disable flaky testFetchResultsOfLogWithOrientation
Key: HIVE-27966
URL: https://issues.apache.org/jira/browse/HIVE-27966
Project: Hive
Issue Type: Bug
Components: Hive
Reporter: Wechar
Assignee: Wechar
{{org.apache.hive.service.cli.operation.TestOperationLoggingAPIWithMr#testFetchResultsOfLogWithOrientation}}
test is flaky and can easily result into CI failed.
Here are some affected pipelines:
http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4959/3/tests
http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4755/15/tests
http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4966/1/tests
http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4517/25/tests
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27386) Avoid duplicate audit log in cleanupHandlerContext
[
https://issues.apache.org/jira/browse/HIVE-27386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar resolved HIVE-27386.
---
Fix Version/s: 4.0.0-beta-1
Resolution: Fixed
> Avoid duplicate audit log in cleanupHandlerContext
> --
>
> Key: HIVE-27386
> URL: https://issues.apache.org/jira/browse/HIVE-27386
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-beta-1
>
> Attachments: screenshot-1.png
>
>
> {{HMSHandler#cleanupHandlerContext}} will be called twice if
> {{IMetaStoreClient#close}} was called. In this case, where will create two
> same audit log: Done cleaning up thread local RawStore.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27958) Refactor DirectSqlUpdatePart class
Wechar created HIVE-27958:
-
Summary: Refactor DirectSqlUpdatePart class
Key: HIVE-27958
URL: https://issues.apache.org/jira/browse/HIVE-27958
Project: Hive
Issue Type: Improvement
Components: Hive
Reporter: Wechar
Assignee: Wechar
Followed HIVE-27530 , refactor the {{{}DirectSqlUpdatePart.java{}}}:
# Move lock related operations into a common class, details in
[comment|https://github.com/apache/hive/pull/4517#discussion_r1410553559].
# Use try-for-resources instead of finally close, details in
[comment|https://github.com/apache/hive/pull/4517#discussion_r1410559604].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27914) Fix the missing partitions judgement in drop_partitions_req
Wechar created HIVE-27914:
-
Summary: Fix the missing partitions judgement in
drop_partitions_req
Key: HIVE-27914
URL: https://issues.apache.org/jira/browse/HIVE-27914
Project: Hive
Issue Type: Bug
Reporter: Wechar
Assignee: Wechar
When call {{drop_partitions_req}} by partition expressions, HMS determine if
there are missing partitions by the fetched partitions number, which is not
correct.
For example, assuming that we have a table {{tbl}} with partition names:
{code:bash}
dt=20231129/hr=10
dt=20231129/hr=11
{code}
Then we try to drop partitions by sql:
{code:sql}
alter table tbl drop partition (dt='20231129'), partition (dt='20231130');
{code}
It should throw `NoSuchObjectException` because filter {{dt='20231130'}} can
not match any partitions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27913) Wrap NoSuchObjectException into HiveException in Hive.dropDatabase and Hive.dropConnector
Wechar created HIVE-27913: - Summary: Wrap NoSuchObjectException into HiveException in Hive.dropDatabase and Hive.dropConnector Key: HIVE-27913 URL: https://issues.apache.org/jira/browse/HIVE-27913 Project: Hive Issue Type: Improvement Components: Hive Reporter: Wechar Assignee: Wechar Align the drop_database and drop_dataconnector behavior with drop_table, which wrap NoSuchObjectException into HiveException when the object does not exist and input ifNotExists is false. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27857) Do not check write permission while dropping external table or partition
Wechar created HIVE-27857:
-
Summary: Do not check write permission while dropping external
table or partition
Key: HIVE-27857
URL: https://issues.apache.org/jira/browse/HIVE-27857
Project: Hive
Issue Type: Bug
Components: Hive
Reporter: Wechar
Assignee: Wechar
When drop table for external table, we actually not need delete the table path.
But now HMS will check write permission if
{{AUTHORIZATION_STORAGE_AUTH_CHECKS}} enabled no matter the table is external
or not.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27827) Improve performance of direct SQL implement for getPartitionsByFilter
Wechar created HIVE-27827: - Summary: Improve performance of direct SQL implement for getPartitionsByFilter Key: HIVE-27827 URL: https://issues.apache.org/jira/browse/HIVE-27827 Project: Hive Issue Type: Improvement Reporter: Wechar Assignee: Wechar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27762) ObjectStore GetHelper do not need run jdo query if direct sql exception is unrecoverable
Wechar created HIVE-27762:
-
Summary: ObjectStore GetHelper do not need run jdo query if direct
sql exception is unrecoverable
Key: HIVE-27762
URL: https://issues.apache.org/jira/browse/HIVE-27762
Project: Hive
Issue Type: Improvement
Components: Hive
Reporter: Wechar
Assignee: Wechar
Currently {{GetHelper}} will call {{getJdoResult()}} if {{getSqlResult()}}
throws exception, it can be avoid if the exception is unrecoverable to improve
performance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27725) Remove redundant columns in TAB_COL_STATS and PART_COL_STATS
Wechar created HIVE-27725:
-
Summary: Remove redundant columns in TAB_COL_STATS and
PART_COL_STATS
Key: HIVE-27725
URL: https://issues.apache.org/jira/browse/HIVE-27725
Project: Hive
Issue Type: Improvement
Components: Hive
Affects Versions: 4.0.0-beta-1
Reporter: Wechar
Assignee: Wechar
{{TAB_COL_STATS}} table includes {{CAT_NAME}}, {{DB_NAME}} and {{TABLE_NAME}},
which can be fetched by join {{TBLS}} and {{DBS}} tables on {{TBL_ID}} and
{{DB_ID}} columns.
{{PART_COL_STATS}} table includes {{CAT_NAME}}, {{DB_NAME}}, {{TABLE_NAME}} and
{{PARTITION_NAME}}, which can be fetched by join {{PARTITIONS}}, {{TBLS}} and
{{DBS}} tables on {{PART_ID}}, {{TBL_ID}} and {{DB_ID}}.
In addition, current HMS get table statistics without join other table, while
delete table statistics with join {{TBLS}}. This inconsistency will result
exception if in a corner case where some table column statistics were left when
drop table, then the user recreate the table with same name and database name
but will get another {{TBL_ID}}, in this case user will get the old table
column statistics incorrectly. And if user try delete stats fetched by get api,
the {{NoSuchObjectException}} will be thrown.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27676) Reuse the add_partitions logic for add_partition in ObjectStore
Wechar created HIVE-27676:
-
Summary: Reuse the add_partitions logic for add_partition in
ObjectStore
Key: HIVE-27676
URL: https://issues.apache.org/jira/browse/HIVE-27676
Project: Hive
Issue Type: Improvement
Components: Hive
Affects Versions: 4.0.0-beta-1
Reporter: Wechar
Assignee: Wechar
HIVE-26035 implements direct SQL for {{add_partitions}} to improve performance,
we can also reuse this logic for {{{}add_partition{}}} with following benefits:
* Get the performance improvement in direct SQL
* Code cleaner, reduce the duplicate code.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27530) Implement direct SQL for alter partitions to improve performance
Wechar created HIVE-27530: - Summary: Implement direct SQL for alter partitions to improve performance Key: HIVE-27530 URL: https://issues.apache.org/jira/browse/HIVE-27530 Project: Hive Issue Type: Improvement Reporter: Wechar Assignee: Wechar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27500) Normalize add_partitions related api in HMS
[
https://issues.apache.org/jira/browse/HIVE-27500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-27500:
--
Summary: Normalize add_partitions related api in HMS (was: Optimize
add_partitions related api in HMS)
> Normalize add_partitions related api in HMS
> ---
>
> Key: HIVE-27500
> URL: https://issues.apache.org/jira/browse/HIVE-27500
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
>
> {{HMSHandler}} has three APIs for add partitions: {{add_partitions_req}},
> {{add_partitions}} and {{add_partitions_pspec}}. Currently both
> {{add_partitions_req}} and {{add_partitions}} APIs invoke
> {{add_partitions_core}} function which was improved by direct sql in
> HIVE-26035.
> We can also invoke {{add_partitions_core}} function within
> {{add_partitions_pspec}} API for performance improvement.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27500) Optimize add_partitions related api in HMS
Wechar created HIVE-27500:
-
Summary: Optimize add_partitions related api in HMS
Key: HIVE-27500
URL: https://issues.apache.org/jira/browse/HIVE-27500
Project: Hive
Issue Type: Improvement
Components: Hive
Reporter: Wechar
Assignee: Wechar
{{HMSHandler}} has three APIs for add partitions: {{add_partitions_req}},
{{add_partitions}} and {{add_partitions_pspec}}. Currently both
{{add_partitions_req}} and {{add_partitions}} APIs invoke
{{add_partitions_core}} function which was improved by direct sql in HIVE-26035.
We can also invoke {{add_partitions_core}} function within
{{add_partitions_pspec}} API for performance improvement.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27469) HMSHandler lost root cause of MetaStorePreEventListener
[
https://issues.apache.org/jira/browse/HIVE-27469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-27469:
--
Attachment: screenshot-1.png
> HMSHandler lost root cause of MetaStorePreEventListener
> ---
>
> Key: HIVE-27469
> URL: https://issues.apache.org/jira/browse/HIVE-27469
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Attachments: screenshot-1.png
>
>
> Currently {{HMSHandler}} will convert {{NoSuchObjectException}} and
> {{InvalidOperationException}} to {{MetaException}}, but it will lose the root
> cause. The related code as follows:
> {code:java}
> // HMSHandler.java#L3956
> private void firePreEvent(PreEventContext event) throws MetaException {
> for (MetaStorePreEventListener listener : preListeners) {
> try {
> listener.onEvent(event);
> } catch (NoSuchObjectException e) {
> throw new MetaException(e.getMessage());
> } catch (InvalidOperationException e) {
> throw new MetaException(e.getMessage());
> }
> }
> }
> {code}
> In this patch, we want to add the root cause and it can help troubleshooting
> through HMS log.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27469) HMSHandler lost root cause of MetaStorePreEventListener
Wechar created HIVE-27469:
-
Summary: HMSHandler lost root cause of MetaStorePreEventListener
Key: HIVE-27469
URL: https://issues.apache.org/jira/browse/HIVE-27469
Project: Hive
Issue Type: Improvement
Components: Hive
Reporter: Wechar
Assignee: Wechar
Currently {{HMSHandler}} will convert {{NoSuchObjectException}} and
{{InvalidOperationException}} to {{MetaException}}, but it will lose the root
cause. The related code as follows:
{code:java}
// HMSHandler.java#L3956
private void firePreEvent(PreEventContext event) throws MetaException {
for (MetaStorePreEventListener listener : preListeners) {
try {
listener.onEvent(event);
} catch (NoSuchObjectException e) {
throw new MetaException(e.getMessage());
} catch (InvalidOperationException e) {
throw new MetaException(e.getMessage());
}
}
}
{code}
In this patch, we want to add the root cause and it can help troubleshooting
through HMS log.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27461) HiveMetaStoreAuthorizer lost the root cause of checkPrivileges
Wechar created HIVE-27461:
-
Summary: HiveMetaStoreAuthorizer lost the root cause of
checkPrivileges
Key: HIVE-27461
URL: https://issues.apache.org/jira/browse/HIVE-27461
Project: Hive
Issue Type: Improvement
Components: Hive
Reporter: Wechar
Assignee: Wechar
We discovered that the {{HiveMetaStoreAuthorizer}} fails to retain the
underlying cause of {{checkPrivileges}}, making it difficult for us to
troubleshoot authorization failures effectively.
{code:bash}
2023-06-21T18:45:03,695 ERROR [pool-10-thread-1]:
metastore.HiveMetaStoreAuthorizer (HiveMetaStoreAuthorizer.java:onEvent(111)) -
HiveMeta
StoreAuthorizer.onEvent(): failed
org.apache.hadoop.hive.metastore.api.MetaException: Permission denied:
Principal [name=weiqiang.yu, type=USER] does not have following pri
vileges for operation QUERY [[SELECT] on Object [type=DATABASE, name=default]]
at
org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.checkPrivileges(HiveMetaStoreAuthoriz
er.java:232) ~[hive-exec-3.1.2-sdi-022.jar:3.1.2-sdi-022]
at
org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.onEvent(HiveMetaStoreAuthorizer.java:
108) [hive-exec-3.1.2-sdi-022.jar:3.1.2-sdi-022]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.firePreEvent(HiveMetaStore.java:3469)
[hive-exec-3.1.2-sdi-022.jar:3.
1.2-sdi-022]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database(HiveMetaStore.java:1463)
[hive-exec-3.1.2-sdi-022.jar:3.
1.2-sdi-022]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_252]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_252]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_252]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:158)
[hive-exec-3.1.2-sdi-022.jar:3.
1.2-sdi-022]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:119)
[hive-exec-3.1.2-sdi-022.jar:3.1.2-sdi-
022]
at com.sun.proxy.$Proxy29.get_database(Unknown Source) [?:?]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_d
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27460) HiveMetastoreClient should not wrap TTransportException in MetaException
Wechar created HIVE-27460:
-
Summary: HiveMetastoreClient should not wrap TTransportException
in MetaException
Key: HIVE-27460
URL: https://issues.apache.org/jira/browse/HIVE-27460
Project: Hive
Issue Type: Improvement
Components: Hive
Reporter: Wechar
Assignee: Wechar
When instantiate hive metastore client, we will wrap the
{{TTransportException}} within {{{}MetaException{}}}, and will try to connect
to another metastore thrift uri if encountered {{{}MetaException{}}}.
As discussed in HIVE-27097, client side only concerns about
{{TTransportException}}, which should also apply to the connection building
stage. So we can distinguish {{TTransportException}} from {{MetaException}} and
stop connect immediately when encountered {{MetaException}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27417) Remove Filter.g and unnecessary antlr3 dependency
Wechar created HIVE-27417: - Summary: Remove Filter.g and unnecessary antlr3 dependency Key: HIVE-27417 URL: https://issues.apache.org/jira/browse/HIVE-27417 Project: Hive Issue Type: Sub-task Components: Hive Reporter: Wechar Assignee: Wechar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27386) Avoid duplicate audit log in cleanupHandlerContext
[
https://issues.apache.org/jira/browse/HIVE-27386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-27386:
--
Summary: Avoid duplicate audit log in cleanupHandlerContext (was: Remove
duplicate audit log in cleanupHandlerContext)
> Avoid duplicate audit log in cleanupHandlerContext
> --
>
> Key: HIVE-27386
> URL: https://issues.apache.org/jira/browse/HIVE-27386
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Attachments: screenshot-1.png
>
>
> {{HMSHandler#cleanupHandlerContext}} will be called twice if
> {{IMetaStoreClient#close}} was called. In this case, where will create two
> same audit log: Done cleaning up thread local RawStore.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27386) Remove duplicate audit log in cleanupHandlerContext
[
https://issues.apache.org/jira/browse/HIVE-27386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-27386:
--
Attachment: screenshot-1.png
> Remove duplicate audit log in cleanupHandlerContext
> ---
>
> Key: HIVE-27386
> URL: https://issues.apache.org/jira/browse/HIVE-27386
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Attachments: screenshot-1.png
>
>
> {{HMSHandler#cleanupHandlerContext}} will be called twice if
> {{IMetaStoreClient#close}} was called. In this case, where will create two
> same audit log: Done cleaning up thread local RawStore.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27386) Remove duplicate audit log in cleanupHandlerContext
Wechar created HIVE-27386:
-
Summary: Remove duplicate audit log in cleanupHandlerContext
Key: HIVE-27386
URL: https://issues.apache.org/jira/browse/HIVE-27386
Project: Hive
Issue Type: Improvement
Components: Hive
Reporter: Wechar
Assignee: Wechar
{{HMSHandler#cleanupHandlerContext}} will be called twice if
{{IMetaStoreClient#close}} was called. In this case, where will create two same
audit log: Done cleaning up thread local RawStore.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27362) Migrate Filter.g to Antlr4
Wechar created HIVE-27362: - Summary: Migrate Filter.g to Antlr4 Key: HIVE-27362 URL: https://issues.apache.org/jira/browse/HIVE-27362 Project: Hive Issue Type: Sub-task Components: Hive Reporter: Wechar Assignee: Wechar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27284) Make HMSHandler proxy pluggable
Wechar created HIVE-27284:
-
Summary: Make HMSHandler proxy pluggable
Key: HIVE-27284
URL: https://issues.apache.org/jira/browse/HIVE-27284
Project: Hive
Issue Type: Improvement
Components: Hive
Affects Versions: 4.0.0-alpha-2
Reporter: Wechar
Assignee: Wechar
Currently HMS use the only proxy implementation of HMSHandler i.e
{{RetryingHMSHandler}}, resulting in some code hacks in {{HMSHandler}}. For
example when test HMS timeout, we add additional static fields
{{testTimeoutEnabled}} and {{testTimeoutValue}}, and add sleep code in
{{create_database}} method, which is not elegant and flexible.
So we introduce a new conf {{metastore.hmshandler.proxy}} to configure proxy
class for HMSHandler, it will be more convenient to extend the new proxy of
HMSHandler and can separate test code from HMSHandler.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27266) Retrieve only partNames if not need drop data in HMSHandler.dropPartitionsAndGetLocations
[
https://issues.apache.org/jira/browse/HIVE-27266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-27266:
--
Description:
Followed HIVE-19783, we only need partNames instead of partName and location
pairs if we do not need check location.
We add a new benchmark *dropTableMetadataWithPartitions* to delete only
metadata rather than the real table data.
Test results like:
* Before the patch:
{code:bash}
Operation Mean Med Min Max Err%
dropTableMetaOnlyWithPartitions.10 23.7021.8719.3631.7314.48
dropTableMetaOnlyWithPartitions.100 54.4254.1545.9276.688.891
dropTableMetaOnlyWithPartitions.1000 462.5456.1321.0654.315.96
{code}
* After the patch:
{code:bash}
Operation Mean Med Min Max Err%
dropTableMetaOnlyWithPartitions.10 21.4921.2419.3027.906.661
dropTableMetaOnlyWithPartitions.100 51.5148.3044.8685.2316.91
dropTableMetaOnlyWithPartitions.1000 415.4407.2308.8595.214.28
{code}
was:Followed HIVE-19783, we only need partNames instead of partName and
location pairs if we do not need check location.
> Retrieve only partNames if not need drop data in
> HMSHandler.dropPartitionsAndGetLocations
> -
>
> Key: HIVE-27266
> URL: https://issues.apache.org/jira/browse/HIVE-27266
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0-alpha-2
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
>
> Followed HIVE-19783, we only need partNames instead of partName and location
> pairs if we do not need check location.
> We add a new benchmark *dropTableMetadataWithPartitions* to delete only
> metadata rather than the real table data.
> Test results like:
> * Before the patch:
> {code:bash}
> Operation Mean Med Min Max Err%
> dropTableMetaOnlyWithPartitions.10 23.7021.8719.3631.7314.48
> dropTableMetaOnlyWithPartitions.100 54.4254.1545.9276.688.891
> dropTableMetaOnlyWithPartitions.1000 462.5456.1321.0654.315.96
> {code}
> * After the patch:
> {code:bash}
> Operation Mean Med Min Max Err%
> dropTableMetaOnlyWithPartitions.10 21.4921.2419.3027.906.661
> dropTableMetaOnlyWithPartitions.100 51.5148.3044.8685.2316.91
> dropTableMetaOnlyWithPartitions.1000 415.4407.2308.8595.214.28
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-27266) Retrieve only partNames if not need drop data in HMSHandler.dropPartitionsAndGetLocations
Wechar created HIVE-27266: - Summary: Retrieve only partNames if not need drop data in HMSHandler.dropPartitionsAndGetLocations Key: HIVE-27266 URL: https://issues.apache.org/jira/browse/HIVE-27266 Project: Hive Issue Type: Improvement Components: Hive Affects Versions: 4.0.0-alpha-2 Reporter: Wechar Assignee: Wechar Followed HIVE-19783, we only need partNames instead of partName and location pairs if we do not need check location. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27172) Add the HMS client connection timeout config
[
https://issues.apache.org/jira/browse/HIVE-27172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-27172:
--
Issue Type: Task (was: New Feature)
> Add the HMS client connection timeout config
>
>
> Key: HIVE-27172
> URL: https://issues.apache.org/jira/browse/HIVE-27172
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
>
> Currently {{HiveMetastoreClient}} use {{CLIENT_SOCKET_TIMEOUT}} as both
> socket timeout and connection timeout, it's not convenient for users to set a
> smaller connection timeout.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27172) Add the HMS client connection timeout config
[
https://issues.apache.org/jira/browse/HIVE-27172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar reassigned HIVE-27172:
-
> Add the HMS client connection timeout config
>
>
> Key: HIVE-27172
> URL: https://issues.apache.org/jira/browse/HIVE-27172
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
>
> Currently {{HiveMetastoreClient}} use {{CLIENT_SOCKET_TIMEOUT}} as both
> socket timeout and connection timeout, it's not convenient for users to set a
> smaller connection timeout.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27150) Drop single partition can also support direct sql
[ https://issues.apache.org/jira/browse/HIVE-27150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar reassigned HIVE-27150: - > Drop single partition can also support direct sql > - > > Key: HIVE-27150 > URL: https://issues.apache.org/jira/browse/HIVE-27150 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Wechar >Assignee: Wechar >Priority: Major > > *Background:* > [HIVE-6980|https://issues.apache.org/jira/browse/HIVE-6980] supports direct > sql for drop_partitions, we can reuse this huge improvement in drop_partition. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27097) Improve the retry strategy for Metastore client and server
[
https://issues.apache.org/jira/browse/HIVE-27097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-27097:
--
Description:
*Background*
Hive provides *{{RetryingMetaStoreClient}}* and *{{RetryingHMSHandler}}* to do
retry when thrift request failed:
* RetryingMetaStoreClient will retry for *thrift related exception* and some
*MetaException*
* RetryingHMSHandler will retry for all {*}JDOException{*} or
*NucleusException*.
*Motivation*
Current retry mechanism will lead to many unnecessary retries in both client
and server. To simplify the process, we introduce following retry mechanism:
* Client side only concerns the error of communication, i.e.,
{*}TTransportException{*}.
* Server side can skip some exceptions which always turn to fail even with
retry, like {*}SQLIntegrityConstraintViolationException{*}.
was:
*Background*
Hive provides *{{RetryingMetaStoreClient}}* and *{{RetryingHMSHandler}}* to do
retry when thrift request failed:
* RetryingMetaStoreClient will retry for *thrift related exception* and some
*MetaException*
* RetryingHMSHandler will retry for all *JDOException*.
*Motivation*
Current retry mechanism will lead to many unnecessary retries in both client
and server. To simplify the process, we introduce following retry mechanism:
* Client side only concerns the error of communication, i.e.,
*TTransportException*.
* Server side can skip some exceptions which always turn to fail even with
retry, like *SQLIntegrityConstraintViolationException*.
> Improve the retry strategy for Metastore client and server
> --
>
> Key: HIVE-27097
> URL: https://issues.apache.org/jira/browse/HIVE-27097
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0-alpha-2
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
>
> *Background*
> Hive provides *{{RetryingMetaStoreClient}}* and *{{RetryingHMSHandler}}* to
> do retry when thrift request failed:
> * RetryingMetaStoreClient will retry for *thrift related exception* and some
> *MetaException*
> * RetryingHMSHandler will retry for all {*}JDOException{*} or
> *NucleusException*.
> *Motivation*
> Current retry mechanism will lead to many unnecessary retries in both client
> and server. To simplify the process, we introduce following retry mechanism:
> * Client side only concerns the error of communication, i.e.,
> {*}TTransportException{*}.
> * Server side can skip some exceptions which always turn to fail even with
> retry, like {*}SQLIntegrityConstraintViolationException{*}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27097) Improve the retry strategy for Metastore client and server
[
https://issues.apache.org/jira/browse/HIVE-27097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar reassigned HIVE-27097:
-
> Improve the retry strategy for Metastore client and server
> --
>
> Key: HIVE-27097
> URL: https://issues.apache.org/jira/browse/HIVE-27097
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0-alpha-2
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
>
> *Background*
> Hive provides *{{RetryingMetaStoreClient}}* and *{{RetryingHMSHandler}}* to
> do retry when thrift request failed:
> * RetryingMetaStoreClient will retry for *thrift related exception* and some
> *MetaException*
> * RetryingHMSHandler will retry for all *JDOException*.
> *Motivation*
> Current retry mechanism will lead to many unnecessary retries in both client
> and server. To simplify the process, we introduce following retry mechanism:
> * Client side only concerns the error of communication, i.e.,
> *TTransportException*.
> * Server side can skip some exceptions which always turn to fail even with
> retry, like *SQLIntegrityConstraintViolationException*.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27093) Fix NPE in initialize() of Partition class
[ https://issues.apache.org/jira/browse/HIVE-27093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar reassigned HIVE-27093: - > Fix NPE in initialize() of Partition class > -- > > Key: HIVE-27093 > URL: https://issues.apache.org/jira/browse/HIVE-27093 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0-alpha-2 >Reporter: Wechar >Assignee: Wechar >Priority: Major > Attachments: image-2023-02-19-02-00-43-537.png > > > We will get a *NullPointerException* when get_partitions from HMS if the > partition SD does not exists. > !image-2023-02-19-02-00-43-537.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26935) Expose root cause of MetaException in RetryingHMSHandler
[
https://issues.apache.org/jira/browse/HIVE-26935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683633#comment-17683633
]
Wechar commented on HIVE-26935:
---
Thanks for your review and suggestions [~zabetak]!
> Expose root cause of MetaException in RetryingHMSHandler
>
>
> Key: HIVE-26935
> URL: https://issues.apache.org/jira/browse/HIVE-26935
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0-alpha-2
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> MetaException is generated by thrift, and only {{message}} filed will be
> transport to client, we should expose the root cause in message to the
> clients with following advantages:
> * More friendly for user troubleshooting
> * Some root cause is unrecoverable, exposing it can disable the unnecessary
> retry.
> *How to Reproduce:*
> - Step 1: Disable direct sql for HMS for our test case.
> - Step 2: Add an illegal {{PART_COL_STATS}} for a partition,
> - Step 3: Try to {{drop table}} with Spark.
> The exception in Hive metastore is:
> {code:sh}
> 2023-01-11T17:13:51,259 ERROR [Metastore-Handler-Pool: Thread-39]:
> metastore.ObjectStore (ObjectStore.java:run(4369)) -
> javax.jdo.JDOUserException: One or more instances could not be deleted
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:625)
> ~[datanucleus-api-jdo-5.2.8.jar:?]
> at
> org.datanucleus.api.jdo.JDOQuery.deletePersistentInternal(JDOQuery.java:530)
> ~[datanucleus-api-jdo-5.2.8.jar:?]
> at
> org.datanucleus.api.jdo.JDOQuery.deletePersistentAll(JDOQuery.java:499)
> ~[datanucleus-api-jdo-5.2.8.jar:?]
> at
> org.apache.hadoop.hive.metastore.QueryWrapper.deletePersistentAll(QueryWrapper.java:108)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionsNoTxn(ObjectStore.java:4207)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$1000(ObjectStore.java:285)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$7.run(ObjectStore.java:3086)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:74)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionsViaJdo(ObjectStore.java:3074)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$400(ObjectStore.java:285)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$6.getJdoResult(ObjectStore.java:3058)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$6.getJdoResult(ObjectStore.java:3050)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:4362)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionsInternal(ObjectStore.java:3061)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.dropPartitions(ObjectStore.java:3040)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_332]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_332]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_332]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_332]
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy24.dropPartitions(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.HMSHandler.dropPartitionsAndGetLocations(HMSHandler.java:3186)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HMSHandler.drop_table_core(HMSHandler.java:2963)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HMSHandler.drop_table_with_environment_context(HMSHandler.java:3211)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Assigned] (HIVE-26935) Expose root cause of MetaException to client sides
[
https://issues.apache.org/jira/browse/HIVE-26935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar reassigned HIVE-26935:
-
> Expose root cause of MetaException to client sides
> --
>
> Key: HIVE-26935
> URL: https://issues.apache.org/jira/browse/HIVE-26935
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0-alpha-2
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
>
> MetaException is generated by thrift, and only {{message}} filed will be
> transport to client, we should expose the root cause in message to the
> clients with following advantages:
> * More friendly for user troubleshooting
> * Some root cause is unrecoverable, exposing it can disable the unnecessary
> retry.
> *How to Reproduce:*
> - Step 1: Disable direct sql for HMS for our test case.
> - Step 2: Add an illegal {{PART_COL_STATS}} for a partition,
> - Step 3: Try to {{drop table}} with Spark.
> The exception in Hive metastore is:
> {code:sh}
> 2023-01-11T17:13:51,259 ERROR [Metastore-Handler-Pool: Thread-39]:
> metastore.ObjectStore (ObjectStore.java:run(4369)) -
> javax.jdo.JDOUserException: One or more instances could not be deleted
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:625)
> ~[datanucleus-api-jdo-5.2.8.jar:?]
> at
> org.datanucleus.api.jdo.JDOQuery.deletePersistentInternal(JDOQuery.java:530)
> ~[datanucleus-api-jdo-5.2.8.jar:?]
> at
> org.datanucleus.api.jdo.JDOQuery.deletePersistentAll(JDOQuery.java:499)
> ~[datanucleus-api-jdo-5.2.8.jar:?]
> at
> org.apache.hadoop.hive.metastore.QueryWrapper.deletePersistentAll(QueryWrapper.java:108)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionsNoTxn(ObjectStore.java:4207)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$1000(ObjectStore.java:285)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$7.run(ObjectStore.java:3086)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:74)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionsViaJdo(ObjectStore.java:3074)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$400(ObjectStore.java:285)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$6.getJdoResult(ObjectStore.java:3058)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$6.getJdoResult(ObjectStore.java:3050)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:4362)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionsInternal(ObjectStore.java:3061)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.dropPartitions(ObjectStore.java:3040)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_332]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_332]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_332]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_332]
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy24.dropPartitions(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.HMSHandler.dropPartitionsAndGetLocations(HMSHandler.java:3186)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HMSHandler.drop_table_core(HMSHandler.java:2963)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HMSHandler.drop_table_with_environment_context(HMSHandler.java:3211)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HMSHandler.drop_table_with_environment_context(HMSHandler.java:3199)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_332]
> at
>
[jira] [Assigned] (HIVE-26661) Support partition filter for char and varchar types on Hive metastore
[ https://issues.apache.org/jira/browse/HIVE-26661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wechar reassigned HIVE-26661: - > Support partition filter for char and varchar types on Hive metastore > - > > Key: HIVE-26661 > URL: https://issues.apache.org/jira/browse/HIVE-26661 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0-alpha-1 >Reporter: Wechar >Assignee: Wechar >Priority: Minor > > HMS stores partition value as string in backend database, so it supports > "equal filter" for string (include date type) and integer types, and other > filter (for example greater than > ) for only string type. > The char and varchar types can also considered as string, and we can support > the partition filter for them on HMS. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26401) Refine the log of add_partitions if the partition already exists
[
https://issues.apache.org/jira/browse/HIVE-26401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar reassigned HIVE-26401:
-
> Refine the log of add_partitions if the partition already exists
>
>
> Key: HIVE-26401
> URL: https://issues.apache.org/jira/browse/HIVE-26401
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0-alpha-1
>Reporter: Wechar
>Assignee: Wechar
>Priority: Minor
> Fix For: 4.0.0-alpha-2
>
>
> Currently {{*add_partitions_xxx*}} will log the complete information of a
> partition if it already exists, see in
> [HMSHandler.java#L4320|https://github.com/apache/hive/blob/e3751ab545370f9b252d0b4a07bc315037541a95/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java#L4320]:
> {code:java}
> if (!shouldAdd) {
> LOG.info("Not adding partition {} as it already exists", part);
> return false;
> }
> {code}
> It will print a long message including the columns of this partition, we
> think it is unnecessary based on the following two points:
> {color:red}1. The long message is redundant.{color}
> We can get enough information from just
> *cat_name.db_name.tbl_name[part_col1=part_val1/part_col2=part_val2...]*
> {color:red}2. The long message is not friendly to save and query.{color}
> This log message will take up a large log space especially when the user need
> to execute *MSCK REPAIR TABLE* operation regularly because the old partition
> must be already existed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26386) Exceptions thrown in ObjectStore should contain all causes
[
https://issues.apache.org/jira/browse/HIVE-26386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-26386:
--
Description:
{{*RetryingHMSHandler*}} will retry the failed operations if the exception is
caused by {{*JDOException*}} or {{{}*NucleusException*{}}}, the logic is in
[RetryingHMSHandler.java#L185:|https://github.com/apache/hive/blob/723d52270488b8dbc108db1a9925d1c569415039/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RetryingHMSHandler.java#L185]
{code:java}
} else if (e.getCause() instanceof MetaException && e.getCause().getCause() !=
null) {
if (e.getCause().getCause() instanceof javax.jdo.JDOException ||
e.getCause().getCause() instanceof NucleusException) {
// The JDOException or the Nucleus Exception may be wrapped further
in a MetaException
caughtException = e.getCause().getCause();
{code}
But in {{{}*ObjectStore.java*{}}}, we found some exceptions thrown as
{{{}*MetaException*{}}} are not wrapped by the causes, which cause some
operations failed by {{JDOException}} can not do retry.
was:
{{*RetryingHMSHandler*}} will retry the failed operations if the exception is
caused by {{*JDOException*}} or {{{}*NucleusException*{}}}, the logic is in
[RetryingHMSHandler.java#L185:|https://github.com/apache/hive/blob/723d52270488b8dbc108db1a9925d1c569415039/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RetryingHMSHandler.java#L185]
{code:java}
} else if (e.getCause() instanceof MetaException && e.getCause().getCause() !=
null) {
if (e.getCause().getCause() instanceof javax.jdo.JDOException ||
e.getCause().getCause() instanceof NucleusException) {
// The JDOException or the Nucleus Exception may be wrapped further
in a MetaException
caughtException = e.getCause().getCause();
{code}
But in {{{}*ObjectStore.java*{}}}, we found some exceptions thrown as
{{{}*MetaException*{} are not wrapped by the causes, which cause some
operations failed by {{JDOException}} can not do retry.
> Exceptions thrown in ObjectStore should contain all causes
> --
>
> Key: HIVE-26386
> URL: https://issues.apache.org/jira/browse/HIVE-26386
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0-alpha-1
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Fix For: 4.0.0-alpha-2
>
>
> {{*RetryingHMSHandler*}} will retry the failed operations if the exception is
> caused by {{*JDOException*}} or {{{}*NucleusException*{}}}, the logic is in
> [RetryingHMSHandler.java#L185:|https://github.com/apache/hive/blob/723d52270488b8dbc108db1a9925d1c569415039/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RetryingHMSHandler.java#L185]
> {code:java}
> } else if (e.getCause() instanceof MetaException && e.getCause().getCause()
> != null) {
> if (e.getCause().getCause() instanceof javax.jdo.JDOException ||
> e.getCause().getCause() instanceof NucleusException) {
> // The JDOException or the Nucleus Exception may be wrapped
> further in a MetaException
> caughtException = e.getCause().getCause();
> {code}
> But in {{{}*ObjectStore.java*{}}}, we found some exceptions thrown as
> {{{}*MetaException*{}}} are not wrapped by the causes, which cause some
> operations failed by {{JDOException}} can not do retry.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26386) Exceptions thrown in ObjectStore should contain all causes
[
https://issues.apache.org/jira/browse/HIVE-26386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar reassigned HIVE-26386:
-
> Exceptions thrown in ObjectStore should contain all causes
> --
>
> Key: HIVE-26386
> URL: https://issues.apache.org/jira/browse/HIVE-26386
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0-alpha-1
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Fix For: 4.0.0-alpha-2
>
>
> {{*RetryingHMSHandler*}} will retry the failed operations if the exception is
> caused by {{*JDOException*}} or {{{}*NucleusException*{}}}, the logic is in
> [RetryingHMSHandler.java#L185:|https://github.com/apache/hive/blob/723d52270488b8dbc108db1a9925d1c569415039/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RetryingHMSHandler.java#L185]
> {code:java}
> } else if (e.getCause() instanceof MetaException && e.getCause().getCause()
> != null) {
> if (e.getCause().getCause() instanceof javax.jdo.JDOException ||
> e.getCause().getCause() instanceof NucleusException) {
> // The JDOException or the Nucleus Exception may be wrapped
> further in a MetaException
> caughtException = e.getCause().getCause();
> {code}
> But in {{{}*ObjectStore.java*{}}}, we found some exceptions thrown as
> {{{}*MetaException*{} are not wrapped by the causes, which cause some
> operations failed by {{JDOException}} can not do retry.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26355) Column compare should be case insensitive for name
[
https://issues.apache.org/jira/browse/HIVE-26355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-26355:
--
Summary: Column compare should be case insensitive for name (was: Column
compare should be case insensitive)
> Column compare should be case insensitive for name
> --
>
> Key: HIVE-26355
> URL: https://issues.apache.org/jira/browse/HIVE-26355
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0-alpha-1
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Fix For: 4.0.0-alpha-2
>
>
> Hive stores all name related value as lower case, such as db_name, tbl_name,
> col_name etc. But the compare of {{FieldSchema}} does not ignore the case of
> name, which may cause incorrect result of compare.
> *Bug Description:*
> Some computing engines are case sensitive for column name. For example, Spark
> will add a table property to save the column fields when creating a table,
> and will replace column fields with this property when fetching table fields.
> When calling {{*ALTER TABLE ... ADD COLUMNS*}}, the compare of fields between
> old table and new table will be not expected, and the ADD COLUMNS operation
> will be cascaded to PARTITIONS, which is unnecessary and time consuming if
> the table has many partitions.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Assigned] (HIVE-26355) Column compare should be case insensitive
[
https://issues.apache.org/jira/browse/HIVE-26355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar reassigned HIVE-26355:
-
> Column compare should be case insensitive
> -
>
> Key: HIVE-26355
> URL: https://issues.apache.org/jira/browse/HIVE-26355
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0-alpha-1
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Fix For: 4.0.0-alpha-2
>
>
> Hive stores all name related value as lower case, such as db_name, tbl_name,
> col_name etc. But the compare of {{FieldSchema}} does not ignore the case of
> name, which may cause incorrect result of compare.
> *Bug Description:*
> Some computing engines are case sensitive for column name. For example, Spark
> will add a table property to save the column fields when creating a table,
> and will replace column fields with this property when fetching table fields.
> When calling {{*ALTER TABLE ... ADD COLUMNS*}}, the compare of fields between
> old table and new table will be not expected, and the ADD COLUMNS operation
> will be cascaded to PARTITIONS, which is unnecessary and time consuming if
> the table has many partitions.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (HIVE-26314) Support alter function in Hive DDL
[
https://issues.apache.org/jira/browse/HIVE-26314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-26314:
--
Description:
Hive SQL does not support {{*ALTER FUNCTION*}} yet, we can refer to the
{{*CREATE [OR REPLACE] FUNCTION*}} of
[Spark|https://spark.apache.org/docs/3.1.2/sql-ref-syntax-ddl-create-function.html]
to implement the alter function .
{code:sql}
CREATE [ TEMPORARY ] FUNCTION [ OR REPLACE ] [IF NOT EXISTS ]
[db_name.]function_name AS class_name
[USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
{code}
* *OR REPLACE*
If specified, the resources for the function are reloaded. This is mainly
useful to pick up any changes made to the implementation of the function. This
parameter is mutually exclusive to {{*IF NOT EXISTS*}} and can not be specified
together.
was:
Hive SQL does not support {{*ALTER FUNCTION*}} yet, we can refer to the
{{*CREATE [OR REPLACE] FUNCTION*}} of
[Spark|https://spark.apache.org/docs/3.1.2/sql-ref-syntax-ddl-create-function.html]
to implement the alter function .
{code:sql}
CREATE [ OR REPLACE ] [ TEMPORARY ] FUNCTION [IF NOT EXISTS ]
[db_name.]function_name AS class_name
[USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
{code}
* *OR REPLACE*
If specified, the resources for the function are reloaded. This is mainly
useful to pick up any changes made to the implementation of the function. This
parameter is mutually exclusive to {{*IF NOT EXISTS*}} and can not be specified
together.
> Support alter function in Hive DDL
> --
>
> Key: HIVE-26314
> URL: https://issues.apache.org/jira/browse/HIVE-26314
> Project: Hive
> Issue Type: Task
> Components: Hive
>Affects Versions: 4.0.0-alpha-1
>Reporter: Wechar
>Assignee: Wechar
>Priority: Minor
> Fix For: 4.0.0-alpha-2
>
>
> Hive SQL does not support {{*ALTER FUNCTION*}} yet, we can refer to the
> {{*CREATE [OR REPLACE] FUNCTION*}} of
> [Spark|https://spark.apache.org/docs/3.1.2/sql-ref-syntax-ddl-create-function.html]
> to implement the alter function .
> {code:sql}
> CREATE [ TEMPORARY ] FUNCTION [ OR REPLACE ] [IF NOT EXISTS ]
> [db_name.]function_name AS class_name
> [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
> {code}
> * *OR REPLACE*
> If specified, the resources for the function are reloaded. This is mainly
> useful to pick up any changes made to the implementation of the function.
> This parameter is mutually exclusive to {{*IF NOT EXISTS*}} and can not be
> specified together.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (HIVE-26314) Support alter function in Hive DDL
[
https://issues.apache.org/jira/browse/HIVE-26314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-26314:
--
Description:
Hive SQL does not support {{*ALTER FUNCTION*}} yet, we can refer to the
{{*CREATE [OR REPLACE] FUNCTION*}} of
[Spark|https://spark.apache.org/docs/3.1.2/sql-ref-syntax-ddl-create-function.html]
to implement the alter function .
{code:sql}
CREATE [ OR REPLACE ] [ TEMPORARY ] FUNCTION [IF NOT EXISTS ]
[db_name.]function_name AS class_name
[USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
{code}
* *OR REPLACE*
If specified, the resources for the function are reloaded. This is mainly
useful to pick up any changes made to the implementation of the function. This
parameter is mutually exclusive to {{*IF NOT EXISTS*}} and can not be specified
together.
was:
Hive SQL does not support {{*ALTER FUNCTION*}} yet, we can refer to the
{{*CREATE [OR REPLACE] FUNCTION*}} of
[Spark|https://spark.apache.org/docs/3.1.2/sql-ref-syntax-ddl-create-function.html]
to implement the alter function .
{code:sql}
CREATE [ OR REPLACE ] [ TEMPORARY ] FUNCTION [IF NOT EXISTS ]
[db_name.]function_name AS class_name
[USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
{code}
* *OR REPLACE*
If specified, the resources for the function are reloaded. This is mainly
useful to pick up any changes made to the implementation of the function. This
parameter is mutually exclusive to {{IF NOT EXISTS}} and can not be specified
together.
> Support alter function in Hive DDL
> --
>
> Key: HIVE-26314
> URL: https://issues.apache.org/jira/browse/HIVE-26314
> Project: Hive
> Issue Type: Task
> Components: Hive
>Affects Versions: 4.0.0-alpha-1
>Reporter: Wechar
>Assignee: Wechar
>Priority: Minor
> Fix For: 4.0.0-alpha-2
>
>
> Hive SQL does not support {{*ALTER FUNCTION*}} yet, we can refer to the
> {{*CREATE [OR REPLACE] FUNCTION*}} of
> [Spark|https://spark.apache.org/docs/3.1.2/sql-ref-syntax-ddl-create-function.html]
> to implement the alter function .
> {code:sql}
> CREATE [ OR REPLACE ] [ TEMPORARY ] FUNCTION [IF NOT EXISTS ]
> [db_name.]function_name AS class_name
> [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
> {code}
> * *OR REPLACE*
> If specified, the resources for the function are reloaded. This is mainly
> useful to pick up any changes made to the implementation of the function.
> This parameter is mutually exclusive to {{*IF NOT EXISTS*}} and can not be
> specified together.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (HIVE-26314) Support alter function in Hive DDL
[
https://issues.apache.org/jira/browse/HIVE-26314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-26314:
--
Description:
Hive SQL does not support {{*ALTER FUNCTION*}} yet, we can refer to the
{{*CREATE [OR REPLACE] FUNCTION*}} of
[Spark|https://spark.apache.org/docs/3.1.2/sql-ref-syntax-ddl-create-function.html]
to implement the alter function .
{code:sql}
CREATE [ OR REPLACE ] [ TEMPORARY ] FUNCTION [IF NOT EXISTS ]
[db_name.]function_name AS class_name
[USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
{code}
* *OR REPLACE*
If specified, the resources for the function are reloaded. This is mainly
useful to pick up any changes made to the implementation of the function. This
parameter is mutually exclusive to {{IF NOT EXISTS}} and can not be specified
together.
was:Hive SQL does not support {{*ALTER FUNCTION*}} yet, we can refer to the
{{*CREATE [OR REPLACE] FUNCTION*}} of
[Spark|https://spark.apache.org/docs/3.1.2/sql-ref-syntax-ddl-create-function.html]
to implement the alter function .
> Support alter function in Hive DDL
> --
>
> Key: HIVE-26314
> URL: https://issues.apache.org/jira/browse/HIVE-26314
> Project: Hive
> Issue Type: Task
> Components: Hive
>Affects Versions: 4.0.0-alpha-1
>Reporter: Wechar
>Assignee: Wechar
>Priority: Minor
> Fix For: 4.0.0-alpha-2
>
>
> Hive SQL does not support {{*ALTER FUNCTION*}} yet, we can refer to the
> {{*CREATE [OR REPLACE] FUNCTION*}} of
> [Spark|https://spark.apache.org/docs/3.1.2/sql-ref-syntax-ddl-create-function.html]
> to implement the alter function .
> {code:sql}
> CREATE [ OR REPLACE ] [ TEMPORARY ] FUNCTION [IF NOT EXISTS ]
> [db_name.]function_name AS class_name
> [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
> {code}
> * *OR REPLACE*
> If specified, the resources for the function are reloaded. This is mainly
> useful to pick up any changes made to the implementation of the function.
> This parameter is mutually exclusive to {{IF NOT EXISTS}} and can not be
> specified together.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Assigned] (HIVE-26314) Support alter function in Hive DDL
[
https://issues.apache.org/jira/browse/HIVE-26314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar reassigned HIVE-26314:
-
> Support alter function in Hive DDL
> --
>
> Key: HIVE-26314
> URL: https://issues.apache.org/jira/browse/HIVE-26314
> Project: Hive
> Issue Type: Task
> Components: Hive
>Affects Versions: 4.0.0-alpha-1
>Reporter: Wechar
>Assignee: Wechar
>Priority: Minor
> Fix For: 4.0.0-alpha-2
>
>
> Hive SQL does not support {{*ALTER FUNCTION*}} yet, we can refer to the
> {{*CREATE [OR REPLACE] FUNCTION*}} of
> [Spark|https://spark.apache.org/docs/3.1.2/sql-ref-syntax-ddl-create-function.html]
> to implement the alter function .
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Assigned] (HIVE-26259) Alter Function does not update resource uris
[
https://issues.apache.org/jira/browse/HIVE-26259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar reassigned HIVE-26259:
-
> Alter Function does not update resource uris
>
>
> Key: HIVE-26259
> URL: https://issues.apache.org/jira/browse/HIVE-26259
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.2
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Fix For: 4.0.0-alpha-2
>
>
> *Bug Description:*
> The jar of Hive permanent UDF can be loaded based on the resource uris, but
> we encountered an issue after changing the resource uris through spark-sql:
> {code:sql}
> CREATE OR REPLACE FUNCTION test_db.test_udf AS 'com.xxx.xxx'
> USING JAR 'hdfs://path/to/jar';
> {code}
> Then when we use the UDF `test_db.test_udf`, an error occured like this:
> {code:sh}
> Error in query: Can not load class 'com.xxx.xxx' when registering the
> function 'test_db.test_udf'...
> {code}
> *Root Cause:*
> Hive metastore does not update resource uris while executing
> `alter_function()`, which should be included and will not make any side
> effect.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HIVE-26227) Add support of catalog related statements for Hive ql
[
https://issues.apache.org/jira/browse/HIVE-26227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17539577#comment-17539577
]
Wechar commented on HIVE-26227:
---
Not quite yet. In our scene, HMS is in charge of handling catalogs as the
metadata center, the other computing engines will only use catalogs while
querying. Simply put, we extend the original `db_name.tbl_name` to
`cat_name.db_name.tbl_name` in computing engines to support data from different
systems or sources.
So we do not plan to manage catalogs in other components now.
> Add support of catalog related statements for Hive ql
> -
>
> Key: HIVE-26227
> URL: https://issues.apache.org/jira/browse/HIVE-26227
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Catalog concept is proposed to Hive 3.0 to allow different systems to connect
> to different catalogs in the metastore. But so far we can not query catalog
> through Hive ql, this task aims to implement the ddl statements related to
> catalog.
> *Create Catalog*
> {code:sql}
> CREATE CATALOG [IF NOT EXISTS] catalog_name
> LOCATION hdfs_path
> [COMMENT catalog_comment];
> {code}
> LOCATION is required for creating a new catalog now.
> *Alter Catalog*
> {code:sql}
> ALTER CATALOG catalog_name SET LOCATION hdfs_path;
> {code}
> Only location metadata can be altered for catalog.
> *Drop Catalog*
> {code:sql}
> DROP CATALOG [IF EXISTS] catalog_name;
> {code}
> DROP CATALOG is always RESTRICT, which means DROP CATALOG will fail if there
> are non-default databases in the catalog.
> *Show Catalogs*
> {code:sql}
> SHOW CATALOGS [LIKE 'identifier_with_wildcards'];
> {code}
> SHOW CATALOGS lists all of the catalogs defined in the metastore.
> The optional LIKE clause allows the list of catalogs to be filtered using a
> regular expression.
> *Describe Catalog*
> {code:sql}
> DESC[RIBE] CATALOG [EXTENDED] cat_name;
> {code}
> DESCRIBE CATALOG shows the name of the catalog, its comment (if one has been
> set), and its root location on the filesystem.
> EXTENDED also shows the create time.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HIVE-26227) Add support of catalog related statements for Hive ql
[
https://issues.apache.org/jira/browse/HIVE-26227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17536689#comment-17536689
]
Wechar commented on HIVE-26227:
---
Sure [~zabetak]. We plan to provide a unified metadata management service
through the Hive metastore, which means the metadata of various systems are
stored in Hive metastore and divided by catalog.
Currently we want to manage the metadata from Hive, Hbase, Kafka, Jdbc, etc,
and computing engines like Hive, Spark, Presto, Flink can join data from
different systems based on the metadata in Hive metastore.
> Add support of catalog related statements for Hive ql
> -
>
> Key: HIVE-26227
> URL: https://issues.apache.org/jira/browse/HIVE-26227
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Catalog concept is proposed to Hive 3.0 to allow different systems to connect
> to different catalogs in the metastore. But so far we can not query catalog
> through Hive ql, this task aims to implement the ddl statements related to
> catalog.
> *Create Catalog*
> {code:sql}
> CREATE CATALOG [IF NOT EXISTS] catalog_name
> LOCATION hdfs_path
> [COMMENT catalog_comment];
> {code}
> LOCATION is required for creating a new catalog now.
> *Alter Catalog*
> {code:sql}
> ALTER CATALOG catalog_name SET LOCATION hdfs_path;
> {code}
> Only location metadata can be altered for catalog.
> *Drop Catalog*
> {code:sql}
> DROP CATALOG [IF EXISTS] catalog_name;
> {code}
> DROP CATALOG is always RESTRICT, which means DROP CATALOG will fail if there
> are non-default databases in the catalog.
> *Show Catalogs*
> {code:sql}
> SHOW CATALOGS [LIKE 'identifier_with_wildcards'];
> {code}
> SHOW CATALOGS lists all of the catalogs defined in the metastore.
> The optional LIKE clause allows the list of catalogs to be filtered using a
> regular expression.
> *Describe Catalog*
> {code:sql}
> DESC[RIBE] CATALOG [EXTENDED] cat_name;
> {code}
> DESCRIBE CATALOG shows the name of the catalog, its comment (if one has been
> set), and its root location on the filesystem.
> EXTENDED also shows the create time.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (HIVE-26227) Add support of catalog related statements for Hive ql
[
https://issues.apache.org/jira/browse/HIVE-26227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-26227:
--
Description:
Catalog concept is proposed to Hive 3.0 to allow different systems to connect
to different catalogs in the metastore. But so far we can not query catalog
through Hive ql, this task aims to implement the ddl statements related to
catalog.
*Create Catalog*
{code:sql}
CREATE CATALOG [IF NOT EXISTS] catalog_name
LOCATION hdfs_path
[COMMENT catalog_comment];
{code}
LOCATION is required for creating a new catalog now.
*Alter Catalog*
{code:sql}
ALTER CATALOG catalog_name SET LOCATION hdfs_path;
{code}
Only location metadata can be altered for catalog.
*Drop Catalog*
{code:sql}
DROP CATALOG [IF EXISTS] catalog_name;
{code}
DROP CATALOG is always RESTRICT, which means DROP CATALOG will fail if there
are non-default databases in the catalog.
*Show Catalogs*
{code:sql}
SHOW CATALOGS [LIKE 'identifier_with_wildcards'];
{code}
SHOW CATALOGS lists all of the catalogs defined in the metastore.
The optional LIKE clause allows the list of catalogs to be filtered using a
regular expression.
*Describe Catalog*
{code:sql}
DESC[RIBE] CATALOG [EXTENDED] cat_name;
{code}
DESCRIBE CATALOG shows the name of the catalog, its comment (if one has been
set), and its root location on the filesystem.
EXTENDED also shows the create time.
was:
Catalog concept is proposed to Hive 3.0 to allow different systems to connect
to different catalogs in the metastore. But so far we can not query catalog
through Hive ql, this task aims to implement the ddl statements related to
catalog.
*Create Catalog*
{code:sql}
CREATE CATALOG [IF NOT EXISTS] catalog_name
LOCATION hdfs_path
[COMMENT catalog_comment];
{code}
LOCATION is required for creating a new catalog now.
*Alter Catalog*
{code:sql}
ALTER CATALOG catalog_name SET LOCATION hdfs_path;
{code}
Only location metadata can be altered for catalog.
*Drop Catalog*
{code:sql}
DROP CATALOG [IF EXISTS] catalog_name;
{code}
DROP CATALOG is always RESTRICT, which means DROP CATALOG will fail if there
are non-default databases in the catalog.
*Show Catalog*
{code:sql}
SHOW CATALOGS [LIKE 'identifier_with_wildcards'];
{code}
SHOW CATALOGS lists all of the catalogs defined in the metastore.
The optional LIKE clause allows the list of catalogs to be filtered using a
regular expression.
*Describe Catalog*
{code:sql}
DESC[RIBE] CATALOG [EXTENDED] cat_name;
{code}
DESCRIBE CATALOG shows the name of the catalog, its comment (if one has been
set), and its root location on the filesystem.
EXTENDED also shows the create time.
> Add support of catalog related statements for Hive ql
> -
>
> Key: HIVE-26227
> URL: https://issues.apache.org/jira/browse/HIVE-26227
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Minor
> Fix For: 4.0.0-alpha-2
>
>
> Catalog concept is proposed to Hive 3.0 to allow different systems to connect
> to different catalogs in the metastore. But so far we can not query catalog
> through Hive ql, this task aims to implement the ddl statements related to
> catalog.
> *Create Catalog*
> {code:sql}
> CREATE CATALOG [IF NOT EXISTS] catalog_name
> LOCATION hdfs_path
> [COMMENT catalog_comment];
> {code}
> LOCATION is required for creating a new catalog now.
> *Alter Catalog*
> {code:sql}
> ALTER CATALOG catalog_name SET LOCATION hdfs_path;
> {code}
> Only location metadata can be altered for catalog.
> *Drop Catalog*
> {code:sql}
> DROP CATALOG [IF EXISTS] catalog_name;
> {code}
> DROP CATALOG is always RESTRICT, which means DROP CATALOG will fail if there
> are non-default databases in the catalog.
> *Show Catalogs*
> {code:sql}
> SHOW CATALOGS [LIKE 'identifier_with_wildcards'];
> {code}
> SHOW CATALOGS lists all of the catalogs defined in the metastore.
> The optional LIKE clause allows the list of catalogs to be filtered using a
> regular expression.
> *Describe Catalog*
> {code:sql}
> DESC[RIBE] CATALOG [EXTENDED] cat_name;
> {code}
> DESCRIBE CATALOG shows the name of the catalog, its comment (if one has been
> set), and its root location on the filesystem.
> EXTENDED also shows the create time.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Assigned] (HIVE-26227) Add support of catalog related statements for Hive ql
[
https://issues.apache.org/jira/browse/HIVE-26227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar reassigned HIVE-26227:
-
> Add support of catalog related statements for Hive ql
> -
>
> Key: HIVE-26227
> URL: https://issues.apache.org/jira/browse/HIVE-26227
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Minor
> Fix For: 4.0.0-alpha-2
>
>
> Catalog concept is proposed to Hive 3.0 to allow different systems to connect
> to different catalogs in the metastore. But so far we can not query catalog
> through Hive ql, this task aims to implement the ddl statements related to
> catalog.
> *Create Catalog*
> {code:sql}
> CREATE CATALOG [IF NOT EXISTS] catalog_name
> LOCATION hdfs_path
> [COMMENT catalog_comment];
> {code}
> LOCATION is required for creating a new catalog now.
> *Alter Catalog*
> {code:sql}
> ALTER CATALOG catalog_name SET LOCATION hdfs_path;
> {code}
> Only location metadata can be altered for catalog.
> *Drop Catalog*
> {code:sql}
> DROP CATALOG [IF EXISTS] catalog_name;
> {code}
> DROP CATALOG is always RESTRICT, which means DROP CATALOG will fail if there
> are non-default databases in the catalog.
> *Show Catalog*
> {code:sql}
> SHOW CATALOGS [LIKE 'identifier_with_wildcards'];
> {code}
> SHOW CATALOGS lists all of the catalogs defined in the metastore.
> The optional LIKE clause allows the list of catalogs to be filtered using a
> regular expression.
> *Describe Catalog*
> {code:sql}
> DESC[RIBE] CATALOG [EXTENDED] cat_name;
> {code}
> DESCRIBE CATALOG shows the name of the catalog, its comment (if one has been
> set), and its root location on the filesystem.
> EXTENDED also shows the create time.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Assigned] (HIVE-26205) Remove the incorrect org.slf4j dependency in kafka-handler
[
https://issues.apache.org/jira/browse/HIVE-26205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar reassigned HIVE-26205:
-
> Remove the incorrect org.slf4j dependency in kafka-handler
> --
>
> Key: HIVE-26205
> URL: https://issues.apache.org/jira/browse/HIVE-26205
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0-alpha-2
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Fix For: 4.0.0-alpha-2
>
>
> Get a compile error while executing:
> {code:bash}
> mvn clean install -DskipTests
> {code}
> The error message is:
> {code:bash}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile
> (default-compile) on project kafka-handler: Compilation failure: Compilation
> failure:
> [ERROR]
> /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaStorageHandler.java:[53,17]
> package org.slf4j does not exist
> [ERROR]
> /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaStorageHandler.java:[54,17]
> package org.slf4j does not exist
> [ERROR]
> /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaStorageHandler.java:[73,24]
> cannot find symbol
> [ERROR] symbol: class Logger
> [ERROR] location: class org.apache.hadoop.hive.kafka.KafkaStorageHandler
> [ERROR]
> /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/VectorizedKafkaRecordReader.java:[37,17]
> package org.slf4j does not exist
> [ERROR]
> /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/VectorizedKafkaRecordReader.java:[47,24]
> cannot find symbol
> [ERROR] symbol: class Logger
> [ERROR] location: class
> org.apache.hadoop.hive.kafka.VectorizedKafkaRecordReader
> [ERROR]
> /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaJsonSerDe.java:[63,17]
> package org.slf4j does not exist
> [ERROR]
> /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/SimpleKafkaWriter.java:[35,17]
> package org.slf4j does not exist
> [ERROR]
> /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/SimpleKafkaWriter.java:[50,24]
> cannot find symbol
> [ERROR] symbol: class Logger
> [ERROR] location: class org.apache.hadoop.hive.kafka.SimpleKafkaWriter
> [ERROR]
> /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaOutputFormat.java:[34,17]
> package org.slf4j does not exist
> [ERROR]
> /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaOutputFormat.java:[43,24]
> cannot find symbol
> [ERROR] symbol: class Logger
> [ERROR] location: class org.apache.hadoop.hive.kafka.KafkaOutputFormat
> [ERROR]
> /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/RetryUtils.java:[24,17]
> package org.slf4j does not exist
> [ERROR]
> /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/RetryUtils.java:[34,24]
> cannot find symbol
> [ERROR] symbol: class Logger
> [ERROR] location: class org.apache.hadoop.hive.kafka.RetryUtils
> [ERROR]
> /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaScanTrimmer.java:[51,17]
> package org.slf4j does not exist
> [ERROR]
> /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaScanTrimmer.java:[65,24]
> cannot find symbol
> [ERROR] symbol: class Logger
> [ERROR] location: class org.apache.hadoop.hive.kafka.KafkaScanTrimmer
> [ERROR]
> /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/TransactionalKafkaWriter.java:[45,17]
> package org.slf4j does not exist
> [ERROR]
> /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/TransactionalKafkaWriter.java:[65,24]
> cannot find symbol
> [ERROR] symbol: class Logger
> [ERROR] location: class
> org.apache.hadoop.hive.kafka.TransactionalKafkaWriter
> [ERROR]
> /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/HiveKafkaProducer.java:[37,17]
> package org.slf4j does not exist
> [ERROR]
> /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/HiveKafkaProducer.java:[59,24]
> cannot find symbol
> [ERROR] symbol: class Logger
> [ERROR] location: class org.apache.hadoop.hive.kafka.HiveKafkaProducer
> [ERROR]
> /ldap_home/weiqiang.yu/forked-hive/kafka-handler/src/java/org/apache/hadoop/hive/kafka/KafkaRecordIterator.java:[30,17]
> package org.slf4j does not exist
> [ERROR]
>
[jira] [Updated] (HIVE-26159) hive cli is unavailable from hive command
[
https://issues.apache.org/jira/browse/HIVE-26159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar updated HIVE-26159:
--
Description:
Hive cli is a convenient tool to connect to hive metastore service, but now
hive cli can not start even if we use *--service cli* option, it should be a
bug of ticket HIVE-24348.
*Steps to reproduce:*
{code:bash}
hive@hive:/root$ /usr/share/hive/bin/hive --service cli --hiveconf
hive.metastore.uris=thrift://hive:9084
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/opt/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/opt/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Beeline version 4.0.0-alpha-2-SNAPSHOT by Apache Hive
beeline>
{code}
was:
Hive cli is a convenient tool to connect to hive metastore service, but now
hive cli can not start even if we use *--service cli* option, it should be a
bug of ticket [HIVE-24348|https://issues.apache.org/jira/browse/HIVE-24348].
*Step to reproduce:*
{code:bash}
hive@hive:/root$ /usr/share/hive/bin/hive --service cli --hiveconf
hive.metastore.uris=thrift://hive:9084
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/opt/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/opt/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Beeline version 4.0.0-alpha-2-SNAPSHOT by Apache Hive
beeline>
{code}
> hive cli is unavailable from hive command
> -
>
> Key: HIVE-26159
> URL: https://issues.apache.org/jira/browse/HIVE-26159
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0-alpha-1
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Fix For: 4.0.0
>
>
> Hive cli is a convenient tool to connect to hive metastore service, but now
> hive cli can not start even if we use *--service cli* option, it should be a
> bug of ticket HIVE-24348.
> *Steps to reproduce:*
> {code:bash}
> hive@hive:/root$ /usr/share/hive/bin/hive --service cli --hiveconf
> hive.metastore.uris=thrift://hive:9084
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Beeline version 4.0.0-alpha-2-SNAPSHOT by Apache Hive
> beeline>
> {code}
--
[jira] [Assigned] (HIVE-26159) hive cli is unavailable from hive command
[
https://issues.apache.org/jira/browse/HIVE-26159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar reassigned HIVE-26159:
-
> hive cli is unavailable from hive command
> -
>
> Key: HIVE-26159
> URL: https://issues.apache.org/jira/browse/HIVE-26159
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0-alpha-1
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Fix For: 4.0.0
>
>
> Hive cli is a convenient tool to connect to hive metastore service, but now
> hive cli can not start even if we use *--service cli* option, it should be a
> bug of ticket [HIVE-24348|https://issues.apache.org/jira/browse/HIVE-24348].
> *Step to reproduce:*
> {code:bash}
> hive@hive:/root$ /usr/share/hive/bin/hive --service cli --hiveconf
> hive.metastore.uris=thrift://hive:9084
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Beeline version 4.0.0-alpha-2-SNAPSHOT by Apache Hive
> beeline>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Assigned] (HIVE-26148) Keep MetaStoreFilterHook interface compatibility after introducing catalogs
[
https://issues.apache.org/jira/browse/HIVE-26148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar reassigned HIVE-26148:
-
> Keep MetaStoreFilterHook interface compatibility after introducing catalogs
> ---
>
> Key: HIVE-26148
> URL: https://issues.apache.org/jira/browse/HIVE-26148
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Wechar
>Assignee: Wechar
>Priority: Minor
> Fix For: 4.0.0-alpha-1
>
>
> Hive 3.0 introduce catalog concept, when we upgrade hive dependency version
> from 2.3 to 3.x, we found some interfaces of *MetaStoreFilterHook* are not
> compatible:
> {code:bash}
> git show ba8a99e115 --
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreFilterHook.java
> {code}
> {code:bash}
> ---
> a/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreFilterHook.java
> +++
> b/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreFilterHook.java
>/**
> * Filter given list of tables
> - * @param dbName
> - * @param tableList
> + * @param catName catalog name
> + * @param dbName database name
> + * @param tableList list of table returned by the metastore
> * @return List of filtered table names
> */
> - public List filterTableNames(String dbName, List
> tableList) throws MetaException;
> + List filterTableNames(String catName, String dbName, List
> tableList)
> + throws MetaException;
> {code}
> We can remain the previous interfaces and use the default catalog to
> implement.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
