[jira] [Created] (HIVE-26644) Introduce auto sizing in HMS
Zhihua Deng created HIVE-26644: -- Summary: Introduce auto sizing in HMS Key: HIVE-26644 URL: https://issues.apache.org/jira/browse/HIVE-26644 Project: Hive Issue Type: Improvement Components: Standalone Metastore Reporter: Zhihua Deng Assignee: Zhihua Deng HMS should have some ability to auto-size itself based on enabled features. Server thread pool sizes-to-HMS connection pool sizes, larger pool sizes on compaction-disabled-instances for better performance etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26643) HiveUnionPullUpConstantsRule fails when pulling up constants over nullable fields
Alessandro Solimando created HIVE-26643:
---
Summary: HiveUnionPullUpConstantsRule fails when pulling up
constants over nullable fields
Key: HIVE-26643
URL: https://issues.apache.org/jira/browse/HIVE-26643
Project: Hive
Issue Type: Bug
Components: CBO
Affects Versions: 4.0.0-alpha-2
Reporter: Alessandro Solimando
Assignee: Alessandro Solimando
The rule does pull up constants without checking/adjusting nullability to match
that of the field type.
Here is the stack-trace when a nullable type is involved:
{code:java}
java.lang.AssertionError: Cannot add expression of different type to set:
set type is RecordType(JavaType(class java.lang.Integer) f1, JavaType(int) NOT
NULL f2) NOT NULL
expression type is RecordType(JavaType(int) NOT NULL f1, JavaType(int) NOT NULL
f2) NOT NULL
set is
rel#38:HiveUnion.(input#0=HepRelVertex#35,input#1=HepRelVertex#35,all=true)
expression is HiveProject(f1=[1], f2=[$0])
HiveUnion(all=[true])
HiveProject(f2=[$1])
HiveProject(f1=[$0], f2=[$1])
HiveFilter(condition=[=($0, 1)])
LogicalTableScan(table=[[]])
HiveProject(f2=[$1])
HiveProject(f1=[$0], f2=[$1])
HiveFilter(condition=[=($0, 1)])
LogicalTableScan(table=[[]])
{code}
The solution is to check nullability and add a cast when the field is nullable,
since the constant's type is not.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26642) Replace HiveFilterMergeRule with Calcite's built-in implementation
Stamatis Zampetakis created HIVE-26642: -- Summary: Replace HiveFilterMergeRule with Calcite's built-in implementation Key: HIVE-26642 URL: https://issues.apache.org/jira/browse/HIVE-26642 Project: Hive Issue Type: Improvement Reporter: Stamatis Zampetakis Assignee: Stamatis Zampetakis The rule was copied from Calcite to address HIVE-23389 as a temporary workaround till the next Calcite upgrade. Now that Hive is on calcite 1.25.0 (HIVE-23456) the in-house copy can be removed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26641) Upgrade Guava: Google Core Libraries for Java to v28.2/31.1-jre due to medium CVEs
Devaspati Krishnatri created HIVE-26641: --- Summary: Upgrade Guava: Google Core Libraries for Java to v28.2/31.1-jre due to medium CVEs Key: HIVE-26641 URL: https://issues.apache.org/jira/browse/HIVE-26641 Project: Hive Issue Type: Task Reporter: Devaspati Krishnatri -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26640) Upgrade JUnit to 4.13.2 due to medium CVEs
Devaspati Krishnatri created HIVE-26640: --- Summary: Upgrade JUnit to 4.13.2 due to medium CVEs Key: HIVE-26640 URL: https://issues.apache.org/jira/browse/HIVE-26640 Project: Hive Issue Type: Task Reporter: Devaspati Krishnatri -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26639) ConstantVectorExpression shouldn't rely on default charset
László Bodor created HIVE-26639: --- Summary: ConstantVectorExpression shouldn't rely on default charset Key: HIVE-26639 URL: https://issues.apache.org/jira/browse/HIVE-26639 Project: Hive Issue Type: Bug Reporter: László Bodor -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26638) Replace in-house CBO reduce expressions rules with Calcite's built-in classes
Stamatis Zampetakis created HIVE-26638:
--
Summary: Replace in-house CBO reduce expressions rules with
Calcite's built-in classes
Key: HIVE-26638
URL: https://issues.apache.org/jira/browse/HIVE-26638
Project: Hive
Issue Type: Improvement
Components: CBO
Reporter: Stamatis Zampetakis
Assignee: Stamatis Zampetakis
The goal of this ticket is to remove Hive specific code in
[HiveReduceExpressionsRule|https://github.com/apache/hive/blob/b48c1bf11c4f75ba2c894e4732a96813ddde1414/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java]
and use exclusively the respective Calcite classes (i.e.,
[ReduceExpressionsRule|https://github.com/apache/calcite/blob/2c30a56158cdd351d35725006bc1f76bb6aac75b/core/src/main/java/org/apache/calcite/rel/rules/ReduceExpressionsRule.java])
to reduce maintenance overhead and facilitate code evolution.
Currently the only difference between in-house (HiveReduceExpressionsRule) and
built-in (ReduceExpressionsRule) reduce expressions rules lies in the way we
treat the {{Filter}} operator (i.e., FilterReduceExpressionsRule).
There are four differences when comparing the in-house code with the respective
part in Calcite 1.25.0 that are Hive specific.
+Match nullability when reducing expressions+
When we reduce filters we always set {{matchNullability}} (last parameter) to
false.
{code:java}
if (reduceExpressions(filter, expList, predicates, true, false)) {
{code}
This means that the original and reduced expression can have a slightly
different type in terms of nullability; the original is nullable and the
reduced is not nullable. When the value is true the type can be preserved by
adding a "nullability" CAST, which is a cast to the same type which differs
only to if it is nullable or not.
Hardcoding {{matchNullability}} to false was done as part of the upgrade in
Calcite 1.15.0 (HIVE-18068) where the behavior of the rule became configurable
(CALCITE-2041).
+Remove nullability cast explicitly+
When the expression is reduced we try to remove the nullability cast; if there
is one.
{code:java}
if (RexUtil.isNullabilityCast(filter.getCluster().getTypeFactory(),
newConditionExp)) {
newConditionExp = ((RexCall) newConditionExp).getOperands().get(0);
}
{code}
The code was added as part of the upgrade to Calcite 1.10.0 (HIVE-13316).
However, the code is redundant as of HIVE-18068; setting {{matchNullability}}
to {{false}} no longer generates nullability casts during the reduction.
+Avoid creating filters with condition of type NULL+
{code:java}
if(newConditionExp.getType().getSqlTypeName() == SqlTypeName.NULL) {
newConditionExp = call.builder().cast(newConditionExp,
SqlTypeName.BOOLEAN);
}
{code}
Hive tries to cast such expressions to BOOLEAN to avoid the weird (and possibly
problematic) situation of having a condition with NULL type.
In Calcite, there is specific code for detecting if the new condition is the
NULL literal (with NULL type) and if that's the case it turns the relation to
empty.
{code:java}
} else if (newConditionExp instanceof RexLiteral
|| RexUtil.isNullLiteral(newConditionExp, true)) {
call.transformTo(createEmptyRelOrEquivalent(call, filter));
{code}
Due to that the Hive specific code is redundant if the Calcite rule is used.
+Bail out when input to reduceNotNullableFilter is not a RexCall+
{code:java}
if (!(rexCall.getOperands().get(0) instanceof RexCall)) {
// If child is not a RexCall instance, we can bail out
return;
}
{code}
The code was added as part of the upgrade to Calcite 1.10.0 (HIVE-13316) but it
does not add any functional value.
The instanceof check is redundant since the code in reduceNotNullableFilter [is
a
noop|https://github.com/apache/hive/blob/6e8fc53fb68898d1a404435859cea5bbc79200a4/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java#L228]
when the expression/call is not one of the following: IS_NULL, IS_UNKNOWN,
IS_NOT_NULL, which are all rex calls.
+Summary+
All of the Hive specific changes mentioned previously can be safely replaced by
appropriate uses of the Calcite APIs without affecting the behavior of CBO.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26637) package failed without -DskipTests=true
happyziqi2 created HIVE-26637: - Summary: package failed without -DskipTests=true Key: HIVE-26637 URL: https://issues.apache.org/jira/browse/HIVE-26637 Project: Hive Issue Type: Bug Components: Hive Affects Versions: 3.1.2, All Versions Reporter: happyziqi2 Fix For: All Versions mvn package failed Test testInsertOverwriteForPartitionedMmTable() in org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable fail -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26636) Hyperbolic functions
Gopinath created HIVE-26636: --- Summary: Hyperbolic functions Key: HIVE-26636 URL: https://issues.apache.org/jira/browse/HIVE-26636 Project: Hive Issue Type: New Feature Reporter: Gopinath Adding hyperbolic functions. |Function|Description|Example|Result| |sinh(x)|hyperbolic sine|sinh(0)|0| |cosh(x)|hyperbolic cosine|cosh(0)|1| |tanh(x)|hyperbolic tangent|tanh(0)|0| |asinh(x)|inverse hyperbolic sine|asinh(0)|0| |acosh(x)|inverse hyperbolic cosine|acosh(1)|0| |atanh(x)|inverse hyperbolic tangent|atanh(0)|0| -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26635) Maintain update and merge mode on Iceberg V2 tables
Ádám Szita created HIVE-26635: - Summary: Maintain update and merge mode on Iceberg V2 tables Key: HIVE-26635 URL: https://issues.apache.org/jira/browse/HIVE-26635 Project: Hive Issue Type: Improvement Reporter: Ádám Szita Assignee: Ádám Szita HIVE-26596 took care of maintaining the delete-mode setting on Iceberg V2 tables, but lacks the same for update- and merge modes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
