Re: Review Request 71761: HIVE-22489

[Krisztian Kasa]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71761/
---

(Updated Dec. 17, 2019, 8:57 a.m.)


Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.


Bugs: HIVE-22489
https://issues.apache.org/jira/browse/HIVE-22489


Repository: hive-git


Description (updated)
---

Reduce Sink operator orders nulls first
===
1. Set the default null sort order by hive config when creating Reduce Sink 
Desc.
2. Hash join uses 
`org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableSerializeWrite`
 or `BinarySortableDeserializeRead` for selializing keys. For bigtable keys 
always ascending and nulls first ordering was hardcoded. This patch changes 
this behaviour to use the `Operator.getConf().TableDesc.getProperties()` (in 
this case `MapJoinOperator`) to setup ordering in `BinarySortableSerializeWrite`
3. Use null ordering set in ReduceRecordSource at Reduce phase when comparing 
keys in `CommonMergeJoinOperator` (This is the null ordering of the children 
Reduce Sink operators)


Diffs (updated)
-

  accumulo-handler/src/test/results/positive/accumulo_queries.q.out 7c552621f2 
  contrib/src/test/results/clientpositive/udaf_example_group_concat.q.out 
6846720d95 
  hbase-handler/src/test/results/positive/hbase_queries.q.out a32ef81a7b 
  
itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
 e997fa65cf 
  kudu-handler/src/test/results/positive/kudu_complex_queries.q.out 1324b27f8e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/CommonMergeJoinOperator.java 
3974627a24 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordSource.java 
72446afeda 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinCommonOperator.java
 2380d936f2 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerBigOnlyMultiKeyOperator.java
 f587517b08 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerMultiKeyOperator.java
 cdee3fd957 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinLeftSemiMultiKeyOperator.java
 e5d9fdae19 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinOuterMultiKeyOperator.java
 29c531bd51 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMap.java
 a4cda921a5 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMultiSet.java
 43f093d906 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashSet.java
 8dce5b82d3 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashTable.java
 a35401d9b2 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringCommon.java
 1b108a8c14 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMap.java
 446feb2526 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMultiSet.java
 c28ef9be2b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashSet.java
 17bd5fda93 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastTableContainer.java
 4ab8902a3f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedCreateHashTable.java
 21c355cb42 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongCommon.java
 de1ee15c3b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongHashMap.java
 42573f0898 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongHashMultiSet.java
 829a03737d 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongHashSet.java
 18e1435019 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringCommon.java
 da0e8365b1 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashMap.java
 6c4d8a81d1 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashMultiSet.java
 a6b754c7eb 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashSet.java
 fdcd83dde7 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
 5c409e4573 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/CountDistinctRewriteProc.java 
a50ad78e8f 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
 0f95d7788c 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
89b55001f0 
  
ql/src/java/org/apache

[jira] [Created] (HIVE-22652) TopNKey push through Group by with Grouping sets

[Krisztian Kasa (Jira)]

Krisztian Kasa created HIVE-22652:
-

 Summary: TopNKey push through Group by with Grouping sets
 Key: HIVE-22652
 URL: https://issues.apache.org/jira/browse/HIVE-22652
 Project: Hive
  Issue Type: Improvement
Reporter: Krisztian Kasa






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22653) Remove commons-lang leftovers

[David Lavati (Jira)]

David Lavati created HIVE-22653:
---

 Summary: Remove commons-lang leftovers
 Key: HIVE-22653
 URL: https://issues.apache.org/jira/browse/HIVE-22653
 Project: Hive
  Issue Type: Bug
Reporter: David Lavati
Assignee: David Lavati
 Fix For: 4.0.0


HIVE-7145 removed commons-lang - in favor of commons-lang3 - as a direct 
dependency, however a high number of files still refer to commons-lang, which 
is transitively brought in either way.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 71820: HIVE-20150

[Ramesh Kumar Thangarajan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/#review219047
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
Lines 4341 (patched)


What is the bug in Decimal64 to Decimal conversion? 

Do we need a call to the function fixDecimalDataTypePhysicalVariations() ? 
Because I see the function getVectorExpressionsUpConvertDecimal64() is calling 
wrapWithDecimal64ToDecimalConversion() on all the child expressions.


- Ramesh Kumar Thangarajan


On Dec. 14, 2019, 10:31 a.m., Krisztian Kasa wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71820/
> ---
> 
> (Updated Dec. 14, 2019, 10:31 a.m.)
> 
> 
> Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.
> 
> 
> Bugs: HIVE-20150
> https://issues.apache.org/jira/browse/HIVE-20150
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TopNKey pushdown
> 
> 1. Apply patch: 
> https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
> 2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
> 3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
> sink, left outer join, other topnkey
> 4. Add sort order and null sort order direction check when determining if the 
> topnkey op can be pushed
> 5. Implement handling cases when topnkey op and the parent op has a common 
> key prefix only.
> 6. fix Key object inspectors non-vectorized mode 
> 7. fix decimal64 to decimal cast issues when creating VectorExpressions of 
> keyExpression during TopnKey vectorization
> 
> 
> Diffs
> -
> 
>   kudu-handler/src/test/results/positive/kudu_complex_queries.q.out 
> 1324b27f8e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java bbbde7978b 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
> 0d6cf3c755 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 6876787e11 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/CommonKeyPrefix.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 5c7a64c950 
>   
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/topnkey/TestCommonKeyPrefix.java
>  PRE-CREATION 
>   ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
>   ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
>   ql/src/test/results/clientpositive/druid/druidmini_expressions.q.out 
> 31862546b0 
>   ql/src/test/results/clientpositive/druid/druidmini_test1.q.out fe75ddd894 
>   ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 0c051c926b 
>   ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out 53e57b892f 
>   ql/src/test/results/clientpositive/llap/check_constraint.q.out 9f2c9a1cd0 
>   ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
> b6d210becf 
>   ql/src/test/results/clientpositive/llap/ctas.q.out c10d75d081 
>   
> ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out 
> 4e62779446 
>   ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out 
> 6688944b78 
>   ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
> 9343e078b7 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 64fdc481eb 
>   ql/src/test/results/clientpositive/llap/explainuser_2.q.out 0219af8833 
>   ql/src/test/results/clientpositive/llap/external_jdbc_table_perf.q.out 
> 545cce75a9 
>   ql/src/test/results/clientpositive/llap/filter_union.q.out 0df77762a0 
>   ql/src/test/results/clientpositive/llap/groupby_groupingset_bug.q.out 
> 63d56172b7 
>   ql/src/test/results/clientpositive/llap/insert1_overwrite_partitions.q.out 
> d51654bc5b 
>   ql/src/test/results/clientpositive/llap/insert_into1.q.out dd9140d3b6 
>   ql/src/test/results/clientpositive/llap/insert_into2.q.out 161318b720 
>   ql/src/test/results/clientpositive/llap/lateral_view.q.out b742230a71 
>   ql/src/test/results/clientpositive/llap/limit_join_transpose.q.out 
> 6258f4fe1e 
>   ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 3fdd77d802 
>   ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out efa8c38d7c 
>   ql/src/test/results/clientpositive/llap/materialized_view_rewrite_8.q.out 
> 303111a7d4 
>   ql/src/test/results/clientpositive/llap/offset_limit.q.out 23f2de46e5 
>   ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
> 4ecb7bc46d 
>   ql/src/test/results/clientpositive/llap/orc_predicate_pushdown.q.out 
> 2fd8

[jira] [Created] (HIVE-22654) ACID: Allow TxnHandler::checkLock to chunk partitions by 1000

[Gopal Vijayaraghavan (Jira)]

Gopal Vijayaraghavan created HIVE-22654:
---

 Summary: ACID: Allow TxnHandler::checkLock to chunk partitions by 
1000 
 Key: HIVE-22654
 URL: https://issues.apache.org/jira/browse/HIVE-22654
 Project: Hive
  Issue Type: Bug
Reporter: Gopal Vijayaraghavan


The following loop can end up with too many entries within the IN clause 
throwing 

{code:java}
        // If any of the partition requests are null, then I need to pull all
        // partition locks for this table.
        sawNull = false;
        strings.clear();
        for (LockInfo info : locksBeingChecked) {
          if (info.partition == null) {
            sawNull = true;
            break;
          } else {
            strings.add(info.partition);
          }
        } 
{code}

{code}
2019-12-17T04:28:57,991 ERROR [pool-8-thread-143]: metastore.RetryingHMSHandler 
(RetryingHMSHandler.java:invokeInternal(201)) - MetaException(message:Unable to 
update transaction database java.sql.SQLSyntaxErrorException: ORA-01795: 
maximum number of expressions in a list is 1000
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22655) Unable to pass Bearer Token in http.header.Authorization in JDBC Header

[Neeraj Verma (Jira)]

Neeraj Verma created HIVE-22655:
---

 Summary: Unable to pass Bearer Token in http.header.Authorization  
in JDBC Header
 Key: HIVE-22655
 URL: https://issues.apache.org/jira/browse/HIVE-22655
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Reporter: Neeraj Verma


While trying to submit Hive Job over JDBC  ,  I faced 2 issues
 # Space not supported between "{{Bearer ey.."}}
 # {{if we remove space then http }}{{Authorization}} header is overwritten by 
default Basic token .

 

{{beeline -u 
"jdbc:hive2://localhost:8443/;ssl=true;sslTrustStore=/Users/abc/knox/install/knox-1.3.0/data/security/keystores/gateway.jks;trustStorePassword=knox;transportMode=http;httpPath=gateway/tokenbased/hive;http.header.Authorization=Bearer
 eyJraWQiOiJp...df"}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 71820: HIVE-20150

[Krisztian Kasa]

> On Dec. 17, 2019, 10:01 p.m., Ramesh Kumar Thangarajan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
> > Lines 4341 (patched)
> > 
> >
> > What is the bug in Decimal64 to Decimal conversion? 
> > 
> > Do we need a call to the function 
> > fixDecimalDataTypePhysicalVariations() ? Because I see the function 
> > getVectorExpressionsUpConvertDecimal64() is calling 
> > wrapWithDecimal64ToDecimalConversion() on all the child expressions.

Please consider the following query: 
```
CREATE TABLE decimal_test_small_n0 STORED AS ORC AS SELECT cdouble, CAST 
(((cdouble*22.1)/37) AS DECIMAL(10,3)) AS cdecimal1, CAST (((cdouble*9.3)/13) 
AS DECIMAL(7,2)) AS cdecimal2 FROM alltypesorc;

SELECT cdecimal1 - (2*cdecimal2) as c2 FROM decimal_test_small_n0
ORDER BY c2
LIMIT 10;
```

With given keyColumns: GenericUDFOPMinus(Column[cdecimal1], 
GenericUDFOPMultiply(Const decimal(1,0) 2, Column[cdecimal2])) 
```
keyExpressions = vContext.getVectorExpressionsUpConvertDecimal64(keyColumns);
```
will produce keyExpressions: 
DecimalColSubtractDecimalColumn(col 4:decimal(10,3), col 
5:decimal(9,2)/DECIMAL_64)
(children: 
ConvertDecimal64ToDecimal(col 1:decimal(10,3)/DECIMAL_64) -> 
4:decimal(10,3), 
Decimal64ScalarMultiplyDecimal64ColumnUnscaled(decimal64Val 2, decimalVal 
2, col 2:decimal(7,2)/DECIMAL_64) -> 5:decimal(9,2)/DECIMAL_64
) -> 6:decimal(11,3)

So the 2nd child of DecimalColSubtractDecimalColumn is not converted from 
decimal64 to decimal and I got a 
```
 java.lang.ClassCastException: 
org.apache.hadoop.hive.ql.exec.vector.Decimal64ColumnVector cannot be cast to 
org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
at 
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.DecimalColSubtractDecimalColumn.evaluate(DecimalColSubtractDecimalColumn.java:69)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorTopNKeyOperator.process(VectorTopNKeyOperator.java:101)
```

Checking the code of `getVectorExpressionsUpConvertDecimal64` i found that it 
calls `getVectorExpressions` and for each result expression if that 
expression's output type is decimal64 then it wraps it with conversion. But it 
does not checks the children expressions.

Probably we don't need both function calls, please let me check in a follow up 
patch if `fixDecimalDataTypePhysicalVariations` is enough.


- Krisztian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71820/#review219047
---


On Dec. 14, 2019, 10:31 a.m., Krisztian Kasa wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71820/
> ---
> 
> (Updated Dec. 14, 2019, 10:31 a.m.)
> 
> 
> Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.
> 
> 
> Bugs: HIVE-20150
> https://issues.apache.org/jira/browse/HIVE-20150
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TopNKey pushdown
> 
> 1. Apply patch: 
> https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
> 2. TopNKey introduction depends only from Reduce Sink with topn property >= 0
> 3. Implement TopNKey operator pushdown through: projection, group by, redeuce 
> sink, left outer join, other topnkey
> 4. Add sort order and null sort order direction check when determining if the 
> topnkey op can be pushed
> 5. Implement handling cases when topnkey op and the parent op has a common 
> key prefix only.
> 6. fix Key object inspectors non-vectorized mode 
> 7. fix decimal64 to decimal cast issues when creating VectorExpressions of 
> keyExpression during TopnKey vectorization
> 
> 
> Diffs
> -
> 
>   kudu-handler/src/test/results/positive/kudu_complex_queries.q.out 
> 1324b27f8e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java bbbde7978b 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/TopNKeyProcessor.java 
> 0d6cf3c755 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 6876787e11 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/CommonKeyPrefix.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 5c7a64c950 
>   
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/topnkey/TestCommonKeyPrefix.java
>  PRE-CREATION 
>   ql/src/test/queries/clientpositive/topnkey.q 057b6a45ba 
>   ql/src/test/queries/clientpositive/vector_topnkey.q 85c5880cd6 
>   ql/src/test/results/clientpositive/druid/druidmini_expressions.q.out 
> 31862546b0 
>   ql/src/test/results/clientpositive/druid/druidmini_test1.