Re: Review Request 41457: HIVE-11865

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41457/
---

(Updated Dec. 18, 2015, 5:28 p.m.)


Review request for hive, Ashutosh Chauhan and John Pullokkaran.


Bugs: HIVE-11865
https://issues.apache.org/jira/browse/HIVE-11865


Repository: hive-git


Description
---

Disable Hive PPD optimizer when CBO has optimized the plan


Diffs (updated)
-

  hbase-handler/src/test/results/positive/hbase_queries.q.out 
3907bc946679c0ce2492692dcb7e2cf252c6e188 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 
8f48e7df4b9c081528b82545d445f75f67f93bfc 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
 142812ca703d5e667bd532a711d29594123bde50 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTSTransposeRule.java
 8321504774c5ec5c7c9de3aecdf78dcd8408c4cf 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java
 7e484b90deed2dfb120faa2699eb9b9b7a4e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortTransposeRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java
 c8de1d89a1df083cd125ae8c11bf854b09915680 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
 efde6652ca270d8e8d8f682fc0ff260702f46d5a 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
 feec3c25929cac0bb3b6f50d074557a3c2745889 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
7c5a43ff4060c9a4d9c4c5523d851ef959da3932 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
27549dceac8b5913b1fb1604bde213ae26b243bf 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
ccc4bb454e61e3c39b1f90f29de5e754f8d886bf 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 
1702628447d38b09c9b7d0dcf38218a9037a2c64 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/SimplePredicatePushDown.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
accfb3b9f5631f25220c88b759dba6c5299b4044 
  ql/src/test/results/clientpositive/auto_join12.q.out 
6f08aa8f043113f0a99acd10c848f60a122a3f0b 
  ql/src/test/results/clientpositive/auto_join13.q.out 
d29818cf438449a6a1413edd7d474c4ab35f2148 
  ql/src/test/results/clientpositive/auto_join16.q.out 
38738ede2b4cd49aa24490eafb81dec1de78bea5 
  ql/src/test/results/clientpositive/auto_join2.q.out 
ce40ff30cb9cc2ab734ee37c0f3dc0ae85bbd202 
  ql/src/test/results/clientpositive/auto_join33.q.out 
8b13bd0cb7fd869c58c9d092b2e8710391c96503 
  ql/src/test/results/clientpositive/auto_join_filters.q.out 
e0ed3737f9b25d45bdb365b13aeed3a40c50135f 
  ql/src/test/results/clientpositive/auto_join_nulls.q.out 
954bf069760c6f0d7e5420c7e9a3b9756d5aa4df 
  ql/src/test/results/clientpositive/auto_join_stats.q.out 
feb818679df7d9f63f2d2d7a7c9afb2d0f5874d8 
  ql/src/test/results/clientpositive/auto_join_stats2.q.out 
e0d0146dd6b86e9f69a467dcde0c020c31af499d 
  ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
c63016d0a3610eaa6cd82a4adc5d1bf6f0925cce 
  ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out 
096f5d445bc7e61381775270bef04e911eac5bb6 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 
30853c42a52e1def5a913352580e0b5391ac2050 
  ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 
3e8dbcfd5bcc66ccd83b01c08c7745b02f05328f 
  ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 
e6c2832003a1bb804f99125866bff6117e4dab73 
  ql/src/test/results/clientpositive/bucketsortoptimize_insert_7.q.out 
326e4d67f00158929f4aaa0a2ec53cedabe3e92c 
  ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
62b611b1e43fed3326c8c43e3266ad281c86a054 
  ql/src/test/results/clientpositive/cbo_rp_join0.q.out 
3c6bb7319fd7605a62f082b989f723d5aab6c3fe 
  ql/src/test/results/clientpositive/cbo_rp_join1.q.out 
e77002866174f85168859c4145e7016c9d829bb6 
  ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out 
b14caa80b3346036f9ec835fe4e2294f47eb7ee5 
  ql/src/test/results/clientpositive/cbo_rp_outer_join_ppr.q.java1.7.out 
263053705347793513a44d49968dedd4fdffecfe 
  ql/src/test/results/clientpositive/correlationoptimizer10.q.out 
860452ef9bbf38df7aaeb00eb2636f53730ae923 
  ql/src/test/results/clientpositive/correlationoptimizer13.q.out 
82304d8ff96e14bf18cde3f0867b25c6757d40d6 
  ql/src/test/results/clientpositive/correlationoptimizer9.q.out 
d32003dead268bd435713751d5e5956bdd4f5f74 
  ql/src/test/results/clientpositive/cross_join.q.out 
f36496e8503fedc67d6c9555b6e0dd5b0c9bd6ed 
  ql/src/test/results/clientpositive/cross_product_check_1.q.out 
e7d6900d64851fa21f976ca20b5d7d753aa5d28f 
  ql/src/test/results/clientpositive/cross_product_check_2.q.out 

[jira] [Created] (HIVE-12708) Hive on Spark doesn't work with Kerboresed HBase [Spark Branch]

[Xuefu Zhang (JIRA)]

Xuefu Zhang created HIVE-12708:
--

 Summary: Hive on Spark doesn't work with Kerboresed HBase [Spark 
Branch]
 Key: HIVE-12708
 URL: https://issues.apache.org/jira/browse/HIVE-12708
 Project: Hive
  Issue Type: Bug
  Components: Spark
Affects Versions: 1.1.0, 1.2.0, 2.0.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang


Spark application launcher (spark-submit) acquires HBase delegation token on 
Hive user's behalf when the application is launched. This mechanism, which 
doesn't work for long-running sessions, is not in line with what Hive is doing. 
Hive actually acquires the token automatically whenever a job needs it. The 
right approach for Spark should be allowing applications to dynamically add 
whatever tokens they need to the spark context. While this needs work on Spark 
side, we provide a workaround solution in Hive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41457: HIVE-11865

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41457/#review111254
---



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 491)


This is fragile; i.e reusing the flag whether to run CBO or not for whether 
CBO was actually run successfully.

We already keep state in parsecontext about this.


- John Pullokkaran


On Dec. 18, 2015, 5:28 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41457/
> ---
> 
> (Updated Dec. 18, 2015, 5:28 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Bugs: HIVE-11865
> https://issues.apache.org/jira/browse/HIVE-11865
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Disable Hive PPD optimizer when CBO has optimized the plan
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out 
> 3907bc946679c0ce2492692dcb7e2cf252c6e188 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 
> 8f48e7df4b9c081528b82545d445f75f67f93bfc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  142812ca703d5e667bd532a711d29594123bde50 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTSTransposeRule.java
>  8321504774c5ec5c7c9de3aecdf78dcd8408c4cf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java
>  7e484b90deed2dfb120faa2699eb9b9b7a4e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java
>  c8de1d89a1df083cd125ae8c11bf854b09915680 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
>  efde6652ca270d8e8d8f682fc0ff260702f46d5a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  feec3c25929cac0bb3b6f50d074557a3c2745889 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 7c5a43ff4060c9a4d9c4c5523d851ef959da3932 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 27549dceac8b5913b1fb1604bde213ae26b243bf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
> ccc4bb454e61e3c39b1f90f29de5e754f8d886bf 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 
> 1702628447d38b09c9b7d0dcf38218a9037a2c64 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SimplePredicatePushDown.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
> accfb3b9f5631f25220c88b759dba6c5299b4044 
>   ql/src/test/results/clientpositive/auto_join12.q.out 
> 6f08aa8f043113f0a99acd10c848f60a122a3f0b 
>   ql/src/test/results/clientpositive/auto_join13.q.out 
> d29818cf438449a6a1413edd7d474c4ab35f2148 
>   ql/src/test/results/clientpositive/auto_join16.q.out 
> 38738ede2b4cd49aa24490eafb81dec1de78bea5 
>   ql/src/test/results/clientpositive/auto_join2.q.out 
> ce40ff30cb9cc2ab734ee37c0f3dc0ae85bbd202 
>   ql/src/test/results/clientpositive/auto_join33.q.out 
> 8b13bd0cb7fd869c58c9d092b2e8710391c96503 
>   ql/src/test/results/clientpositive/auto_join_filters.q.out 
> e0ed3737f9b25d45bdb365b13aeed3a40c50135f 
>   ql/src/test/results/clientpositive/auto_join_nulls.q.out 
> 954bf069760c6f0d7e5420c7e9a3b9756d5aa4df 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out 
> feb818679df7d9f63f2d2d7a7c9afb2d0f5874d8 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 
> e0d0146dd6b86e9f69a467dcde0c020c31af499d 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
> c63016d0a3610eaa6cd82a4adc5d1bf6f0925cce 
>   ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out 
> 096f5d445bc7e61381775270bef04e911eac5bb6 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 
> 30853c42a52e1def5a913352580e0b5391ac2050 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 
> 3e8dbcfd5bcc66ccd83b01c08c7745b02f05328f 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 
> e6c2832003a1bb804f99125866bff6117e4dab73 
>   ql/src/test/results/clientpositive/bucketsortoptimize_insert_7.q.out 
> 326e4d67f00158929f4aaa0a2ec53cedabe3e92c 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b1e43fed3326c8c43e3266ad281c86a054 
>   ql/src/test/results/clientpositive/cbo_rp_join0.q.out 
> 3c6bb7319fd7605a62f082b989f723d5aab6c3fe 
>   ql/src/test/results/clientpositive/cbo_rp_join1.q.out 
> e77002866174f85168859c4145e7016c9d829bb6 
>   

Re: Review Request 41431: HIVE-12674 HS2 Tez session lifetime

[Sergey Shelukhin]

> On Dec. 17, 2015, 10:51 a.m., Vikram Dixit Kumaraswamy wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java, 
> > line 129
> > 
> >
> > expiration queue length is fixed at 11? Shouldn't it be a configurable 
> > value?

that's the initial length


> On Dec. 17, 2015, 10:51 a.m., Vikram Dixit Kumaraswamy wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java, 
> > line 245
> > 
> >
> > Given the assertion below, you should rename this method to 
> > returnSessionToPool or something like that if it is guaranteed to be a pool 
> > session or change the assertion below.

the method is on the pool, so it kind of implies you are returning to the pool 
what you have taken from the pool :)


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41431/#review110852
---


On Dec. 16, 2015, 1:57 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41431/
> ---
> 
> (Updated Dec. 16, 2015, 1:57 a.m.)
> 
> 
> Review request for hive, Siddharth Seth and Vikram Dixit Kumaraswamy.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 243f281 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 971dac9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java f6bc19c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 0d84340 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java e1a8041 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java c066c7a 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SampleTezSessionState.java 
> d55c9fe 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> 11c0325 
> 
> Diff: https://reviews.apache.org/r/41431/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

using the Hive SQL parser in Spark

[Reynold Xin]

Hi Hive devs,

I would like to share with you Spark's plan w.r.t. SQL parser going
forward. As you may (or may not) know, Spark SQL has had two parsers so far:

- a very simple one based on Scala's parser combinator; and
- one that depends on Hive's

The Scala parser combinator one was written quickly so we could parse SQL
queries even when Hive dependency is off. However, it suffers from some
major problems, the most important of which are (1) really bad error
messages and (2) no warning when grammars rules conflict.

We really like the Hive parser. It calls into Hive itself and translates
the generated AST into Spark's logical plans. However, because the grammar
definition was not in Spark, we could not introduce new grammars or fix
bugs when needed.

These two parsers have been a major source of confusions for Spark users,
because depending on which mode Spark SQL is running on, you get subtle
differences in grammar. It has been our intention to replace both of them
with a built-in parser.

We have looked into various options, and it looks like the best option is
to copy the ANTLR grammar file from Hive into Spark. Because the grammar
file is tightly coupled with Hive's semantic analysis, we need to refactor
some code to use them so it will end up becoming the .g file plus some
coupled code. We already have a prototype that somewhat works. I expect we
will get this done in early 2016.


We have also looked into creating an independent library for the SQL parser
that both Hive and Spark share. However, we eventually decided that it
wouldn't make much sense with this approach, because it is a lot of work
for both Hive and Spark to refactor existing code to introduce an external
parser. From Hive's perspective this does not provide any immediate
benefits. From Spark's perspective, we iterate very quickly so having to
depend on an external component also slow down our development. We also
have some requirements that simply don't apply in other projects (e.g.
being able to parse DataFrame expressions).


Thanks a lot for developing this parser, and we will try our best to
contribute back as we fix bugs. I will also make sure we have the proper
acknowledgment when we do this.

Cheers.

- Reynold

Re: Review Request 41457: HIVE-11865

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41457/#review111247
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java (line 110)


In the CBO Success path the additional const prop seems unnecessary. All we 
have done between two const prop is is synthetic join predicates (not null 
predicates) && Simple predicate push down.


- John Pullokkaran


On Dec. 18, 2015, 5:28 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41457/
> ---
> 
> (Updated Dec. 18, 2015, 5:28 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Bugs: HIVE-11865
> https://issues.apache.org/jira/browse/HIVE-11865
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Disable Hive PPD optimizer when CBO has optimized the plan
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out 
> 3907bc946679c0ce2492692dcb7e2cf252c6e188 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 
> 8f48e7df4b9c081528b82545d445f75f67f93bfc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  142812ca703d5e667bd532a711d29594123bde50 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTSTransposeRule.java
>  8321504774c5ec5c7c9de3aecdf78dcd8408c4cf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java
>  7e484b90deed2dfb120faa2699eb9b9b7a4e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java
>  c8de1d89a1df083cd125ae8c11bf854b09915680 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
>  efde6652ca270d8e8d8f682fc0ff260702f46d5a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  feec3c25929cac0bb3b6f50d074557a3c2745889 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 7c5a43ff4060c9a4d9c4c5523d851ef959da3932 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 27549dceac8b5913b1fb1604bde213ae26b243bf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
> ccc4bb454e61e3c39b1f90f29de5e754f8d886bf 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 
> 1702628447d38b09c9b7d0dcf38218a9037a2c64 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SimplePredicatePushDown.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
> accfb3b9f5631f25220c88b759dba6c5299b4044 
>   ql/src/test/results/clientpositive/auto_join12.q.out 
> 6f08aa8f043113f0a99acd10c848f60a122a3f0b 
>   ql/src/test/results/clientpositive/auto_join13.q.out 
> d29818cf438449a6a1413edd7d474c4ab35f2148 
>   ql/src/test/results/clientpositive/auto_join16.q.out 
> 38738ede2b4cd49aa24490eafb81dec1de78bea5 
>   ql/src/test/results/clientpositive/auto_join2.q.out 
> ce40ff30cb9cc2ab734ee37c0f3dc0ae85bbd202 
>   ql/src/test/results/clientpositive/auto_join33.q.out 
> 8b13bd0cb7fd869c58c9d092b2e8710391c96503 
>   ql/src/test/results/clientpositive/auto_join_filters.q.out 
> e0ed3737f9b25d45bdb365b13aeed3a40c50135f 
>   ql/src/test/results/clientpositive/auto_join_nulls.q.out 
> 954bf069760c6f0d7e5420c7e9a3b9756d5aa4df 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out 
> feb818679df7d9f63f2d2d7a7c9afb2d0f5874d8 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 
> e0d0146dd6b86e9f69a467dcde0c020c31af499d 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
> c63016d0a3610eaa6cd82a4adc5d1bf6f0925cce 
>   ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out 
> 096f5d445bc7e61381775270bef04e911eac5bb6 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 
> 30853c42a52e1def5a913352580e0b5391ac2050 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 
> 3e8dbcfd5bcc66ccd83b01c08c7745b02f05328f 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 
> e6c2832003a1bb804f99125866bff6117e4dab73 
>   ql/src/test/results/clientpositive/bucketsortoptimize_insert_7.q.out 
> 326e4d67f00158929f4aaa0a2ec53cedabe3e92c 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b1e43fed3326c8c43e3266ad281c86a054 
>   ql/src/test/results/clientpositive/cbo_rp_join0.q.out 
> 3c6bb7319fd7605a62f082b989f723d5aab6c3fe 
>   ql/src/test/results/clientpositive/cbo_rp_join1.q.out 
> 

Re: Review Request 41457: HIVE-11865

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41457/#review111255
---



ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java (line 167)


This is fragile.
State will have to be maintained; i.e unset if a filter merge happened.


- John Pullokkaran


On Dec. 18, 2015, 5:28 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41457/
> ---
> 
> (Updated Dec. 18, 2015, 5:28 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Bugs: HIVE-11865
> https://issues.apache.org/jira/browse/HIVE-11865
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Disable Hive PPD optimizer when CBO has optimized the plan
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out 
> 3907bc946679c0ce2492692dcb7e2cf252c6e188 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 
> 8f48e7df4b9c081528b82545d445f75f67f93bfc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  142812ca703d5e667bd532a711d29594123bde50 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTSTransposeRule.java
>  8321504774c5ec5c7c9de3aecdf78dcd8408c4cf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java
>  7e484b90deed2dfb120faa2699eb9b9b7a4e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java
>  c8de1d89a1df083cd125ae8c11bf854b09915680 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
>  efde6652ca270d8e8d8f682fc0ff260702f46d5a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  feec3c25929cac0bb3b6f50d074557a3c2745889 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 7c5a43ff4060c9a4d9c4c5523d851ef959da3932 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 27549dceac8b5913b1fb1604bde213ae26b243bf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
> ccc4bb454e61e3c39b1f90f29de5e754f8d886bf 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 
> 1702628447d38b09c9b7d0dcf38218a9037a2c64 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SimplePredicatePushDown.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
> accfb3b9f5631f25220c88b759dba6c5299b4044 
>   ql/src/test/results/clientpositive/auto_join12.q.out 
> 6f08aa8f043113f0a99acd10c848f60a122a3f0b 
>   ql/src/test/results/clientpositive/auto_join13.q.out 
> d29818cf438449a6a1413edd7d474c4ab35f2148 
>   ql/src/test/results/clientpositive/auto_join16.q.out 
> 38738ede2b4cd49aa24490eafb81dec1de78bea5 
>   ql/src/test/results/clientpositive/auto_join2.q.out 
> ce40ff30cb9cc2ab734ee37c0f3dc0ae85bbd202 
>   ql/src/test/results/clientpositive/auto_join33.q.out 
> 8b13bd0cb7fd869c58c9d092b2e8710391c96503 
>   ql/src/test/results/clientpositive/auto_join_filters.q.out 
> e0ed3737f9b25d45bdb365b13aeed3a40c50135f 
>   ql/src/test/results/clientpositive/auto_join_nulls.q.out 
> 954bf069760c6f0d7e5420c7e9a3b9756d5aa4df 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out 
> feb818679df7d9f63f2d2d7a7c9afb2d0f5874d8 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 
> e0d0146dd6b86e9f69a467dcde0c020c31af499d 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
> c63016d0a3610eaa6cd82a4adc5d1bf6f0925cce 
>   ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out 
> 096f5d445bc7e61381775270bef04e911eac5bb6 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 
> 30853c42a52e1def5a913352580e0b5391ac2050 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 
> 3e8dbcfd5bcc66ccd83b01c08c7745b02f05328f 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 
> e6c2832003a1bb804f99125866bff6117e4dab73 
>   ql/src/test/results/clientpositive/bucketsortoptimize_insert_7.q.out 
> 326e4d67f00158929f4aaa0a2ec53cedabe3e92c 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b1e43fed3326c8c43e3266ad281c86a054 
>   ql/src/test/results/clientpositive/cbo_rp_join0.q.out 
> 3c6bb7319fd7605a62f082b989f723d5aab6c3fe 
>   ql/src/test/results/clientpositive/cbo_rp_join1.q.out 
> e77002866174f85168859c4145e7016c9d829bb6 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out 
> 

Re: using the Hive SQL parser in Spark

[Gopal Vijayaraghavan]

>We have looked into various options, and it looks like the best option is
>to copy the ANTLR grammar file from Hive into Spark. Because the grammar
>file is tightly coupled with Hive's semantic analysis, we need to refactor
>some code to use them so it will end up becoming the .g file plus some
>coupled code.

Is the eventual goal to contribute that fork back into Hive & have Hive
devs maintain a compatible parser for SparkSQL?

Would that affect Hive's ability to refactor the SQL parser in the future
or is this a one-time only deal?

>parser. From Hive's perspective this does not provide any immediate
>benefits. From Spark's perspective, we iterate very quickly so having to
>depend on an external component also slow down our development. We also
>have some requirements that simply don't apply in other projects (e.g.
>being able to parse DataFrame expressions).

>From that I assume, this involves some form of cut-paste duplication of
the code into SparkSQL project with that version diverging away from
Hive's.

> Thanks a lot for developing this parser, and we will try our best to
> contribute back as we fix bugs. I will also make sure we have the proper
> acknowledgment when we do this.


Under the Apache license, there's no actual restriction against a hostile
embrace-extend by copying hive's code verbatim as long as the fork retains
license notices.

The maintainability concerns are mostly around whether this is intended as
an ongoing relationship, including any compatibility committments from
hive-dev@.


Cheers,
Gopal

Re: Review Request 41457: HIVE-11865

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41457/#review111248
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
 (line 1)


These seems cosmetic, do we need these?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
 (line 10)


These seems cosmetic, do we need these?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
 (line 1)


These seems cosmetic, do we need these?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
 (line 10)


These seems cosmetic, do we need these?


- John Pullokkaran


On Dec. 18, 2015, 5:28 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41457/
> ---
> 
> (Updated Dec. 18, 2015, 5:28 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Bugs: HIVE-11865
> https://issues.apache.org/jira/browse/HIVE-11865
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Disable Hive PPD optimizer when CBO has optimized the plan
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out 
> 3907bc946679c0ce2492692dcb7e2cf252c6e188 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 
> 8f48e7df4b9c081528b82545d445f75f67f93bfc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  142812ca703d5e667bd532a711d29594123bde50 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTSTransposeRule.java
>  8321504774c5ec5c7c9de3aecdf78dcd8408c4cf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java
>  7e484b90deed2dfb120faa2699eb9b9b7a4e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java
>  c8de1d89a1df083cd125ae8c11bf854b09915680 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
>  efde6652ca270d8e8d8f682fc0ff260702f46d5a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  feec3c25929cac0bb3b6f50d074557a3c2745889 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 7c5a43ff4060c9a4d9c4c5523d851ef959da3932 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 27549dceac8b5913b1fb1604bde213ae26b243bf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
> ccc4bb454e61e3c39b1f90f29de5e754f8d886bf 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 
> 1702628447d38b09c9b7d0dcf38218a9037a2c64 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SimplePredicatePushDown.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
> accfb3b9f5631f25220c88b759dba6c5299b4044 
>   ql/src/test/results/clientpositive/auto_join12.q.out 
> 6f08aa8f043113f0a99acd10c848f60a122a3f0b 
>   ql/src/test/results/clientpositive/auto_join13.q.out 
> d29818cf438449a6a1413edd7d474c4ab35f2148 
>   ql/src/test/results/clientpositive/auto_join16.q.out 
> 38738ede2b4cd49aa24490eafb81dec1de78bea5 
>   ql/src/test/results/clientpositive/auto_join2.q.out 
> ce40ff30cb9cc2ab734ee37c0f3dc0ae85bbd202 
>   ql/src/test/results/clientpositive/auto_join33.q.out 
> 8b13bd0cb7fd869c58c9d092b2e8710391c96503 
>   ql/src/test/results/clientpositive/auto_join_filters.q.out 
> e0ed3737f9b25d45bdb365b13aeed3a40c50135f 
>   ql/src/test/results/clientpositive/auto_join_nulls.q.out 
> 954bf069760c6f0d7e5420c7e9a3b9756d5aa4df 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out 
> feb818679df7d9f63f2d2d7a7c9afb2d0f5874d8 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 
> e0d0146dd6b86e9f69a467dcde0c020c31af499d 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
> c63016d0a3610eaa6cd82a4adc5d1bf6f0925cce 
>   ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out 
> 096f5d445bc7e61381775270bef04e911eac5bb6 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 
> 30853c42a52e1def5a913352580e0b5391ac2050 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 
> 3e8dbcfd5bcc66ccd83b01c08c7745b02f05328f 
>   

[jira] [Created] (HIVE-12712) HiveInputFormat may fail to column names to read in some cases

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-12712:


 Summary: HiveInputFormat may fail to column names to read in some 
cases
 Key: HIVE-12712
 URL: https://issues.apache.org/jira/browse/HIVE-12712
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.0.0, 2.1.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


The primary issue is when plan is generated pathToAliases map is populated with 
directory paths to table aliases. pathToAliases.put() uses path.toString() as 
map key. During probing, path.toUri().toString() is used. This can cause probe 
misses when path contains spaces in them. path.toUri() will escape the spaces 
in the path whereas path.toString() does not escape the spaces. As a result, 
HiveInputFormat can trigger a different code path which can fail to set list of 
columns to read from the source table. This was causing unexpected NPE in 
OrcInputFormat (after refactoring HIVE-11705) which removed null check for 
column names. The resulting exception is 

{code}
Caused by: java.lang.RuntimeException: ORC split generation failed with 
exception: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1288)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1354)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:367)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:457)
at 
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:152)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:246)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:240)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:240)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:227)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.util.concurrent.ExecutionException: 
java.lang.NullPointerException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1282)
... 15 more
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.extractNeededColNames(OrcInputFormat.java:422)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.extractNeededColNames(OrcInputFormat.java:417)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.access$2000(OrcInputFormat.java:134)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1072)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:919)
... 4 more

{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41431: HIVE-12674 HS2 Tez session lifetime

[Sergey Shelukhin]

> On Dec. 17, 2015, 10:51 a.m., Vikram Dixit Kumaraswamy wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java, 
> > line 467
> > 
> >
> > Can you raise a jira for follow up on this?

fixed the issue


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41431/#review110852
---


On Dec. 16, 2015, 1:57 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41431/
> ---
> 
> (Updated Dec. 16, 2015, 1:57 a.m.)
> 
> 
> Review request for hive, Siddharth Seth and Vikram Dixit Kumaraswamy.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 243f281 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 971dac9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java f6bc19c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 0d84340 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java e1a8041 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java c066c7a 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SampleTezSessionState.java 
> d55c9fe 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> 11c0325 
> 
> Diff: https://reviews.apache.org/r/41431/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 41535: HIVE-12632 LLAP: don't use IO elevator for ACID tables

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41535/#review111323
---

Ship it!


Ship It!

- Prasanth_J


On Dec. 18, 2015, 9:40 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41535/
> ---
> 
> (Updated Dec. 18, 2015, 9:40 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
>  b57366c 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ColumnVectorProducer.java
>  5c34fa3 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
>  259c483 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  5957d57 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 24137e2 
>   ql/src/java/org/apache/hadoop/hive/ql/io/LlapAwareSplit.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 61cde41 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 27549dc 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java edafdb1 
>   ql/src/test/queries/clientpositive/llap_acid.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap_acid.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/41535/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 41377: HIVE-12528 don't start HS2 Tez sessions in a single thread

[Siddharth Seth]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41377/#review111342
---


TezSessionState - does that need to be synchronized on various methods like 
openSession, getSession, closeSession. 
To me, it seems like access to these variables is not thread safe.

Even something like TezSessionState.queueName. That's setup in the main thread 
in TezSessionPoolManager.setupSessionPool. Then it is accessed in separate 
threads - without being a final field, or protected by synchronzied blocks.


ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java (line 974)


General: Some docs mentioning that this method is meant to work across 
threads and processes, and how that complicates the notifiers / sleep.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java (line 987)


Maybe a higher interval or #of attempts, so that most files in the same JVM 
don't go into the IOException path.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java (line 1025)


Seems like this is chceked far too often. Required here. Wonder if it can 
be removed further up.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java (line 1027)


Name of the file in the log will be useful



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java (line 1032)


waitAttempts isn't actually used. Drop from the api, or use instead of the 
sleep for loop in the exception block.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java (line 
88)


Why is this required ?

and why not in the single thread case.



ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java (line 
118)


Is there any part of the test which verifies that threads are being used / 
sessions are being setup in parallel. Not sure if there's a good way to verify 
that though - without setting up explicit properties for tests only.


- Siddharth Seth


On Dec. 18, 2015, 10:10 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41377/
> ---
> 
> (Updated Dec. 18, 2015, 10:10 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 36e281a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 6e196e6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 0d84340 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java e1a8041 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> 11c0325 
> 
> Diff: https://reviews.apache.org/r/41377/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-12711) Document howto disable web ui in config of hive.server2.webui.port

[Thejas M Nair (JIRA)]

Thejas M Nair created HIVE-12711:


 Summary: Document howto disable web ui in config of 
hive.server2.webui.port
 Key: HIVE-12711
 URL: https://issues.apache.org/jira/browse/HIVE-12711
 Project: Hive
  Issue Type: Sub-task
  Components: HiveServer2
Reporter: Thejas M Nair
Assignee: Thejas M Nair
 Attachments: HIVE-12711.1.patch

hive.server2.webui.port config does not say that it can be used to disable 
webui as well.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 40589: Implement limit push down through union all in CBO

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/#review111311
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
 (line 84)


The real value of pushing limit past union is to push it further down (by 
pushing it past sel, join).

If the sortrel ended up just below union & if sortrel also had sort in it 
(i.e if union's input is synthetic limit) then i think we should remove it.

May be we should enhance HiveSortRemoveRule to look for pattern 
SortRel-union-SortRel and remove synthetic SortRel below union if it has sort 
in it.

Otherwise we end up paying double sorting cost.


- John Pullokkaran


On Dec. 17, 2015, 10:47 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40589/
> ---
> 
> (Updated Dec. 17, 2015, 10:47 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11775
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 243f281 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 36a12bf 
>   ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/limit_join_transpose.q 80430c6 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q cecbbd7 
>   ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40589/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

[jira] [Created] (HIVE-12710) add better logging for Tez session creation thread failures

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-12710:
---

 Summary: add better logging for Tez session creation thread 
failures
 Key: HIVE-12710
 URL: https://issues.apache.org/jira/browse/HIVE-12710
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review111316
---



jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 314)


Nit: LOG.error has an overload to log an exception, LOG.error(, ex);


- Sergey Shelukhin


On Dec. 18, 2015, 7:13 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated Dec. 18, 2015, 7:13 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b5aee00 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 245c6a3 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 3f9024a 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 3d5f3b5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
>   service/if/TCLIService.thrift baf583f 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
>  0b9aa0f 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
>  c936ada 
>   service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote e167d5b 
>   service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/java/org/apache/hive/service/cli/CLIService.java adc9809 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> d2b3f9c 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 92135cd 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> d90dd0d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 50c912e 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8434965 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Re: Review Request 40865: Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40865/
---

(Updated Dec. 19, 2015, 12:28 a.m.)


Review request for hive and John Pullokkaran.


Changes
---

address john's comments. rebase for relfactories.


Repository: hive-git


Description
---

HIVE-11927


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
 336745b 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
 425514d 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
 3f2267d 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
 631a4ca 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
 2825f77 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7c5a43f 
  ql/src/test/queries/clientpositive/cbo_const.q PRE-CREATION 
  ql/src/test/queries/clientpositive/constantfolding.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out c4d59c8 
  ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 277b0f7 
  ql/src/test/results/clientpositive/cast1.q.out 0bdecba 
  ql/src/test/results/clientpositive/cbo_const.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 62b611b 
  ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out b14caa8 
  ql/src/test/results/clientpositive/constantfolding.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/cross_product_check_1.q.out e7d6900 
  ql/src/test/results/clientpositive/cross_product_check_2.q.out df438c9 
  ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out b85d387 
  ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 24ac550 
  ql/src/test/results/clientpositive/groupby_ppd.q.out 6164a26 
  ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 7333677 
  ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out e19d1de 
  ql/src/test/results/clientpositive/input_part1.q.out d6f4d3e 
  ql/src/test/results/clientpositive/input_part5.q.out f2d7335 
  ql/src/test/results/clientpositive/input_part6.q.out fa51cdf 
  ql/src/test/results/clientpositive/lineage2.q.out ec8b76b 
  ql/src/test/results/clientpositive/lineage3.q.out 747dc9a 
  ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out be77ba8 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 79348f3 
  ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out f6d8388 
  ql/src/test/results/clientpositive/parquet_predicate_pushdown.q.out b322ef1 
  ql/src/test/results/clientpositive/partition_multilevels.q.out c1c8778 
  ql/src/test/results/clientpositive/perf/query31.q.out c479d91 
  ql/src/test/results/clientpositive/perf/query39.q.out 0ad62cc 
  ql/src/test/results/clientpositive/perf/query42.q.out a3264c7 
  ql/src/test/results/clientpositive/perf/query52.q.out ac0c1e6 
  ql/src/test/results/clientpositive/perf/query64.q.out 9735b45 
  ql/src/test/results/clientpositive/perf/query66.q.out 42bcb83 
  ql/src/test/results/clientpositive/perf/query75.q.out 70a1649 
  ql/src/test/results/clientpositive/pointlookup2.q.out a442425 
  ql/src/test/results/clientpositive/quotedid_basic.q.out 519f647 
  ql/src/test/results/clientpositive/quotedid_partition.q.out d34a005 
  ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out b5e7846 
  ql/src/test/results/clientpositive/spark/cross_product_check_1.q.out 5b03dcf 
  ql/src/test/results/clientpositive/spark/cross_product_check_2.q.out 93c502d 
  ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out b597ebd 
  ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 239e803 
  ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 8370bbe 
  ql/src/test/results/clientpositive/spark/union_remove_25.q.out b771fe9 
  ql/src/test/results/clientpositive/spark/union_view.q.out cce7710 
  ql/src/test/results/clientpositive/subquery_notin.q.out ed86079 
  ql/src/test/results/clientpositive/subquery_notin_having.q.java1.7.out 
7b7ccda 
  ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out 3241787 
  ql/src/test/results/clientpositive/tez/bucket_map_join_tez1.q.out 9582334 
  ql/src/test/results/clientpositive/tez/cross_product_check_1.q.out 0e7c681 
  ql/src/test/results/clientpositive/tez/cross_product_check_2.q.out efd8b5d 
  ql/src/test/results/clientpositive/tez/dynpart_sort_optimization2.q.out 
346e52f 
  ql/src/test/results/clientpositive/tez/explainuser_1.q.out c081309 
  

Re: Review Request 41431: HIVE-12674 HS2 Tez session lifetime

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41431/
---

(Updated Dec. 19, 2015, 12:59 a.m.)


Review request for hive, Siddharth Seth and Vikram Dixit Kumaraswamy.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 60ac0c0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 971dac9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java f6bc19c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
0d84340 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java e1a8041 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java c066c7a 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SampleTezSessionState.java 
d55c9fe 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
11c0325 

Diff: https://reviews.apache.org/r/41431/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 41431: HIVE-12674 HS2 Tez session lifetime

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41431/#review111327
---



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java (line 
456)


note: probably need min(0, ...)


- Sergey Shelukhin


On Dec. 19, 2015, 12:59 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41431/
> ---
> 
> (Updated Dec. 19, 2015, 12:59 a.m.)
> 
> 
> Review request for hive, Siddharth Seth and Vikram Dixit Kumaraswamy.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 60ac0c0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 971dac9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java f6bc19c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 0d84340 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java e1a8041 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java c066c7a 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SampleTezSessionState.java 
> d55c9fe 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> 11c0325 
> 
> Diff: https://reviews.apache.org/r/41431/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 41431: HIVE-12674 HS2 Tez session lifetime

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41431/
---

(Updated Dec. 19, 2015, 1:03 a.m.)


Review request for hive, Siddharth Seth and Vikram Dixit Kumaraswamy.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 60ac0c0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 971dac9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java f6bc19c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
0d84340 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java e1a8041 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java c066c7a 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SampleTezSessionState.java 
d55c9fe 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
11c0325 

Diff: https://reviews.apache.org/r/41431/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-12709) further improve user level explain

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-12709:
--

 Summary: further improve user level explain
 Key: HIVE-12709
 URL: https://issues.apache.org/jira/browse/HIVE-12709
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong


Need to address more feedbacks from Hive users for the user level explain: (1) 
Put stats in the same line as operator; (2) Avoid stats on *Sink; (3) Avoid col 
types; (4) TS should list pruned col names; (5) TS should list fully qualified 
table name, along with alias; etc 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41457: HIVE-11865

[Jesús Camacho Rodríguez]

> On Dec. 18, 2015, 7:59 p.m., John Pullokkaran wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java,
> >  line 1
> > 
> >
> > These seems cosmetic, do we need these?

These are all cosmetic, I changed them by mistake. They will not be changed in 
the final version.


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41457/#review111248
---


On Dec. 18, 2015, 5:28 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41457/
> ---
> 
> (Updated Dec. 18, 2015, 5:28 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Bugs: HIVE-11865
> https://issues.apache.org/jira/browse/HIVE-11865
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Disable Hive PPD optimizer when CBO has optimized the plan
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out 
> 3907bc946679c0ce2492692dcb7e2cf252c6e188 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 
> 8f48e7df4b9c081528b82545d445f75f67f93bfc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  142812ca703d5e667bd532a711d29594123bde50 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTSTransposeRule.java
>  8321504774c5ec5c7c9de3aecdf78dcd8408c4cf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java
>  7e484b90deed2dfb120faa2699eb9b9b7a4e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java
>  c8de1d89a1df083cd125ae8c11bf854b09915680 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
>  efde6652ca270d8e8d8f682fc0ff260702f46d5a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  feec3c25929cac0bb3b6f50d074557a3c2745889 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 7c5a43ff4060c9a4d9c4c5523d851ef959da3932 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 27549dceac8b5913b1fb1604bde213ae26b243bf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
> ccc4bb454e61e3c39b1f90f29de5e754f8d886bf 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 
> 1702628447d38b09c9b7d0dcf38218a9037a2c64 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SimplePredicatePushDown.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
> accfb3b9f5631f25220c88b759dba6c5299b4044 
>   ql/src/test/results/clientpositive/auto_join12.q.out 
> 6f08aa8f043113f0a99acd10c848f60a122a3f0b 
>   ql/src/test/results/clientpositive/auto_join13.q.out 
> d29818cf438449a6a1413edd7d474c4ab35f2148 
>   ql/src/test/results/clientpositive/auto_join16.q.out 
> 38738ede2b4cd49aa24490eafb81dec1de78bea5 
>   ql/src/test/results/clientpositive/auto_join2.q.out 
> ce40ff30cb9cc2ab734ee37c0f3dc0ae85bbd202 
>   ql/src/test/results/clientpositive/auto_join33.q.out 
> 8b13bd0cb7fd869c58c9d092b2e8710391c96503 
>   ql/src/test/results/clientpositive/auto_join_filters.q.out 
> e0ed3737f9b25d45bdb365b13aeed3a40c50135f 
>   ql/src/test/results/clientpositive/auto_join_nulls.q.out 
> 954bf069760c6f0d7e5420c7e9a3b9756d5aa4df 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out 
> feb818679df7d9f63f2d2d7a7c9afb2d0f5874d8 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 
> e0d0146dd6b86e9f69a467dcde0c020c31af499d 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
> c63016d0a3610eaa6cd82a4adc5d1bf6f0925cce 
>   ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out 
> 096f5d445bc7e61381775270bef04e911eac5bb6 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 
> 30853c42a52e1def5a913352580e0b5391ac2050 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 
> 3e8dbcfd5bcc66ccd83b01c08c7745b02f05328f 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 
> e6c2832003a1bb804f99125866bff6117e4dab73 
>   ql/src/test/results/clientpositive/bucketsortoptimize_insert_7.q.out 
> 326e4d67f00158929f4aaa0a2ec53cedabe3e92c 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b1e43fed3326c8c43e3266ad281c86a054 
>   ql/src/test/results/clientpositive/cbo_rp_join0.q.out 
> 3c6bb7319fd7605a62f082b989f723d5aab6c3fe 
>   

Re: Review Request 41457: HIVE-11865

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41457/#review111288
---



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 491)


I did not create the variable, I just created an access method i.e. the 
variable was already there. It is only checked at SemanticAnalyzer to know 
whether it should introduce isnotnull filters below Join or not.


- Jesús Camacho Rodríguez


On Dec. 18, 2015, 5:28 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41457/
> ---
> 
> (Updated Dec. 18, 2015, 5:28 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Bugs: HIVE-11865
> https://issues.apache.org/jira/browse/HIVE-11865
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Disable Hive PPD optimizer when CBO has optimized the plan
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out 
> 3907bc946679c0ce2492692dcb7e2cf252c6e188 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 
> 8f48e7df4b9c081528b82545d445f75f67f93bfc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  142812ca703d5e667bd532a711d29594123bde50 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTSTransposeRule.java
>  8321504774c5ec5c7c9de3aecdf78dcd8408c4cf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java
>  7e484b90deed2dfb120faa2699eb9b9b7a4e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java
>  c8de1d89a1df083cd125ae8c11bf854b09915680 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
>  efde6652ca270d8e8d8f682fc0ff260702f46d5a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  feec3c25929cac0bb3b6f50d074557a3c2745889 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 7c5a43ff4060c9a4d9c4c5523d851ef959da3932 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 27549dceac8b5913b1fb1604bde213ae26b243bf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
> ccc4bb454e61e3c39b1f90f29de5e754f8d886bf 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 
> 1702628447d38b09c9b7d0dcf38218a9037a2c64 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SimplePredicatePushDown.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
> accfb3b9f5631f25220c88b759dba6c5299b4044 
>   ql/src/test/results/clientpositive/auto_join12.q.out 
> 6f08aa8f043113f0a99acd10c848f60a122a3f0b 
>   ql/src/test/results/clientpositive/auto_join13.q.out 
> d29818cf438449a6a1413edd7d474c4ab35f2148 
>   ql/src/test/results/clientpositive/auto_join16.q.out 
> 38738ede2b4cd49aa24490eafb81dec1de78bea5 
>   ql/src/test/results/clientpositive/auto_join2.q.out 
> ce40ff30cb9cc2ab734ee37c0f3dc0ae85bbd202 
>   ql/src/test/results/clientpositive/auto_join33.q.out 
> 8b13bd0cb7fd869c58c9d092b2e8710391c96503 
>   ql/src/test/results/clientpositive/auto_join_filters.q.out 
> e0ed3737f9b25d45bdb365b13aeed3a40c50135f 
>   ql/src/test/results/clientpositive/auto_join_nulls.q.out 
> 954bf069760c6f0d7e5420c7e9a3b9756d5aa4df 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out 
> feb818679df7d9f63f2d2d7a7c9afb2d0f5874d8 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 
> e0d0146dd6b86e9f69a467dcde0c020c31af499d 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
> c63016d0a3610eaa6cd82a4adc5d1bf6f0925cce 
>   ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out 
> 096f5d445bc7e61381775270bef04e911eac5bb6 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 
> 30853c42a52e1def5a913352580e0b5391ac2050 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 
> 3e8dbcfd5bcc66ccd83b01c08c7745b02f05328f 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 
> e6c2832003a1bb804f99125866bff6117e4dab73 
>   ql/src/test/results/clientpositive/bucketsortoptimize_insert_7.q.out 
> 326e4d67f00158929f4aaa0a2ec53cedabe3e92c 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b1e43fed3326c8c43e3266ad281c86a054 
>   ql/src/test/results/clientpositive/cbo_rp_join0.q.out 
> 3c6bb7319fd7605a62f082b989f723d5aab6c3fe 
>   ql/src/test/results/clientpositive/cbo_rp_join1.q.out 

Re: Review Request 41457: HIVE-11865

[Jesús Camacho Rodríguez]

> On Dec. 18, 2015, 8:09 p.m., John Pullokkaran wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java, line 492
> > 
> >
> > This is fragile; i.e reusing the flag whether to run CBO or not for 
> > whether CBO was actually run successfully.
> > 
> > We already keep state in parsecontext about this.
> 
> Jesús Camacho Rodríguez wrote:
> I did not create the variable, I just created an access method i.e. the 
> variable was already there. It is only checked at SemanticAnalyzer to know 
> whether it should introduce isnotnull filters below Join or not.

In addition, I meant to say that variable is set to false if CBO fails (line 
330 in CalcitePlanner: runCBO = false;)


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41457/#review111254
---


On Dec. 18, 2015, 10:19 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41457/
> ---
> 
> (Updated Dec. 18, 2015, 10:19 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Bugs: HIVE-11865
> https://issues.apache.org/jira/browse/HIVE-11865
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Disable Hive PPD optimizer when CBO has optimized the plan
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out 
> 3907bc946679c0ce2492692dcb7e2cf252c6e188 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 
> 8f48e7df4b9c081528b82545d445f75f67f93bfc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  142812ca703d5e667bd532a711d29594123bde50 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTSTransposeRule.java
>  8321504774c5ec5c7c9de3aecdf78dcd8408c4cf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java
>  7e484b90deed2dfb120faa2699eb9b9b7a4e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java
>  c8de1d89a1df083cd125ae8c11bf854b09915680 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 7c5a43ff4060c9a4d9c4c5523d851ef959da3932 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 27549dceac8b5913b1fb1604bde213ae26b243bf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
> ccc4bb454e61e3c39b1f90f29de5e754f8d886bf 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 
> 1702628447d38b09c9b7d0dcf38218a9037a2c64 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SimplePredicatePushDown.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
> accfb3b9f5631f25220c88b759dba6c5299b4044 
>   ql/src/test/results/clientpositive/auto_join12.q.out 
> 6f08aa8f043113f0a99acd10c848f60a122a3f0b 
>   ql/src/test/results/clientpositive/auto_join13.q.out 
> d29818cf438449a6a1413edd7d474c4ab35f2148 
>   ql/src/test/results/clientpositive/auto_join16.q.out 
> 38738ede2b4cd49aa24490eafb81dec1de78bea5 
>   ql/src/test/results/clientpositive/auto_join2.q.out 
> ce40ff30cb9cc2ab734ee37c0f3dc0ae85bbd202 
>   ql/src/test/results/clientpositive/auto_join33.q.out 
> 8b13bd0cb7fd869c58c9d092b2e8710391c96503 
>   ql/src/test/results/clientpositive/auto_join_filters.q.out 
> e0ed3737f9b25d45bdb365b13aeed3a40c50135f 
>   ql/src/test/results/clientpositive/auto_join_nulls.q.out 
> 954bf069760c6f0d7e5420c7e9a3b9756d5aa4df 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out 
> feb818679df7d9f63f2d2d7a7c9afb2d0f5874d8 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 
> e0d0146dd6b86e9f69a467dcde0c020c31af499d 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
> c63016d0a3610eaa6cd82a4adc5d1bf6f0925cce 
>   ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out 
> 096f5d445bc7e61381775270bef04e911eac5bb6 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 
> 30853c42a52e1def5a913352580e0b5391ac2050 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 
> 3e8dbcfd5bcc66ccd83b01c08c7745b02f05328f 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 
> e6c2832003a1bb804f99125866bff6117e4dab73 
>   ql/src/test/results/clientpositive/bucketsortoptimize_insert_7.q.out 
> 326e4d67f00158929f4aaa0a2ec53cedabe3e92c 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b1e43fed3326c8c43e3266ad281c86a054 
>   

Re: using the Hive SQL parser in Spark

[Reynold Xin]

(Please use reply-all so I see the replies)

Responses inline.


On Fri, Dec 18, 2015 at 1:17 PM, Yin Huai  wrote:

> Let me add Reynold to the thread.
>
> On Fri, Dec 18, 2015 at 12:36 PM, Gopal Vijayaraghavan 
> wrote:
>
>>
>> >We have looked into various options, and it looks like the best option is
>> >to copy the ANTLR grammar file from Hive into Spark. Because the grammar
>> >file is tightly coupled with Hive's semantic analysis, we need to
>> refactor
>> >some code to use them so it will end up becoming the .g file plus some
>> >coupled code.
>>
>> Is the eventual goal to contribute that fork back into Hive & have Hive
>> devs maintain a compatible parser for SparkSQL?
>>
>> Would that affect Hive's ability to refactor the SQL parser in the future
>> or is this a one-time only deal?
>
>
I am not sure if it is useful at all to port that back to Hive since it has
zero user facing benefit, and would require Hive devs to spend a lot of
time reviewing the changes. Refactoring like this is always risky for an
established project.


>
>>
>> >parser. From Hive's perspective this does not provide any immediate
>> >benefits. From Spark's perspective, we iterate very quickly so having to
>> >depend on an external component also slow down our development. We also
>> >have some requirements that simply don't apply in other projects (e.g.
>> >being able to parse DataFrame expressions).
>>
>> From that I assume, this involves some form of cut-paste duplication of
>> the code into SparkSQL project with that version diverging away from
>> Hive's.
>
>
That is correct.


>
>>
>> > Thanks a lot for developing this parser, and we will try our best to
>> > contribute back as we fix bugs. I will also make sure we have the proper
>> > acknowledgment when we do this.
>>
>>
>> Under the Apache license, there's no actual restriction against a hostile
>> embrace-extend by copying hive's code verbatim as long as the fork retains
>> license notices.
>>
>> The maintainability concerns are mostly around whether this is intended as
>> an ongoing relationship, including any compatibility committments from
>> hive-dev@.
>>
>
No commitments needed from Hive. You should update/improve the parser as
you see fit. We do have a pretty comprehensive suite of Hive compatibility
tests (by using the Hive tests directly) to ensure SQL compatibility with
Hive. We will continue running those. We will also try our best to
contribute back bug fixes to the parser.

Re: Review Request 41457: HIVE-11865

[Jesús Camacho Rodríguez]

> On Dec. 18, 2015, 7:52 p.m., John Pullokkaran wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java, line 112
> > 
> >
> > In the CBO Success path the additional const prop seems unnecessary. 
> > All we have done between two const prop is is synthetic join predicates 
> > (not null predicates) && Simple predicate push down.

You are right; the latest version of the patch addresses this issue.


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41457/#review111247
---


On Dec. 18, 2015, 5:28 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41457/
> ---
> 
> (Updated Dec. 18, 2015, 5:28 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Bugs: HIVE-11865
> https://issues.apache.org/jira/browse/HIVE-11865
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Disable Hive PPD optimizer when CBO has optimized the plan
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out 
> 3907bc946679c0ce2492692dcb7e2cf252c6e188 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 
> 8f48e7df4b9c081528b82545d445f75f67f93bfc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  142812ca703d5e667bd532a711d29594123bde50 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTSTransposeRule.java
>  8321504774c5ec5c7c9de3aecdf78dcd8408c4cf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java
>  7e484b90deed2dfb120faa2699eb9b9b7a4e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java
>  c8de1d89a1df083cd125ae8c11bf854b09915680 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
>  efde6652ca270d8e8d8f682fc0ff260702f46d5a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  feec3c25929cac0bb3b6f50d074557a3c2745889 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 7c5a43ff4060c9a4d9c4c5523d851ef959da3932 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 27549dceac8b5913b1fb1604bde213ae26b243bf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
> ccc4bb454e61e3c39b1f90f29de5e754f8d886bf 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 
> 1702628447d38b09c9b7d0dcf38218a9037a2c64 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SimplePredicatePushDown.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
> accfb3b9f5631f25220c88b759dba6c5299b4044 
>   ql/src/test/results/clientpositive/auto_join12.q.out 
> 6f08aa8f043113f0a99acd10c848f60a122a3f0b 
>   ql/src/test/results/clientpositive/auto_join13.q.out 
> d29818cf438449a6a1413edd7d474c4ab35f2148 
>   ql/src/test/results/clientpositive/auto_join16.q.out 
> 38738ede2b4cd49aa24490eafb81dec1de78bea5 
>   ql/src/test/results/clientpositive/auto_join2.q.out 
> ce40ff30cb9cc2ab734ee37c0f3dc0ae85bbd202 
>   ql/src/test/results/clientpositive/auto_join33.q.out 
> 8b13bd0cb7fd869c58c9d092b2e8710391c96503 
>   ql/src/test/results/clientpositive/auto_join_filters.q.out 
> e0ed3737f9b25d45bdb365b13aeed3a40c50135f 
>   ql/src/test/results/clientpositive/auto_join_nulls.q.out 
> 954bf069760c6f0d7e5420c7e9a3b9756d5aa4df 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out 
> feb818679df7d9f63f2d2d7a7c9afb2d0f5874d8 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 
> e0d0146dd6b86e9f69a467dcde0c020c31af499d 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
> c63016d0a3610eaa6cd82a4adc5d1bf6f0925cce 
>   ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out 
> 096f5d445bc7e61381775270bef04e911eac5bb6 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 
> 30853c42a52e1def5a913352580e0b5391ac2050 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 
> 3e8dbcfd5bcc66ccd83b01c08c7745b02f05328f 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 
> e6c2832003a1bb804f99125866bff6117e4dab73 
>   ql/src/test/results/clientpositive/bucketsortoptimize_insert_7.q.out 
> 326e4d67f00158929f4aaa0a2ec53cedabe3e92c 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b1e43fed3326c8c43e3266ad281c86a054 
>   

Re: Review Request 41377: HIVE-12528 don't start HS2 Tez sessions in a single thread

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41377/
---

(Updated Dec. 18, 2015, 10:10 p.m.)


Review request for hive, Gunther Hagleitner and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 36e281a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 6e196e6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
0d84340 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java e1a8041 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
11c0325 

Diff: https://reviews.apache.org/r/41377/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 41457: HIVE-11865

[Jesús Camacho Rodríguez]

> On Dec. 18, 2015, 8:09 p.m., John Pullokkaran wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java, line 492
> > 
> >
> > This is fragile; i.e reusing the flag whether to run CBO or not for 
> > whether CBO was actually run successfully.
> > 
> > We already keep state in parsecontext about this.

I did not create the variable, I just created an access method i.e. the 
variable was already there. It is only checked at SemanticAnalyzer to know 
whether it should introduce isnotnull filters below Join or not.


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41457/#review111254
---


On Dec. 18, 2015, 5:28 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41457/
> ---
> 
> (Updated Dec. 18, 2015, 5:28 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Bugs: HIVE-11865
> https://issues.apache.org/jira/browse/HIVE-11865
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Disable Hive PPD optimizer when CBO has optimized the plan
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out 
> 3907bc946679c0ce2492692dcb7e2cf252c6e188 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 
> 8f48e7df4b9c081528b82545d445f75f67f93bfc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  142812ca703d5e667bd532a711d29594123bde50 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTSTransposeRule.java
>  8321504774c5ec5c7c9de3aecdf78dcd8408c4cf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java
>  7e484b90deed2dfb120faa2699eb9b9b7a4e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java
>  c8de1d89a1df083cd125ae8c11bf854b09915680 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
>  efde6652ca270d8e8d8f682fc0ff260702f46d5a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  feec3c25929cac0bb3b6f50d074557a3c2745889 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 7c5a43ff4060c9a4d9c4c5523d851ef959da3932 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 27549dceac8b5913b1fb1604bde213ae26b243bf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
> ccc4bb454e61e3c39b1f90f29de5e754f8d886bf 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 
> 1702628447d38b09c9b7d0dcf38218a9037a2c64 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SimplePredicatePushDown.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
> accfb3b9f5631f25220c88b759dba6c5299b4044 
>   ql/src/test/results/clientpositive/auto_join12.q.out 
> 6f08aa8f043113f0a99acd10c848f60a122a3f0b 
>   ql/src/test/results/clientpositive/auto_join13.q.out 
> d29818cf438449a6a1413edd7d474c4ab35f2148 
>   ql/src/test/results/clientpositive/auto_join16.q.out 
> 38738ede2b4cd49aa24490eafb81dec1de78bea5 
>   ql/src/test/results/clientpositive/auto_join2.q.out 
> ce40ff30cb9cc2ab734ee37c0f3dc0ae85bbd202 
>   ql/src/test/results/clientpositive/auto_join33.q.out 
> 8b13bd0cb7fd869c58c9d092b2e8710391c96503 
>   ql/src/test/results/clientpositive/auto_join_filters.q.out 
> e0ed3737f9b25d45bdb365b13aeed3a40c50135f 
>   ql/src/test/results/clientpositive/auto_join_nulls.q.out 
> 954bf069760c6f0d7e5420c7e9a3b9756d5aa4df 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out 
> feb818679df7d9f63f2d2d7a7c9afb2d0f5874d8 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 
> e0d0146dd6b86e9f69a467dcde0c020c31af499d 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
> c63016d0a3610eaa6cd82a4adc5d1bf6f0925cce 
>   ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out 
> 096f5d445bc7e61381775270bef04e911eac5bb6 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 
> 30853c42a52e1def5a913352580e0b5391ac2050 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 
> 3e8dbcfd5bcc66ccd83b01c08c7745b02f05328f 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 
> e6c2832003a1bb804f99125866bff6117e4dab73 
>   ql/src/test/results/clientpositive/bucketsortoptimize_insert_7.q.out 
> 326e4d67f00158929f4aaa0a2ec53cedabe3e92c 
>   

Re: using the Hive SQL parser in Spark

[Yin Huai]

Let me add Reynold to the thread.

On Fri, Dec 18, 2015 at 12:36 PM, Gopal Vijayaraghavan 
wrote:

>
> >We have looked into various options, and it looks like the best option is
> >to copy the ANTLR grammar file from Hive into Spark. Because the grammar
> >file is tightly coupled with Hive's semantic analysis, we need to refactor
> >some code to use them so it will end up becoming the .g file plus some
> >coupled code.
>
> Is the eventual goal to contribute that fork back into Hive & have Hive
> devs maintain a compatible parser for SparkSQL?
>
> Would that affect Hive's ability to refactor the SQL parser in the future
> or is this a one-time only deal?
>
> >parser. From Hive's perspective this does not provide any immediate
> >benefits. From Spark's perspective, we iterate very quickly so having to
> >depend on an external component also slow down our development. We also
> >have some requirements that simply don't apply in other projects (e.g.
> >being able to parse DataFrame expressions).
>
> From that I assume, this involves some form of cut-paste duplication of
> the code into SparkSQL project with that version diverging away from
> Hive's.
>
> > Thanks a lot for developing this parser, and we will try our best to
> > contribute back as we fix bugs. I will also make sure we have the proper
> > acknowledgment when we do this.
>
>
> Under the Apache license, there's no actual restriction against a hostile
> embrace-extend by copying hive's code verbatim as long as the fork retains
> license notices.
>
> The maintainability concerns are mostly around whether this is intended as
> an ongoing relationship, including any compatibility committments from
> hive-dev@.
>
>
> Cheers,
> Gopal
>
>
>

Re: Review Request 41535: HIVE-12632 LLAP: don't use IO elevator for ACID tables

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41535/
---

(Updated Dec. 18, 2015, 9:40 p.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
 b57366c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ColumnVectorProducer.java
 5c34fa3 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
 259c483 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 5957d57 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 24137e2 
  ql/src/java/org/apache/hadoop/hive/ql/io/LlapAwareSplit.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 61cde41 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 27549dc 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java edafdb1 
  ql/src/test/queries/clientpositive/llap_acid.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap_acid.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/41535/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 41457: HIVE-11865

[Jesús Camacho Rodríguez]

> On Dec. 18, 2015, 8:12 p.m., John Pullokkaran wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java, line 167
> > 
> >
> > This is fragile.
> > State will have to be maintained; i.e unset if a filter merge happened.

This is already taken care of, as new Filter is created and the property is set 
to false by default.


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41457/#review111255
---


On Dec. 18, 2015, 5:28 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41457/
> ---
> 
> (Updated Dec. 18, 2015, 5:28 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Bugs: HIVE-11865
> https://issues.apache.org/jira/browse/HIVE-11865
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Disable Hive PPD optimizer when CBO has optimized the plan
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out 
> 3907bc946679c0ce2492692dcb7e2cf252c6e188 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 
> 8f48e7df4b9c081528b82545d445f75f67f93bfc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  142812ca703d5e667bd532a711d29594123bde50 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTSTransposeRule.java
>  8321504774c5ec5c7c9de3aecdf78dcd8408c4cf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java
>  7e484b90deed2dfb120faa2699eb9b9b7a4e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java
>  c8de1d89a1df083cd125ae8c11bf854b09915680 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
>  efde6652ca270d8e8d8f682fc0ff260702f46d5a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  feec3c25929cac0bb3b6f50d074557a3c2745889 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 7c5a43ff4060c9a4d9c4c5523d851ef959da3932 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 27549dceac8b5913b1fb1604bde213ae26b243bf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
> ccc4bb454e61e3c39b1f90f29de5e754f8d886bf 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 
> 1702628447d38b09c9b7d0dcf38218a9037a2c64 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SimplePredicatePushDown.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
> accfb3b9f5631f25220c88b759dba6c5299b4044 
>   ql/src/test/results/clientpositive/auto_join12.q.out 
> 6f08aa8f043113f0a99acd10c848f60a122a3f0b 
>   ql/src/test/results/clientpositive/auto_join13.q.out 
> d29818cf438449a6a1413edd7d474c4ab35f2148 
>   ql/src/test/results/clientpositive/auto_join16.q.out 
> 38738ede2b4cd49aa24490eafb81dec1de78bea5 
>   ql/src/test/results/clientpositive/auto_join2.q.out 
> ce40ff30cb9cc2ab734ee37c0f3dc0ae85bbd202 
>   ql/src/test/results/clientpositive/auto_join33.q.out 
> 8b13bd0cb7fd869c58c9d092b2e8710391c96503 
>   ql/src/test/results/clientpositive/auto_join_filters.q.out 
> e0ed3737f9b25d45bdb365b13aeed3a40c50135f 
>   ql/src/test/results/clientpositive/auto_join_nulls.q.out 
> 954bf069760c6f0d7e5420c7e9a3b9756d5aa4df 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out 
> feb818679df7d9f63f2d2d7a7c9afb2d0f5874d8 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 
> e0d0146dd6b86e9f69a467dcde0c020c31af499d 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
> c63016d0a3610eaa6cd82a4adc5d1bf6f0925cce 
>   ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out 
> 096f5d445bc7e61381775270bef04e911eac5bb6 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 
> 30853c42a52e1def5a913352580e0b5391ac2050 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 
> 3e8dbcfd5bcc66ccd83b01c08c7745b02f05328f 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 
> e6c2832003a1bb804f99125866bff6117e4dab73 
>   ql/src/test/results/clientpositive/bucketsortoptimize_insert_7.q.out 
> 326e4d67f00158929f4aaa0a2ec53cedabe3e92c 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b1e43fed3326c8c43e3266ad281c86a054 
>   ql/src/test/results/clientpositive/cbo_rp_join0.q.out 
> 3c6bb7319fd7605a62f082b989f723d5aab6c3fe 

Re: Review Request 41457: HIVE-11865

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41457/
---

(Updated Dec. 18, 2015, 10:19 p.m.)


Review request for hive, Ashutosh Chauhan and John Pullokkaran.


Bugs: HIVE-11865
https://issues.apache.org/jira/browse/HIVE-11865


Repository: hive-git


Description
---

Disable Hive PPD optimizer when CBO has optimized the plan


Diffs (updated)
-

  hbase-handler/src/test/results/positive/hbase_queries.q.out 
3907bc946679c0ce2492692dcb7e2cf252c6e188 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 
8f48e7df4b9c081528b82545d445f75f67f93bfc 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
 142812ca703d5e667bd532a711d29594123bde50 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTSTransposeRule.java
 8321504774c5ec5c7c9de3aecdf78dcd8408c4cf 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java
 7e484b90deed2dfb120faa2699eb9b9b7a4e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortTransposeRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java
 c8de1d89a1df083cd125ae8c11bf854b09915680 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
7c5a43ff4060c9a4d9c4c5523d851ef959da3932 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
27549dceac8b5913b1fb1604bde213ae26b243bf 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
ccc4bb454e61e3c39b1f90f29de5e754f8d886bf 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 
1702628447d38b09c9b7d0dcf38218a9037a2c64 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/SimplePredicatePushDown.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
accfb3b9f5631f25220c88b759dba6c5299b4044 
  ql/src/test/results/clientpositive/auto_join12.q.out 
6f08aa8f043113f0a99acd10c848f60a122a3f0b 
  ql/src/test/results/clientpositive/auto_join13.q.out 
d29818cf438449a6a1413edd7d474c4ab35f2148 
  ql/src/test/results/clientpositive/auto_join16.q.out 
38738ede2b4cd49aa24490eafb81dec1de78bea5 
  ql/src/test/results/clientpositive/auto_join2.q.out 
ce40ff30cb9cc2ab734ee37c0f3dc0ae85bbd202 
  ql/src/test/results/clientpositive/auto_join33.q.out 
8b13bd0cb7fd869c58c9d092b2e8710391c96503 
  ql/src/test/results/clientpositive/auto_join_filters.q.out 
e0ed3737f9b25d45bdb365b13aeed3a40c50135f 
  ql/src/test/results/clientpositive/auto_join_nulls.q.out 
954bf069760c6f0d7e5420c7e9a3b9756d5aa4df 
  ql/src/test/results/clientpositive/auto_join_stats.q.out 
feb818679df7d9f63f2d2d7a7c9afb2d0f5874d8 
  ql/src/test/results/clientpositive/auto_join_stats2.q.out 
e0d0146dd6b86e9f69a467dcde0c020c31af499d 
  ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
c63016d0a3610eaa6cd82a4adc5d1bf6f0925cce 
  ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out 
096f5d445bc7e61381775270bef04e911eac5bb6 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 
30853c42a52e1def5a913352580e0b5391ac2050 
  ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 
3e8dbcfd5bcc66ccd83b01c08c7745b02f05328f 
  ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 
e6c2832003a1bb804f99125866bff6117e4dab73 
  ql/src/test/results/clientpositive/bucketsortoptimize_insert_7.q.out 
326e4d67f00158929f4aaa0a2ec53cedabe3e92c 
  ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
62b611b1e43fed3326c8c43e3266ad281c86a054 
  ql/src/test/results/clientpositive/cbo_rp_join0.q.out 
3c6bb7319fd7605a62f082b989f723d5aab6c3fe 
  ql/src/test/results/clientpositive/cbo_rp_join1.q.out 
e77002866174f85168859c4145e7016c9d829bb6 
  ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out 
b14caa80b3346036f9ec835fe4e2294f47eb7ee5 
  ql/src/test/results/clientpositive/cbo_rp_outer_join_ppr.q.java1.7.out 
263053705347793513a44d49968dedd4fdffecfe 
  ql/src/test/results/clientpositive/correlationoptimizer10.q.out 
860452ef9bbf38df7aaeb00eb2636f53730ae923 
  ql/src/test/results/clientpositive/correlationoptimizer13.q.out 
82304d8ff96e14bf18cde3f0867b25c6757d40d6 
  ql/src/test/results/clientpositive/correlationoptimizer9.q.out 
d32003dead268bd435713751d5e5956bdd4f5f74 
  ql/src/test/results/clientpositive/cross_join.q.out 
f36496e8503fedc67d6c9555b6e0dd5b0c9bd6ed 
  ql/src/test/results/clientpositive/cross_product_check_1.q.out 
e7d6900d64851fa21f976ca20b5d7d753aa5d28f 
  ql/src/test/results/clientpositive/cross_product_check_2.q.out 
df438c9ae7c95033383eae28bf8bada55c08b5c4 
  ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out 
b85d387f3910598a3300df9adc34dce952715a2e 
  
ql/src/test/results/clientpositive/encrypted/encryption_join_unencrypted_tbl.q.out
 

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

[Sergey Shelukhin]

> On Dec. 3, 2015, 9:40 p.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 390
> > 
> >
> > some existing configuration is probably needed (and better)
> 
> Takanobu Asanuma wrote:
> IIUC, hive configurations are in the server side and HiveQueryResultSet 
> is in the client side. So I don't understand how to load hive configurations 
> in HiveQueryResultSet. Is there any idea?

configuration does exist on client side, although it looks like noone on JDBC 
currently creates and object. So these changes make sense


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review108889
---


On Dec. 18, 2015, 7:13 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated Dec. 18, 2015, 7:13 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b5aee00 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 245c6a3 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 3f9024a 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 3d5f3b5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
>   service/if/TCLIService.thrift baf583f 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.h b078c99 
>   service/src/gen/thrift/gen-cpp/TCLIService_types.cpp b852379 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementResp.java
>  0b9aa0f 
>   
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TProtocolVersion.java
>  c936ada 
>   service/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5 
>   service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote e167d5b 
>   service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4 
>   service/src/java/org/apache/hive/service/cli/CLIService.java adc9809 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> d2b3f9c 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 92135cd 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> d90dd0d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 50c912e 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8434965 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

Review Request 41582: HIVE-12713: Miscellaneous improvements in driver compile and execute logging

[Chaoyu Tang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41582/
---

Review request for hive.


Bugs: HIVE-12713
https://issues.apache.org/jira/browse/HIVE-12713


Repository: hive-git


Description
---

Patch improves the driver compile and execute logging in following:
1. ensuring that only the redacted query to be logged out
2. removing redundant variable substitution in HS2 SQLOperation
3. logging out the query and its compilation time without having to enable 
PerfLogger debug, to help identify badly written queries which take a lot of 
time to compile and probably cause other good queries to be queued (HIVE-12516)


Diffs
-

  common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 98ebd50 
  
itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java 
7cc0acf 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
e9206b9 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/cbo_rp_TestJdbcDriver2.java 
c66f166 
  
itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingAPIWithMr.java
 d21571e 
  
itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingAPIWithTez.java
 8b21647 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 3d5f3b5 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java c33bb66 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
d90dd0d 

Diff: https://reviews.apache.org/r/41582/diff/


Testing
---

1. Manual tests
2. Precommit tests


Thanks,

Chaoyu Tang

Re: Review Request 40865: Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40865/#review111348
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
 (line 334)


Hive's null literal has void type.
This is handled down in void switch.

So we might want to switch calcite null literal to void.


- John Pullokkaran


On Dec. 19, 2015, 12:28 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40865/
> ---
> 
> (Updated Dec. 19, 2015, 12:28 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11927
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  336745b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  425514d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  3f2267d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  631a4ca 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
>  2825f77 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7c5a43f 
>   ql/src/test/queries/clientpositive/cbo_const.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/constantfolding.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out c4d59c8 
>   ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 277b0f7 
>   ql/src/test/results/clientpositive/cast1.q.out 0bdecba 
>   ql/src/test/results/clientpositive/cbo_const.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out b14caa8 
>   ql/src/test/results/clientpositive/constantfolding.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cross_product_check_1.q.out e7d6900 
>   ql/src/test/results/clientpositive/cross_product_check_2.q.out df438c9 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out b85d387 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 24ac550 
>   ql/src/test/results/clientpositive/groupby_ppd.q.out 6164a26 
>   ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 7333677 
>   ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out e19d1de 
>   ql/src/test/results/clientpositive/input_part1.q.out d6f4d3e 
>   ql/src/test/results/clientpositive/input_part5.q.out f2d7335 
>   ql/src/test/results/clientpositive/input_part6.q.out fa51cdf 
>   ql/src/test/results/clientpositive/lineage2.q.out ec8b76b 
>   ql/src/test/results/clientpositive/lineage3.q.out 747dc9a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> be77ba8 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 79348f3 
>   ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out f6d8388 
>   ql/src/test/results/clientpositive/parquet_predicate_pushdown.q.out b322ef1 
>   ql/src/test/results/clientpositive/partition_multilevels.q.out c1c8778 
>   ql/src/test/results/clientpositive/perf/query31.q.out c479d91 
>   ql/src/test/results/clientpositive/perf/query39.q.out 0ad62cc 
>   ql/src/test/results/clientpositive/perf/query42.q.out a3264c7 
>   ql/src/test/results/clientpositive/perf/query52.q.out ac0c1e6 
>   ql/src/test/results/clientpositive/perf/query64.q.out 9735b45 
>   ql/src/test/results/clientpositive/perf/query66.q.out 42bcb83 
>   ql/src/test/results/clientpositive/perf/query75.q.out 70a1649 
>   ql/src/test/results/clientpositive/pointlookup2.q.out a442425 
>   ql/src/test/results/clientpositive/quotedid_basic.q.out 519f647 
>   ql/src/test/results/clientpositive/quotedid_partition.q.out d34a005 
>   ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out b5e7846 
>   ql/src/test/results/clientpositive/spark/cross_product_check_1.q.out 
> 5b03dcf 
>   ql/src/test/results/clientpositive/spark/cross_product_check_2.q.out 
> 93c502d 
>   ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out b597ebd 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 239e803 
>   ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 
> 8370bbe 
>   ql/src/test/results/clientpositive/spark/union_remove_25.q.out b771fe9 
>  

Re: Review Request 40865: Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40865/#review111351
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
 (line 320)


Rename class?


- John Pullokkaran


On Dec. 19, 2015, 12:28 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40865/
> ---
> 
> (Updated Dec. 19, 2015, 12:28 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11927
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  336745b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  425514d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  3f2267d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  631a4ca 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
>  2825f77 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7c5a43f 
>   ql/src/test/queries/clientpositive/cbo_const.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/constantfolding.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out c4d59c8 
>   ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 277b0f7 
>   ql/src/test/results/clientpositive/cast1.q.out 0bdecba 
>   ql/src/test/results/clientpositive/cbo_const.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out b14caa8 
>   ql/src/test/results/clientpositive/constantfolding.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cross_product_check_1.q.out e7d6900 
>   ql/src/test/results/clientpositive/cross_product_check_2.q.out df438c9 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out b85d387 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 24ac550 
>   ql/src/test/results/clientpositive/groupby_ppd.q.out 6164a26 
>   ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 7333677 
>   ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out e19d1de 
>   ql/src/test/results/clientpositive/input_part1.q.out d6f4d3e 
>   ql/src/test/results/clientpositive/input_part5.q.out f2d7335 
>   ql/src/test/results/clientpositive/input_part6.q.out fa51cdf 
>   ql/src/test/results/clientpositive/lineage2.q.out ec8b76b 
>   ql/src/test/results/clientpositive/lineage3.q.out 747dc9a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> be77ba8 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 79348f3 
>   ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out f6d8388 
>   ql/src/test/results/clientpositive/parquet_predicate_pushdown.q.out b322ef1 
>   ql/src/test/results/clientpositive/partition_multilevels.q.out c1c8778 
>   ql/src/test/results/clientpositive/perf/query31.q.out c479d91 
>   ql/src/test/results/clientpositive/perf/query39.q.out 0ad62cc 
>   ql/src/test/results/clientpositive/perf/query42.q.out a3264c7 
>   ql/src/test/results/clientpositive/perf/query52.q.out ac0c1e6 
>   ql/src/test/results/clientpositive/perf/query64.q.out 9735b45 
>   ql/src/test/results/clientpositive/perf/query66.q.out 42bcb83 
>   ql/src/test/results/clientpositive/perf/query75.q.out 70a1649 
>   ql/src/test/results/clientpositive/pointlookup2.q.out a442425 
>   ql/src/test/results/clientpositive/quotedid_basic.q.out 519f647 
>   ql/src/test/results/clientpositive/quotedid_partition.q.out d34a005 
>   ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out b5e7846 
>   ql/src/test/results/clientpositive/spark/cross_product_check_1.q.out 
> 5b03dcf 
>   ql/src/test/results/clientpositive/spark/cross_product_check_2.q.out 
> 93c502d 
>   ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out b597ebd 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 239e803 
>   ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 
> 8370bbe 
>   ql/src/test/results/clientpositive/spark/union_remove_25.q.out b771fe9 
>   ql/src/test/results/clientpositive/spark/union_view.q.out cce7710 
>   

Re: Review Request 40865: Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40865/#review111345
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
 (line 183)


Encoding binary as string literal wouldn't this result in lossy translation?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
 (line 249)


Why add milli seconds?


- John Pullokkaran


On Dec. 19, 2015, 12:28 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40865/
> ---
> 
> (Updated Dec. 19, 2015, 12:28 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11927
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  336745b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  425514d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  3f2267d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  631a4ca 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
>  2825f77 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7c5a43f 
>   ql/src/test/queries/clientpositive/cbo_const.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/constantfolding.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out c4d59c8 
>   ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 277b0f7 
>   ql/src/test/results/clientpositive/cast1.q.out 0bdecba 
>   ql/src/test/results/clientpositive/cbo_const.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out b14caa8 
>   ql/src/test/results/clientpositive/constantfolding.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cross_product_check_1.q.out e7d6900 
>   ql/src/test/results/clientpositive/cross_product_check_2.q.out df438c9 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out b85d387 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 24ac550 
>   ql/src/test/results/clientpositive/groupby_ppd.q.out 6164a26 
>   ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 7333677 
>   ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out e19d1de 
>   ql/src/test/results/clientpositive/input_part1.q.out d6f4d3e 
>   ql/src/test/results/clientpositive/input_part5.q.out f2d7335 
>   ql/src/test/results/clientpositive/input_part6.q.out fa51cdf 
>   ql/src/test/results/clientpositive/lineage2.q.out ec8b76b 
>   ql/src/test/results/clientpositive/lineage3.q.out 747dc9a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> be77ba8 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 79348f3 
>   ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out f6d8388 
>   ql/src/test/results/clientpositive/parquet_predicate_pushdown.q.out b322ef1 
>   ql/src/test/results/clientpositive/partition_multilevels.q.out c1c8778 
>   ql/src/test/results/clientpositive/perf/query31.q.out c479d91 
>   ql/src/test/results/clientpositive/perf/query39.q.out 0ad62cc 
>   ql/src/test/results/clientpositive/perf/query42.q.out a3264c7 
>   ql/src/test/results/clientpositive/perf/query52.q.out ac0c1e6 
>   ql/src/test/results/clientpositive/perf/query64.q.out 9735b45 
>   ql/src/test/results/clientpositive/perf/query66.q.out 42bcb83 
>   ql/src/test/results/clientpositive/perf/query75.q.out 70a1649 
>   ql/src/test/results/clientpositive/pointlookup2.q.out a442425 
>   ql/src/test/results/clientpositive/quotedid_basic.q.out 519f647 
>   ql/src/test/results/clientpositive/quotedid_partition.q.out d34a005 
>   ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out b5e7846 
>   ql/src/test/results/clientpositive/spark/cross_product_check_1.q.out 
> 5b03dcf 
>   ql/src/test/results/clientpositive/spark/cross_product_check_2.q.out 
> 93c502d 
>   ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out b597ebd 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 239e803 
>   

Re: Review Request 40865: Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40865/#review111344
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
 (line 325)


This is a change in interface.
Instead of translating a type now we are translating a RexCall.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
 (line 348)


Why are we looking at operands arguments?


- John Pullokkaran


On Dec. 19, 2015, 12:28 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40865/
> ---
> 
> (Updated Dec. 19, 2015, 12:28 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11927
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  336745b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  425514d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  3f2267d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  631a4ca 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
>  2825f77 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7c5a43f 
>   ql/src/test/queries/clientpositive/cbo_const.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/constantfolding.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out c4d59c8 
>   ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 277b0f7 
>   ql/src/test/results/clientpositive/cast1.q.out 0bdecba 
>   ql/src/test/results/clientpositive/cbo_const.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out b14caa8 
>   ql/src/test/results/clientpositive/constantfolding.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cross_product_check_1.q.out e7d6900 
>   ql/src/test/results/clientpositive/cross_product_check_2.q.out df438c9 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out b85d387 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 24ac550 
>   ql/src/test/results/clientpositive/groupby_ppd.q.out 6164a26 
>   ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 7333677 
>   ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out e19d1de 
>   ql/src/test/results/clientpositive/input_part1.q.out d6f4d3e 
>   ql/src/test/results/clientpositive/input_part5.q.out f2d7335 
>   ql/src/test/results/clientpositive/input_part6.q.out fa51cdf 
>   ql/src/test/results/clientpositive/lineage2.q.out ec8b76b 
>   ql/src/test/results/clientpositive/lineage3.q.out 747dc9a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> be77ba8 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 79348f3 
>   ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out f6d8388 
>   ql/src/test/results/clientpositive/parquet_predicate_pushdown.q.out b322ef1 
>   ql/src/test/results/clientpositive/partition_multilevels.q.out c1c8778 
>   ql/src/test/results/clientpositive/perf/query31.q.out c479d91 
>   ql/src/test/results/clientpositive/perf/query39.q.out 0ad62cc 
>   ql/src/test/results/clientpositive/perf/query42.q.out a3264c7 
>   ql/src/test/results/clientpositive/perf/query52.q.out ac0c1e6 
>   ql/src/test/results/clientpositive/perf/query64.q.out 9735b45 
>   ql/src/test/results/clientpositive/perf/query66.q.out 42bcb83 
>   ql/src/test/results/clientpositive/perf/query75.q.out 70a1649 
>   ql/src/test/results/clientpositive/pointlookup2.q.out a442425 
>   ql/src/test/results/clientpositive/quotedid_basic.q.out 519f647 
>   ql/src/test/results/clientpositive/quotedid_partition.q.out d34a005 
>   ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out b5e7846 
>   ql/src/test/results/clientpositive/spark/cross_product_check_1.q.out 
> 5b03dcf 
>   ql/src/test/results/clientpositive/spark/cross_product_check_2.q.out 
> 93c502d 
>   ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out b597ebd 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 239e803 
>   

Re: Review Request 40865: Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40865/#review111349
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
 (line 396)


Not sure about isInfinite.



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 1210)


Could we add executor to previous hepPlan and check for null



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 1267)


Could we add executor to previous hepPlan and check for null


- John Pullokkaran


On Dec. 19, 2015, 12:28 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40865/
> ---
> 
> (Updated Dec. 19, 2015, 12:28 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11927
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  336745b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  425514d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  3f2267d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  631a4ca 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
>  2825f77 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7c5a43f 
>   ql/src/test/queries/clientpositive/cbo_const.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/constantfolding.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out c4d59c8 
>   ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 277b0f7 
>   ql/src/test/results/clientpositive/cast1.q.out 0bdecba 
>   ql/src/test/results/clientpositive/cbo_const.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out b14caa8 
>   ql/src/test/results/clientpositive/constantfolding.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cross_product_check_1.q.out e7d6900 
>   ql/src/test/results/clientpositive/cross_product_check_2.q.out df438c9 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out b85d387 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 24ac550 
>   ql/src/test/results/clientpositive/groupby_ppd.q.out 6164a26 
>   ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 7333677 
>   ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out e19d1de 
>   ql/src/test/results/clientpositive/input_part1.q.out d6f4d3e 
>   ql/src/test/results/clientpositive/input_part5.q.out f2d7335 
>   ql/src/test/results/clientpositive/input_part6.q.out fa51cdf 
>   ql/src/test/results/clientpositive/lineage2.q.out ec8b76b 
>   ql/src/test/results/clientpositive/lineage3.q.out 747dc9a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> be77ba8 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 79348f3 
>   ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out f6d8388 
>   ql/src/test/results/clientpositive/parquet_predicate_pushdown.q.out b322ef1 
>   ql/src/test/results/clientpositive/partition_multilevels.q.out c1c8778 
>   ql/src/test/results/clientpositive/perf/query31.q.out c479d91 
>   ql/src/test/results/clientpositive/perf/query39.q.out 0ad62cc 
>   ql/src/test/results/clientpositive/perf/query42.q.out a3264c7 
>   ql/src/test/results/clientpositive/perf/query52.q.out ac0c1e6 
>   ql/src/test/results/clientpositive/perf/query64.q.out 9735b45 
>   ql/src/test/results/clientpositive/perf/query66.q.out 42bcb83 
>   ql/src/test/results/clientpositive/perf/query75.q.out 70a1649 
>   ql/src/test/results/clientpositive/pointlookup2.q.out a442425 
>   ql/src/test/results/clientpositive/quotedid_basic.q.out 519f647 
>   ql/src/test/results/clientpositive/quotedid_partition.q.out d34a005 
>   ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out b5e7846 
>   ql/src/test/results/clientpositive/spark/cross_product_check_1.q.out 
> 5b03dcf 
>   ql/src/test/results/clientpositive/spark/cross_product_check_2.q.out 
> 93c502d 
>   

[jira] [Created] (HIVE-12713) Miscellaneous improvements in driver compile and execute logging

[Chaoyu Tang (JIRA)]

Chaoyu Tang created HIVE-12713:
--

 Summary: Miscellaneous improvements in driver compile and execute 
logging
 Key: HIVE-12713
 URL: https://issues.apache.org/jira/browse/HIVE-12713
 Project: Hive
  Issue Type: Improvement
  Components: Logging
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang
Priority: Minor


Miscellaneous compile and execute logging improvements include:
1. ensuring that only the redacted query to be logged out
2. removing redundant variable substitution in HS2 SQLOperation
3. logging out the query and its compilation time without having to enable 
PerfLogger debug, to help identify badly written queries which take a lot of 
time to compile and probably cause other good queries to be queued (HIVE-12516)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)