Re: Review Request 34666: HIVE-9152 - Dynamic Partition Pruning [Spark Branch]

[chengxiang li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34666/#review91427
---

Ship it!


Ship It!

- chengxiang li


On 七月 8, 2015, 6:04 p.m., Chao Sun wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34666/
 ---
 
 (Updated 七月 8, 2015, 6:04 p.m.)
 
 
 Review request for hive, chengxiang li and Xuefu Zhang.
 
 
 Bugs: HIVE-9152
 https://issues.apache.org/jira/browse/HIVE-9152
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Tez implemented dynamic partition pruning in HIVE-7826. This is a nice 
 optimization and we should implement the same in HOS.
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 27f68df 
   itests/src/test/resources/testconfiguration.properties 4f2de12 
   ql/if/queryplan.thrift c8dfa35 
   
 ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java
  e18f935 
   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java f58a10b 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java 
 21398d8 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkDynamicPartitionPruner.java
  PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java 
 ca0ffb6 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSparkPartitionPruningSinkOperator.java
  PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java 
 1de7e40 
   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 2ff3951 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
  8546d21 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java a7cf8b7 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/SparkRemoveDynamicPruningBySize.java
  PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
 ad47547 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkPartitionPruningSinkDesc.java
  PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java 
 447f104 
   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 
 7992c88 
   
 ql/src/java/org/apache/hadoop/hive/ql/parse/spark/OptimizeSparkProcContext.java
  f7586a4 
   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 
 7f2c079 
   
 ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkPartitionPruningSinkOperator.java
  PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SplitOpTreeForDPP.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 3217df2 
   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 9e9a2a2 
   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
 363e49e 
   ql/src/test/queries/clientpositive/spark_dynamic_partition_pruning.q 
 PRE-CREATION 
   ql/src/test/queries/clientpositive/spark_dynamic_partition_pruning_2.q 
 PRE-CREATION 
   
 ql/src/test/queries/clientpositive/spark_vectorized_dynamic_partition_pruning.q
  PRE-CREATION 
   
 ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning.q.out
  PRE-CREATION 
   
 ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning_2.q.out
  PRE-CREATION 
   
 ql/src/test/results/clientpositive/spark/spark_vectorized_dynamic_partition_pruning.q.out
  PRE-CREATION 
   
 ql/src/test/results/clientpositive/spark/vectorized_dynamic_partition_pruning.q.out
  PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/34666/diff/
 
 
 Testing
 ---
 
 spark_dynamic_partition_pruning.q, spark_dynamic_partition_pruning_2.q - both 
 are clone from tez's test.
 
 
 Thanks,
 
 Chao Sun
 

[jira] [Created] (HIVE-11234) CBO: Calcite Operator To Hive Operator (Calcite Return Path): reconsider the cross product in HiveJoinToMultiJoinRule

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-11234:
--

 Summary: CBO: Calcite Operator To Hive Operator (Calcite Return 
Path): reconsider the cross product in HiveJoinToMultiJoinRule
 Key: HIVE-11234
 URL: https://issues.apache.org/jira/browse/HIVE-11234
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong


Cross product in HiveJoinToMultiJoinRule needs to be reconsidered. Now the 
following explain query will fail at L1116 StatsRulesProcFactory : FAILED: 
IndexOutOfBoundsException Index: 0, Size: 0 because the key in one of the 
branches of the join is empty.
{code}
explain select *  from src 
where src.key not in (select key  from src1);
{code}

If we run
{code}
select *  from src 
where src.key not in (select key  from src1);
{code}
we will have FAILED: Execution Error, return code 2 from 
org.apache.hadoop.hive.ql.exec.mr.MapRedTask. The root cause is the same. 
Other queries that will also fail include cbo_subq_not_in.q etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11233) Support of binary sortable key and values for HBaseStorageHandler

[Svetozar Ivanov (JIRA)]

Svetozar Ivanov created HIVE-11233:
--

 Summary: Support of binary sortable key and values for 
HBaseStorageHandler
 Key: HIVE-11233
 URL: https://issues.apache.org/jira/browse/HIVE-11233
 Project: Hive
  Issue Type: New Feature
  Components: HBase Handler
Affects Versions: 1.2.1
Reporter: Svetozar Ivanov
Assignee: Svetozar Ivanov
Priority: Minor


Currently HBaseStorageHandler doesn't provide mechanism for storage of binary 
sortable key and values. It is necessary when given HBase table is used for 
persistence by Apache Hive and Apache Phoenix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Hive-0.14 - Build # 1010 - Still Failing

[Apache Jenkins Server]

Changes for Build #991

Changes for Build #992

Changes for Build #993

Changes for Build #994

Changes for Build #995

Changes for Build #996

Changes for Build #997

Changes for Build #998

Changes for Build #999

Changes for Build #1000

Changes for Build #1001

Changes for Build #1002

Changes for Build #1003

Changes for Build #1004

Changes for Build #1005

Changes for Build #1006

Changes for Build #1007

Changes for Build #1008

Changes for Build #1009

Changes for Build #1010



No tests ran.

The Apache Jenkins build system has built Hive-0.14 (build #1010)

Status: Still Failing

Check console output at https://builds.apache.org/job/Hive-0.14/1010/ to view 
the results.