Re: Review Request 29800: Apply ColumnPrunning for noop PTFs

2015-01-23 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29800/#review69402
---

Ship it!


Ship It!

- Ashutosh Chauhan


On Jan. 23, 2015, 5:41 a.m., Navis Ryu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/29800/
 ---
 
 (Updated Jan. 23, 2015, 5:41 a.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-9341
 https://issues.apache.org/jira/browse/HIVE-9341
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Currently, PTF disables CP optimization, which can make a huge burden. For 
 example,
 {noformat}
 select p_mfgr, p_name, p_size,
 rank() over (partition by p_mfgr order by p_name) as r,
 dense_rank() over (partition by p_mfgr order by p_name) as dr,
 sum(p_retailprice) over (partition by p_mfgr order by p_name rows between 
 unbounded preceding and current row) as s1
 from noop(on part 
   partition by p_mfgr
   order by p_name
   );
 
 STAGE PLANS:
   Stage: Stage-1
 Map Reduce
   Map Operator Tree:
   TableScan
 alias: part
 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE 
 Column stats: NONE
 Reduce Output Operator
   key expressions: p_mfgr (type: string), p_name (type: string)
   sort order: ++
   Map-reduce partition columns: p_mfgr (type: string)
   Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE 
 Column stats: NONE
   value expressions: p_partkey (type: int), p_name (type: 
 string), p_mfgr (type: string), p_brand (type: string), p_type (type: 
 string), p_size (type: int), p_container (type: string), p_retailprice (type: 
 double), p_comment (type: string), BLOCK__OFFSET__INSIDE__FILE (type: 
 bigint), INPUT__FILE__NAME (type: string), ROW__ID (type: 
 structtransactionid:bigint,bucketid:int,rowid:bigint)
 ...
 {noformat}
 
 There should be a generic way to discern referenced columns but before that, 
 we know CP can be safely applied to noop functions.
 
 
 Diffs
 -
 
   itests/src/test/resources/testconfiguration.properties d08651b 
   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 479af32 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 
 abf32f1 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkSkewJoinProcFactory.java
  fe698ef 
   ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicatePushDown.java ee7328e 
   ql/src/test/queries/clientpositive/ptf.q 56eef0a 
   ql/src/test/queries/clientpositive/ptf_streaming.q 04b674c 
   ql/src/test/results/clientpositive/ptf.q.out 9196b94 
   ql/src/test/results/clientpositive/ptf_streaming.q.out ef7ae88 
   ql/src/test/results/clientpositive/spark/ptf.q.out 9196b94 
   ql/src/test/results/clientpositive/spark/ptf_streaming.q.out ef7ae88 
   ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out f3b61ce 
   ql/src/test/results/clientpositive/tez/ptf.q.out 9196b94 
   ql/src/test/results/clientpositive/tez/ptf_streaming.q.out PRE-CREATION 
   ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out 928c9f0 
   ql/src/test/results/clientpositive/vectorized_ptf.q.out 7fdd1d8 
 
 Diff: https://reviews.apache.org/r/29800/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Navis Ryu

Re: Review Request 29800: Apply ColumnPrunning for noop PTFs

2015-01-22 Thread Navis Ryu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29800/
---

(Updated Jan. 23, 2015, 5:41 a.m.)


Review request for hive.


Changes
---

Updated with full diff


Bugs: HIVE-9341
https://issues.apache.org/jira/browse/HIVE-9341


Repository: hive-git


Description
---

Currently, PTF disables CP optimization, which can make a huge burden. For 
example,
{noformat}
select p_mfgr, p_name, p_size,
rank() over (partition by p_mfgr order by p_name) as r,
dense_rank() over (partition by p_mfgr order by p_name) as dr,
sum(p_retailprice) over (partition by p_mfgr order by p_name rows between 
unbounded preceding and current row) as s1
from noop(on part 
  partition by p_mfgr
  order by p_name
  );

STAGE PLANS:
  Stage: Stage-1
Map Reduce
  Map Operator Tree:
  TableScan
alias: part
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE 
Column stats: NONE
Reduce Output Operator
  key expressions: p_mfgr (type: string), p_name (type: string)
  sort order: ++
  Map-reduce partition columns: p_mfgr (type: string)
  Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE 
Column stats: NONE
  value expressions: p_partkey (type: int), p_name (type: string), 
p_mfgr (type: string), p_brand (type: string), p_type (type: string), p_size 
(type: int), p_container (type: string), p_retailprice (type: double), 
p_comment (type: string), BLOCK__OFFSET__INSIDE__FILE (type: bigint), 
INPUT__FILE__NAME (type: string), ROW__ID (type: 
structtransactionid:bigint,bucketid:int,rowid:bigint)
...
{noformat}

There should be a generic way to discern referenced columns but before that, we 
know CP can be safely applied to noop functions.


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties d08651b 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 479af32 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 
abf32f1 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkSkewJoinProcFactory.java
 fe698ef 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicatePushDown.java ee7328e 
  ql/src/test/queries/clientpositive/ptf.q 56eef0a 
  ql/src/test/queries/clientpositive/ptf_streaming.q 04b674c 
  ql/src/test/results/clientpositive/ptf.q.out 9196b94 
  ql/src/test/results/clientpositive/ptf_streaming.q.out ef7ae88 
  ql/src/test/results/clientpositive/spark/ptf.q.out 9196b94 
  ql/src/test/results/clientpositive/spark/ptf_streaming.q.out ef7ae88 
  ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out f3b61ce 
  ql/src/test/results/clientpositive/tez/ptf.q.out 9196b94 
  ql/src/test/results/clientpositive/tez/ptf_streaming.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out 928c9f0 
  ql/src/test/results/clientpositive/vectorized_ptf.q.out 7fdd1d8 

Diff: https://reviews.apache.org/r/29800/diff/


Testing
---


Thanks,

Navis Ryu

Review Request 29800: Apply ColumnPrunning for noop PTFs

2015-01-11 Thread Navis Ryu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29800/
---

Review request for hive.


Bugs: HIVE-9341
https://issues.apache.org/jira/browse/HIVE-9341


Repository: hive-git


Description
---

Currently, PTF disables CP optimization, which can make a huge burden. For 
example,
{noformat}
select p_mfgr, p_name, p_size,
rank() over (partition by p_mfgr order by p_name) as r,
dense_rank() over (partition by p_mfgr order by p_name) as dr,
sum(p_retailprice) over (partition by p_mfgr order by p_name rows between 
unbounded preceding and current row) as s1
from noop(on part 
  partition by p_mfgr
  order by p_name
  );

STAGE PLANS:
  Stage: Stage-1
Map Reduce
  Map Operator Tree:
  TableScan
alias: part
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE 
Column stats: NONE
Reduce Output Operator
  key expressions: p_mfgr (type: string), p_name (type: string)
  sort order: ++
  Map-reduce partition columns: p_mfgr (type: string)
  Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE 
Column stats: NONE
  value expressions: p_partkey (type: int), p_name (type: string), 
p_mfgr (type: string), p_brand (type: string), p_type (type: string), p_size 
(type: int), p_container (type: string), p_retailprice (type: double), 
p_comment (type: string), BLOCK__OFFSET__INSIDE__FILE (type: bigint), 
INPUT__FILE__NAME (type: string), ROW__ID (type: 
structtransactionid:bigint,bucketid:int,rowid:bigint)
...
{noformat}

There should be a generic way to discern referenced columns but before that, we 
know CP can be safely applied to noop functions.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 
afd1738 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicatePushDown.java ee7328e 

Diff: https://reviews.apache.org/r/29800/diff/


Testing
---


Thanks,

Navis Ryu