[
https://issues.apache.org/jira/browse/HIVE-9341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14274604#comment-14274604
]
Navis commented on HIVE-9341:
-----------------------------
Seemed need updates of golden files, but I cannot find TestSparkCliDriver in
trunk branch. Can I ask how to update spark test results,[~szehon], [~xuefuz]?
> Apply ColumnPrunning for noop PTFs
> ----------------------------------
>
> Key: HIVE-9341
> URL: https://issues.apache.org/jira/browse/HIVE-9341
> Project: Hive
> Issue Type: Improvement
> Components: PTF-Windowing
> Reporter: Navis
> Assignee: Navis
> Priority: Trivial
> Attachments: HIVE-9341.1.patch.txt
>
>
> Currently, PTF disables CP optimization, which can make a huge burden. For
> example,
> {noformat}
> select p_mfgr, p_name, p_size,
> rank() over (partition by p_mfgr order by p_name) as r,
> dense_rank() over (partition by p_mfgr order by p_name) as dr,
> sum(p_retailprice) over (partition by p_mfgr order by p_name rows between
> unbounded preceding and current row) as s1
> from noop(on part
> partition by p_mfgr
> order by p_name
> );
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: part
> Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE
> Column stats: NONE
> Reduce Output Operator
> key expressions: p_mfgr (type: string), p_name (type: string)
> sort order: ++
> Map-reduce partition columns: p_mfgr (type: string)
> Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE
> Column stats: NONE
> value expressions: p_partkey (type: int), p_name (type:
> string), p_mfgr (type: string), p_brand (type: string), p_type (type:
> string), p_size (type: int), p_container (type: string), p_retailprice (type:
> double), p_comment (type: string), BLOCK__OFFSET__INSIDE__FILE (type:
> bigint), INPUT__FILE__NAME (type: string), ROW__ID (type:
> struct<transactionid:bigint,bucketid:int,rowid:bigint>)
> ...
> {noformat}
> There should be a generic way to discern referenced columns but before that,
> we know CP can be safely applied to noop functions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
