[ 
https://issues.apache.org/jira/browse/HIVE-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775896#comment-13775896
 ] 

Harish Butani commented on HIVE-4963:
-------------------------------------

Sorry forgot to respond.
Original plan was to have the user give a hint on whether partitions fits in 
memory. This would aid in reducing serialization/deserialization cost when 
partitions fit in memory. But based on discussions with Ashutosh, we decided to 
move to using RowContainers for holding rows in a Partition; this way we share 
the same code as Joins; get the functionality and performance benefits of using 
RowContainers. PTFPartitions are now controlled by ConfVars.HIVEJOINCACHESIZE; 
use of ConfVars.HIVE_PTF_PARTITION_PERSISTENT_SIZE has been removed.
                
> Support in memory PTF partitions
> --------------------------------
>
>                 Key: HIVE-4963
>                 URL: https://issues.apache.org/jira/browse/HIVE-4963
>             Project: Hive
>          Issue Type: New Feature
>          Components: PTF-Windowing
>            Reporter: Harish Butani
>            Assignee: Harish Butani
>             Fix For: 0.12.0
>
>         Attachments: HIVE-4963.D11955.1.patch, HIVE-4963.D12279.1.patch, 
> HIVE-4963.D12279.2.patch, HIVE-4963.D12279.3.patch, PTFRowContainer.patch
>
>
> PTF partitions apply the defensive mode of assuming that partitions will not 
> fit in memory. Because of this there is a significant deserialization 
> overhead when accessing elements. 
> Allow the user to specify that there is enough memory to hold partitions 
> through a 'hive.ptf.partition.fits.in.mem' option.  
> Savings depends on partition size and in case of windowing the number of 
> UDAFs and the window ranges. For eg for the following (admittedly extreme) 
> case the PTFOperator exec times went from 39 secs to 8 secs.
>  
> {noformat}
> select t, s, i, b, f, d,
> min(t) over(partition by 1 rows between unbounded preceding and current row), 
> min(s) over(partition by 1 rows between unbounded preceding and current row), 
> min(i) over(partition by 1 rows between unbounded preceding and current row), 
> min(b) over(partition by 1 rows between unbounded preceding and current row) 
> from over10k
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Reply via email to