[ https://issues.apache.org/jira/browse/HIVE-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775896#comment-13775896 ]
Harish Butani commented on HIVE-4963: ------------------------------------- Sorry forgot to respond. Original plan was to have the user give a hint on whether partitions fits in memory. This would aid in reducing serialization/deserialization cost when partitions fit in memory. But based on discussions with Ashutosh, we decided to move to using RowContainers for holding rows in a Partition; this way we share the same code as Joins; get the functionality and performance benefits of using RowContainers. PTFPartitions are now controlled by ConfVars.HIVEJOINCACHESIZE; use of ConfVars.HIVE_PTF_PARTITION_PERSISTENT_SIZE has been removed. > Support in memory PTF partitions > -------------------------------- > > Key: HIVE-4963 > URL: https://issues.apache.org/jira/browse/HIVE-4963 > Project: Hive > Issue Type: New Feature > Components: PTF-Windowing > Reporter: Harish Butani > Assignee: Harish Butani > Fix For: 0.12.0 > > Attachments: HIVE-4963.D11955.1.patch, HIVE-4963.D12279.1.patch, > HIVE-4963.D12279.2.patch, HIVE-4963.D12279.3.patch, PTFRowContainer.patch > > > PTF partitions apply the defensive mode of assuming that partitions will not > fit in memory. Because of this there is a significant deserialization > overhead when accessing elements. > Allow the user to specify that there is enough memory to hold partitions > through a 'hive.ptf.partition.fits.in.mem' option. > Savings depends on partition size and in case of windowing the number of > UDAFs and the window ranges. For eg for the following (admittedly extreme) > case the PTFOperator exec times went from 39 secs to 8 secs. > > {noformat} > select t, s, i, b, f, d, > min(t) over(partition by 1 rows between unbounded preceding and current row), > min(s) over(partition by 1 rows between unbounded preceding and current row), > min(i) over(partition by 1 rows between unbounded preceding and current row), > min(b) over(partition by 1 rows between unbounded preceding and current row) > from over10k > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira