[
https://issues.apache.org/jira/browse/HIVE-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13735038#comment-13735038
]
Harish Butani commented on HIVE-4963:
-------------------------------------
We already do this. The rows are accumulated in a ByteBasedList; when it fills
up it is spilled to disk and a new ByteBasedList is added. So if there are less
than 32Mb bytes needed(or whatever is set by the user), there is no I/O.
The saving here comes from not holding the objects in a serialized form.
Currently every field access goes through deserialization. InMemoryPartition
was going to be the case where the user guarantees that there is enough memory
so we just hold the deserialized objects. Am working on a Caching wrapper on
the PTFPartition which would hold onto deserialized objects, but is backed by
the serialized bytes in case we run out of memory.
But yes it would be nice to merge these 2 concepts into one thing. There is an
overhead in Caching over InMemoryPartition: at least an extra serialization,
potentially more in both time and space. But the overhead may not matter that
much. Give me a couple of days to work through this..
> Support in memory PTF partitions
> --------------------------------
>
> Key: HIVE-4963
> URL: https://issues.apache.org/jira/browse/HIVE-4963
> Project: Hive
> Issue Type: Bug
> Components: PTF-Windowing
> Reporter: Harish Butani
> Attachments: HIVE-4963.D11955.1.patch
>
>
> PTF partitions apply the defensive mode of assuming that partitions will not
> fit in memory. Because of this there is a significant deserialization
> overhead when accessing elements.
> Allow the user to specify that there is enough memory to hold partitions
> through a 'hive.ptf.partition.fits.in.mem' option.
> Savings depends on partition size and in case of windowing the number of
> UDAFs and the window ranges. For eg for the following (admittedly extreme)
> case the PTFOperator exec times went from 39 secs to 8 secs.
>
> {noformat}
> select t, s, i, b, f, d,
> min(t) over(partition by 1 rows between unbounded preceding and current row),
> min(s) over(partition by 1 rows between unbounded preceding and current row),
> min(i) over(partition by 1 rows between unbounded preceding and current row),
> min(b) over(partition by 1 rows between unbounded preceding and current row)
> from over10k
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
