[ https://issues.apache.org/jira/browse/HIVE-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13626957#comment-13626957 ]
Owen O'Malley commented on HIVE-4248: ------------------------------------- Kevin, After thinking about it a bit more, how about if I ask the writers to re-check their memory relative to their allocation when the pool has shrunk by more than 10% from the last time they checked. I ran a quick experiment where I had a pool of 1GB and an increasing set of 250MB writers. By only doing the check when the pool has changed by more than 10%, as 1000 writers were added it cut down the number checks from 1000 to 49. Does that sound reasonable? > Implement a memory manager for ORC > ---------------------------------- > > Key: HIVE-4248 > URL: https://issues.apache.org/jira/browse/HIVE-4248 > Project: Hive > Issue Type: New Feature > Components: Serializers/Deserializers > Reporter: Owen O'Malley > Assignee: Owen O'Malley > Attachments: HIVE-4248.D9993.1.patch, HIVE-4248.D9993.2.patch > > > With the large default stripe size (256MB) and dynamic partitions, it is > quite easy for users to run out of memory when writing ORC files. We probably > need a solution that keeps track of the total number of concurrent ORC > writers and divides the available heap space between them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira