[
https://issues.apache.org/jira/browse/HIVE-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-4248:
------------------------------
Attachment: HIVE-4248.D9993.1.patch
omalley requested code review of "HIVE-4248 [jira] Implement a memory manager
for ORC".
Reviewers: JIRA
HIVE-4248 add memory manager for ORC
With the large default stripe size (256MB) and dynamic partitions, it is quite
easy for users to run out of memory when writing ORC files. We probably need a
solution that keeps track of the total number of concurrent ORC writers and
divides the available heap space between them.
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D9993
AFFECTED FILES
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
ql/src/java/org/apache/hadoop/hive/ql/io/orc/MemoryManager.java
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestFileDump.java
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestMemoryManager.java
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java
MANAGE HERALD RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/23895/
To: JIRA, omalley
> Implement a memory manager for ORC
> ----------------------------------
>
> Key: HIVE-4248
> URL: https://issues.apache.org/jira/browse/HIVE-4248
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Attachments: HIVE-4248.D9993.1.patch
>
>
> With the large default stripe size (256MB) and dynamic partitions, it is
> quite easy for users to run out of memory when writing ORC files. We probably
> need a solution that keeps track of the total number of concurrent ORC
> writers and divides the available heap space between them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
