Sushanth Sowmyan created HIVE-9359:
--------------------------------------
Summary: Export of a large table causes OOM in Metastore and Client
Key: HIVE-9359
URL: https://issues.apache.org/jira/browse/HIVE-9359
Project: Hive
Issue Type: Bug
Components: Import/Export, Metastore
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan
Running hive export on a table with a large number of partitions winds up
making the metastore and client run out of memory. The number of places we wind
up having a copy of the entire partitions object wind up being as follows:
Metastore
* (temporarily) Metastore MPartition objects
* List<Partition> that gets persisted before sending to thrift
* thrift copy of all of those partitions
Client side
* thrift copy of partitions
* deepcopy of above to create List<Partition> objects
* JSONObject that contains all of those above partition objects
* List<ReadEntity> which each encapsulates the aforesaid partition objects.
This memory usage needs to be drastically reduced.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
