[
https://issues.apache.org/jira/browse/HIVE-9658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-9658:
------------------------------
Summary: Reduce parquet memory usage by bypassing java primitive objects on
ETypeConverter (was: Reduce parquet memory use by bypassing java primitive
objects on ETypeConverter)
> Reduce parquet memory usage by bypassing java primitive objects on
> ETypeConverter
> ---------------------------------------------------------------------------------
>
> Key: HIVE-9658
> URL: https://issues.apache.org/jira/browse/HIVE-9658
> Project: Hive
> Issue Type: Sub-task
> Reporter: Sergio Peña
> Assignee: Sergio Peña
> Attachments: HIVE-9658.1.patch, HIVE-9658.2.patch, HIVE-9658.3.patch,
> HIVE-9658.4.patch, HIVE-9658.5.patch
>
>
> NO PRECOMMIT TESTS
> The ETypeConverter class passes Writable objects to the collection converters
> in order to be read later by the map/reduce functions. These objects are all
> wrapped in a unique ArrayWritable object.
> We can save some memory by returning the java primitive objects instead in
> order to prevent memory allocation. The only writable object needed by
> map/reduce is ArrayWritable. If we create another writable class where to
> store primitive objects (Object), then we can stop using all primitive
> wirtables.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
