[
https://issues.apache.org/jira/browse/HIVE-16904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis updated HIVE-16904:
---------------------------------------
Fix Version/s: (was: 3.2.0)
I cleared the fixVersion field since this ticket is still open. Please review
this ticket and if the fix is already committed to a specific version please
set the version accordingly and mark the ticket as RESOLVED.
According to the [JIRA
guidelines|https://cwiki.apache.org/confluence/display/Hive/HowToContribute]
the fixVersion should be set only when the issue is resolved/closed.
> during repl load for large number of partitions the metadata file can be huge
> and can lead to out of memory
> ------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-16904
> URL: https://issues.apache.org/jira/browse/HIVE-16904
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
> Affects Versions: 3.0.0
> Reporter: Anishek Agarwal
> Assignee: Anishek Agarwal
> Priority: Major
>
> the metadata pertaining to a table + its partitions is stored in a single
> file, During repl load all the data is loaded in memory in one shot and then
> individual partitions processed. This can lead to huge memory overhead as the
> entire file is read in memory. try to deserialize the partition objects with
> some sort of streaming json deserializer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
