[
https://issues.apache.org/jira/browse/HIVE-30217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sanqingleo updated HIVE-30217:
------------------------------
Description:
Hi,
We build a wrong partition-table that has nearly a half million partitions.
And our data developer do alter action, rename table. And it lead HMS OOM.
I dive into source code, and find that alter action will fetch all partition
meta data into memory. both 3.1.x and 4.1.x version
here is 4.1.x source code.
And I have a question, Why must fetch all meta data in memory at once, because
handle rename and other actions are use batch model. and do commit in once. We
needn't fetch all meta data in once, isn't it? We can fetch in-batch and after
finishing all handle ,do once commit.
Why write in-batch but read all in once?
!image-2026-10-27-15-58-03-412.png|width=709,height=308!
was:
Hi,
We build a wrong partition-table that has nearly a half billion partitions.
And our data developer do alter action, rename table. And it lead HMS OOM.
I dive into source code, and find that alter action will fetch all partition
meta data into memory. both 3.1.x and 4.1.x version
here is 4.1.x source code.
And I have a question, Why must fetch all meta data in memory at once, because
handle rename and other actions are use batch model. and do commit in once. We
needn't fetch all meta data in once, isn't it? We can fetch in-batch and after
finishing all handle ,do once commit.
Why write in-batch but read all in once?
!image-2026-10-27-15-58-03-412.png|width=709,height=308!
> Alter internal Table cause OOM
> ------------------------------
>
> Key: HIVE-30217
> URL: https://issues.apache.org/jira/browse/HIVE-30217
> Project: Hive
> Issue Type: Bug
> Environment: hive 3.1.2
> Reporter: sanqingleo
> Priority: Major
> Fix For: 3.1.2, 4.1.0
>
> Attachments: image-2026-10-27-15-58-03-412.png
>
>
> Hi,
> We build a wrong partition-table that has nearly a half million partitions.
> And our data developer do alter action, rename table. And it lead HMS OOM.
> I dive into source code, and find that alter action will fetch all partition
> meta data into memory. both 3.1.x and 4.1.x version
> here is 4.1.x source code.
> And I have a question, Why must fetch all meta data in memory at once,
> because handle rename and other actions are use batch model. and do commit in
> once. We needn't fetch all meta data in once, isn't it? We can fetch
> in-batch and after finishing all handle ,do once commit.
> Why write in-batch but read all in once?
> !image-2026-10-27-15-58-03-412.png|width=709,height=308!
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
