[ https://issues.apache.org/jira/browse/KUDU-3318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17413462#comment-17413462 ]
Yingchun Lai edited comment on KUDU-3318 at 9/11/21, 3:03 AM: -------------------------------------------------------------- Another way to optimize the situation is to compact metadata at runtime, now it is only compact at bootstrap. We can implement it in the future. was (Author: laiyingchun): Another way to optimize the situation is to compact metadata at runtime, now it is only compact at bootstrap. > Log Block Container metadata consumed too much disk space > --------------------------------------------------------- > > Key: KUDU-3318 > URL: https://issues.apache.org/jira/browse/KUDU-3318 > Project: Kudu > Issue Type: Improvement > Components: fs > Reporter: Yingchun Lai > Priority: Major > > In log block container, blocks in .data file are append only, there is a > related append only .metadata file to trace blocks in .data, this type of > entries in metadata are in CREATE type, the other type of entries in metadata > are type of DELETE, it means mark the corresponding CREATE block as deleted. > If there is a pair of CREATE and DELETE entries of a same block id, LBM use > hole punch to reclaim disk space in .data file, but the entries in .metadata > will not be compacted except bootstrap. > Another way to limit metadata is the .data file offset reach its size > limitation(default 10GB), or block number in metadata reach its limitation(no > limit on default). > I found a case in product environment that metadata consumed too many disk > space and near to .data's disk space, it's a waste, and make users confused > and complain that the actual disk space is far more than user's data. -- This message was sent by Atlassian Jira (v8.3.4#803005)