[ https://issues.apache.org/jira/browse/HBASE-26437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17441416#comment-17441416 ]
Josh Elser commented on HBASE-26437: ------------------------------------ I have a unit test which reproduces this (naively) but am making sure all of the contract tests are still passing. > [hboss] Rename does not clean up znodes for src location > -------------------------------------------------------- > > Key: HBASE-26437 > URL: https://issues.apache.org/jira/browse/HBASE-26437 > Project: HBase > Issue Type: Bug > Components: hboss > Affects Versions: hbase-filesystem-1.0.0-alpha1 > Reporter: Josh Elser > Assignee: Josh Elser > Priority: Critical > > We ran into a fun situation where the partition hosting ZK data was > repeatedly filling up while heavy ExportSnapshot+clone_snapshot operations > were running (10's of TB). The cluster was previously working just fine. > Upon investigation of the ZK tree, we found a large number of znodes beneath > /hboss, specifically many in the corresponding ZK HBOSS path for > $hbase.rootdir/.tmp. > Tracing back from the code, we saw that the CloneSnapshotProcedure (like > CreateTableProcedure) will create the table filesystem layout in > $hbase.rootdir/.tmp and then rename it into $hbase.rootdir/data/<namespace>. > However, it appears that, upon rename, HBOSS was not cleaning up the src > path's znode. This is a bug as it allows ZK to grow unbounded (which explains > why this problem slowly arose and not suddenly). > As a workaround, HBase can be stopped and the corresponding ZK path for > $hbase.rootdir/.tmp can be cleaned up to reclaim 1/2 the space taken up by > znodes for imported hbase tables (we would still have znodes for > $hbase.rootdir/data/...) -- This message was sent by Atlassian Jira (v8.20.1#820001)