[ https://issues.apache.org/jira/browse/HDFS-12866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wei-Chiu Chuang resolved HDFS-12866. ------------------------------------ Resolution: Duplicate Resolve as it duplicates HDFS-11225 > Recursive delete of a large directory or snapshot makes namenode unresponsive > ----------------------------------------------------------------------------- > > Key: HDFS-12866 > URL: https://issues.apache.org/jira/browse/HDFS-12866 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs > Reporter: Yongjun Zhang > Priority: Major > > Currently file/directory deletion happens in two steps (see > {{FSNamesystem#delete(String src, boolean recursive, boolean logRetryCache)}}: > # Do the following under fsn write lock and release the lock afterwards > ** 1.1 recursively traverse the target, collect INodes and all blocks to be > deleted > ** 1.2 delete all INodes > # Delete the blocks to be deleted incrementally, chunk by chunk. That is, in > a loop, do: > ** acquire fsn write lock, > ** delete chunk of blocks > ** release fsn write lock > Breaking the deletion to two steps is to not hold the fsn write lock for too > long thus making NN not responsive. However, even with this, for deleting > large directory, or deleting snapshot that has a lot of contents, step 1 > itself would takes long time thus still hold the fsn write lock for too long > and make NN not responsive. > A possible solution would be to add one more sub step in step 1, and only > hold fsn write lock in sub step 1.1: > * 1.1. hold the fsn write lock, disconnect the target to be deleted from its > parent dir, release the lock > * 1.2 recursively traverse the target, collect INodes and all blocks to be > deleted > * 1.3 delete all INodes > Then do step 2. > This means, any operations on any file/dir need to check if its ancestor is > deleted (ancestor is disconnected), similar to what's done in > FSNamesystem#isFileDeleted method. > I'm throwing the thought here for further discussion. Welcome comments and > inputs. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org