[
https://issues.apache.org/jira/browse/HDFS-173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated HDFS-173:
---------------------------------
Attachment: HDFS-173.patch
The patch changes the following that needs closer review:
# When a large directory is deleted, the target directory is unlinked from the
directory structure first. The deletion is recorded immediately in the editslog.
# Modification time (not sure if this is important) for deletion of
subdirectories and file is the time when delete operation was received.
# Number of inodes to delete at a time is chosen as 1000 arbitrarily.
bq. What about other references to files under the tree. E.g. if a file is
still being written, there are references to it from lease manager, that could
be an inconsistent view of the namesystem. I think this one can be handled
I need more information about this. I do not think this patch is handling it
right, currently. I will talk to you.
bq. What about log entry? Is the deletion log written at the beginning or the
when the whole deletion is completed?
At the beginning
bq. Currently can recursive deletes fail for any reason in the middle (and are
those rolled back?)?
Current code does not seem to roll back the change if a deletion fails for some
reason in the middle. This may be not be a problem because deletion of blocks
is done towards the end and hence no data is lost on failures.
bq. What is is the suspect for taking so long? Is it mainly the memory
allocations?
On my test setup deletion of files takes much shorter time than what was
observed on production cluster. 20 seconds for deleting 1 million files.
> Recursively deleting a directory with millions of files makes NameNode
> unresponsive for other commands until the deletion completes
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-173
> URL: https://issues.apache.org/jira/browse/HDFS-173
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Attachments: HDFS-173.patch
>
>
> Delete a directory with millions of files. This could take several minutes
> (observed 12 mins for 9 million files). While the operation is in progress
> FSNamesystem lock is held and the requests from clients are not handled until
> deletion completes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
