[jira] [Commented] (HDFS-3791) Backport HDFS-173 to Branch-1 : Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes
[ https://issues.apache.org/jira/browse/HDFS-3791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13501502#comment-13501502 ] Matt Foley commented on HDFS-3791: -- included in branch-1.1 Backport HDFS-173 to Branch-1 : Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes Key: HDFS-3791 URL: https://issues.apache.org/jira/browse/HDFS-3791 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 1.0.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Fix For: 1.1.1 Attachments: HDFS-3791.patch, HDFS-3791.patch, HDFS-3791.patch Backport HDFS-173. see the [comment|https://issues.apache.org/jira/browse/HDFS-2815?focusedCommentId=13422007page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13422007] for more details -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3791) Backport HDFS-173 to Branch-1 : Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes
[ https://issues.apache.org/jira/browse/HDFS-3791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13443848#comment-13443848 ] Uma Maheswara Rao G commented on HDFS-3791: --- @Ted, Yes, I was also in the same lines as Suresh said about that parameter. Not sure about any usecase for getting some advantage by making it configurable. Do you have any usecase, where we may get some advatages by tuning that parameter? If yes, feel free to file a JIRA, we can discuss about it there.Thanks a lot, Ted for taking a look on it. Thanks, Uma Backport HDFS-173 to Branch-1 : Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes Key: HDFS-3791 URL: https://issues.apache.org/jira/browse/HDFS-3791 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 1.0.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Fix For: 1.2.0 Attachments: HDFS-3791.patch, HDFS-3791.patch, HDFS-3791.patch Backport HDFS-173. see the [comment|https://issues.apache.org/jira/browse/HDFS-2815?focusedCommentId=13422007page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13422007] for more details -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3791) Backport HDFS-173 to Branch-1 : Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes
[ https://issues.apache.org/jira/browse/HDFS-3791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13443075#comment-13443075 ] Suresh Srinivas commented on HDFS-3791: --- Uma sorry for the delay in reviewing this. +1 for the patch. Backport HDFS-173 to Branch-1 : Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes Key: HDFS-3791 URL: https://issues.apache.org/jira/browse/HDFS-3791 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 1.0.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Attachments: HDFS-3791.patch, HDFS-3791.patch Backport HDFS-173. see the [comment|https://issues.apache.org/jira/browse/HDFS-2815?focusedCommentId=13422007page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13422007] for more details -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3791) Backport HDFS-173 to Branch-1 : Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes
[
https://issues.apache.org/jira/browse/HDFS-3791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13443103#comment-13443103
]
Uma Maheswara Rao G commented on HDFS-3791:
---
Oh, I have just seen the comments.
{quote}
Uma sorry for the delay in reviewing this. +1 for the patch.
{quote}
No problem :-). Thanks a lot, Suresh for the reviews.
Also thanks for rebasing it. I will to get a patch for HDFS-2815 today in some
time.
Backport HDFS-173 to Branch-1 : Recursively deleting a directory with
millions of files makes NameNode unresponsive for other commands until the
deletion completes
Key: HDFS-3791
URL: https://issues.apache.org/jira/browse/HDFS-3791
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 1.0.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Fix For: 1.2.0
Attachments: HDFS-3791.patch, HDFS-3791.patch, HDFS-3791.patch
Backport HDFS-173.
see the
[comment|https://issues.apache.org/jira/browse/HDFS-2815?focusedCommentId=13422007page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13422007]
for more details
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3791) Backport HDFS-173 to Branch-1 : Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes
[
https://issues.apache.org/jira/browse/HDFS-3791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13443245#comment-13443245
]
Ted Yu commented on HDFS-3791:
--
Currently small deletion is determined by the constant BLOCK_DELETION_INCREMENT:
{code}
+ deleteNow = collectedBlocks.size() = BLOCK_DELETION_INCREMENT;
{code}
I wonder if there is use case where the increment should be configurable.
Backport HDFS-173 to Branch-1 : Recursively deleting a directory with
millions of files makes NameNode unresponsive for other commands until the
deletion completes
Key: HDFS-3791
URL: https://issues.apache.org/jira/browse/HDFS-3791
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 1.0.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Fix For: 1.2.0
Attachments: HDFS-3791.patch, HDFS-3791.patch, HDFS-3791.patch
Backport HDFS-173.
see the
[comment|https://issues.apache.org/jira/browse/HDFS-2815?focusedCommentId=13422007page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13422007]
for more details
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3791) Backport HDFS-173 to Branch-1 : Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes
[ https://issues.apache.org/jira/browse/HDFS-3791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13443296#comment-13443296 ] Suresh Srinivas commented on HDFS-3791: --- when I added this in trunk, I was not sure if there is a usecase. The whole idea was to give up lock once deleting some number of blocks. So the number currently is arbitrary. Backport HDFS-173 to Branch-1 : Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes Key: HDFS-3791 URL: https://issues.apache.org/jira/browse/HDFS-3791 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 1.0.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Fix For: 1.2.0 Attachments: HDFS-3791.patch, HDFS-3791.patch, HDFS-3791.patch Backport HDFS-173. see the [comment|https://issues.apache.org/jira/browse/HDFS-2815?focusedCommentId=13422007page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13422007] for more details -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3791) Backport HDFS-173 to Branch-1 : Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes
[ https://issues.apache.org/jira/browse/HDFS-3791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13439803#comment-13439803 ] Uma Maheswara Rao G commented on HDFS-3791: --- Hi Suresh, could you please take a look on this for review? Backport HDFS-173 to Branch-1 : Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes Key: HDFS-3791 URL: https://issues.apache.org/jira/browse/HDFS-3791 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 1.0.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Attachments: HDFS-3791.patch, HDFS-3791.patch Backport HDFS-173. see the [comment|https://issues.apache.org/jira/browse/HDFS-2815?focusedCommentId=13422007page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13422007] for more details -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3791) Backport HDFS-173 to Branch-1 : Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes
[
https://issues.apache.org/jira/browse/HDFS-3791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436000#comment-13436000
]
Suresh Srinivas commented on HDFS-3791:
---
Uma, the ported patch looks good. One minor issue - {{Perform small deletes
right away}} in FSNamesystem.java should move to the synchronized section. Can
HDFS-2815 patch be applied on top of this?
Backport HDFS-173 to Branch-1 : Recursively deleting a directory with
millions of files makes NameNode unresponsive for other commands until the
deletion completes
Key: HDFS-3791
URL: https://issues.apache.org/jira/browse/HDFS-3791
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 1.0.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Attachments: HDFS-3791.patch
Backport HDFS-173.
see the
[comment|https://issues.apache.org/jira/browse/HDFS-2815?focusedCommentId=13422007page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13422007]
for more details
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3791) Backport HDFS-173 to Branch-1 : Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes
[ https://issues.apache.org/jira/browse/HDFS-3791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436038#comment-13436038 ] Uma Maheswara Rao G commented on HDFS-3791: --- Thanks a lot Suresh for the review! True, I should have put that inside synchronization block for small deletes. As anyway there should not be harm and going remove that part in side lock with HDFS-2815, I did not take that movement seriously. Moved that smaller deletes code inside synchronization. Code wise it should be sync with HDFS-2815 now. as folder structure is diff than trunk it is not possible to apply directly right. SO, I will generate one in HDFS-2815 after you commit this. Thanks Uma Backport HDFS-173 to Branch-1 : Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes Key: HDFS-3791 URL: https://issues.apache.org/jira/browse/HDFS-3791 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 1.0.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Attachments: HDFS-3791.patch, HDFS-3791.patch Backport HDFS-173. see the [comment|https://issues.apache.org/jira/browse/HDFS-2815?focusedCommentId=13422007page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13422007] for more details -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3791) Backport HDFS-173 to Branch-1 : Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes
[ https://issues.apache.org/jira/browse/HDFS-3791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13434209#comment-13434209 ] Uma Maheswara Rao G commented on HDFS-3791: --- Hi Suresh, I have just attached a back-port patch here. Could you please take a look. Backport HDFS-173 to Branch-1 : Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes Key: HDFS-3791 URL: https://issues.apache.org/jira/browse/HDFS-3791 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 1.0.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Attachments: HDFS-3791.patch Backport HDFS-173. see the [comment|https://issues.apache.org/jira/browse/HDFS-2815?focusedCommentId=13422007page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13422007] for more details -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
