[jira] [Updated] (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
[ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-611: -- Target Version/s: 0.20.205.1 Fix Version/s: 0.20.205.1 Heartbeats times from Datanodes increase when there are plenty of blocks to delete -- Key: HDFS-611 URL: https://issues.apache.org/jira/browse/HDFS-611 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.20.1, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Zheng Shao Fix For: 0.20.205.1, 0.21.0 Attachments: HDFS-611-branch-0.20-security.patch, HDFS-611.branch-19.patch, HDFS-611.branch-19.v2.patch, HDFS-611.branch-20.patch, HDFS-611.branch-20.v2.patch, HDFS-611.branch-20.v6.patch, HDFS-611.trunk.patch, HDFS-611.trunk.v2.patch, HDFS-611.trunk.v3.patch, HDFS-611.trunk.v4.patch, HDFS-611.trunk.v5.patch, HDFS-611.trunk.v6.patch I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
[ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-611: -- Attachment: HDFS-611-branch-0.20-security.patch Patch for branch-0.20-security is attached. Heartbeats times from Datanodes increase when there are plenty of blocks to delete -- Key: HDFS-611 URL: https://issues.apache.org/jira/browse/HDFS-611 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.20.1, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Zheng Shao Fix For: 0.21.0 Attachments: HDFS-611-branch-0.20-security.patch, HDFS-611.branch-19.patch, HDFS-611.branch-19.v2.patch, HDFS-611.branch-20.patch, HDFS-611.branch-20.v2.patch, HDFS-611.branch-20.v6.patch, HDFS-611.trunk.patch, HDFS-611.trunk.v2.patch, HDFS-611.trunk.v3.patch, HDFS-611.trunk.v4.patch, HDFS-611.trunk.v5.patch, HDFS-611.trunk.v6.patch I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
[ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated HDFS-611: Attachment: HDFS-611.branch-20.v6.patch Attaching the patch for hadoop 0.20. Although we don't need it for apache hadoop 0.20, we can include it into Yahoo and Cloudera's hadoop 0.20 distribution if it approves to be useful. Heartbeats times from Datanodes increase when there are plenty of blocks to delete -- Key: HDFS-611 URL: https://issues.apache.org/jira/browse/HDFS-611 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.20.1, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Zheng Shao Fix For: 0.22.0 Attachments: HDFS-611.branch-19.patch, HDFS-611.branch-19.v2.patch, HDFS-611.branch-20.patch, HDFS-611.branch-20.v2.patch, HDFS-611.branch-20.v6.patch, HDFS-611.trunk.patch, HDFS-611.trunk.v2.patch, HDFS-611.trunk.v3.patch, HDFS-611.trunk.v4.patch, HDFS-611.trunk.v5.patch, HDFS-611.trunk.v6.patch I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
[ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated HDFS-611: Attachment: HDFS-611.trunk.v6.patch Moved ReplicaFileDeletionTask and deleteAsync into FSDatasetAsyncDiskService. Heartbeats times from Datanodes increase when there are plenty of blocks to delete -- Key: HDFS-611 URL: https://issues.apache.org/jira/browse/HDFS-611 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.20.1, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Zheng Shao Fix For: 0.20.2, 0.21.0, 0.22.0 Attachments: HDFS-611.branch-19.patch, HDFS-611.branch-19.v2.patch, HDFS-611.branch-20.patch, HDFS-611.branch-20.v2.patch, HDFS-611.trunk.patch, HDFS-611.trunk.v2.patch, HDFS-611.trunk.v3.patch, HDFS-611.trunk.v4.patch, HDFS-611.trunk.v5.patch, HDFS-611.trunk.v6.patch I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
[ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated HDFS-611: Status: Open (was: Patch Available) Heartbeats times from Datanodes increase when there are plenty of blocks to delete -- Key: HDFS-611 URL: https://issues.apache.org/jira/browse/HDFS-611 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.20.1, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Zheng Shao Fix For: 0.20.2, 0.21.0, 0.22.0 Attachments: HDFS-611.branch-19.patch, HDFS-611.branch-19.v2.patch, HDFS-611.branch-20.patch, HDFS-611.branch-20.v2.patch, HDFS-611.trunk.patch, HDFS-611.trunk.v2.patch, HDFS-611.trunk.v3.patch, HDFS-611.trunk.v4.patch, HDFS-611.trunk.v5.patch, HDFS-611.trunk.v6.patch I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
[ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated HDFS-611: Status: Open (was: Patch Available) Heartbeats times from Datanodes increase when there are plenty of blocks to delete -- Key: HDFS-611 URL: https://issues.apache.org/jira/browse/HDFS-611 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.20.1, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Zheng Shao Fix For: 0.20.2, 0.21.0, 0.22.0 Attachments: HDFS-611.branch-19.patch, HDFS-611.branch-19.v2.patch, HDFS-611.branch-20.patch, HDFS-611.branch-20.v2.patch, HDFS-611.trunk.patch, HDFS-611.trunk.v2.patch, HDFS-611.trunk.v3.patch, HDFS-611.trunk.v4.patch I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
[ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated HDFS-611: Status: Patch Available (was: Open) Re-trigger unit tests. Heartbeats times from Datanodes increase when there are plenty of blocks to delete -- Key: HDFS-611 URL: https://issues.apache.org/jira/browse/HDFS-611 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.20.1, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Zheng Shao Fix For: 0.20.2, 0.21.0, 0.22.0 Attachments: HDFS-611.branch-19.patch, HDFS-611.branch-19.v2.patch, HDFS-611.branch-20.patch, HDFS-611.branch-20.v2.patch, HDFS-611.trunk.patch, HDFS-611.trunk.v2.patch, HDFS-611.trunk.v3.patch, HDFS-611.trunk.v4.patch I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
[ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated HDFS-611: Attachment: HDFS-611.trunk.v4.patch Heartbeats times from Datanodes increase when there are plenty of blocks to delete -- Key: HDFS-611 URL: https://issues.apache.org/jira/browse/HDFS-611 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.20.1, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Zheng Shao Fix For: 0.20.2, 0.21.0, 0.22.0 Attachments: HDFS-611.branch-19.patch, HDFS-611.branch-19.v2.patch, HDFS-611.branch-20.patch, HDFS-611.branch-20.v2.patch, HDFS-611.trunk.patch, HDFS-611.trunk.v2.patch, HDFS-611.trunk.v3.patch, HDFS-611.trunk.v4.patch I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
[ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated HDFS-611: Attachment: (was: HDFS-611.trunk.v4.patch) Heartbeats times from Datanodes increase when there are plenty of blocks to delete -- Key: HDFS-611 URL: https://issues.apache.org/jira/browse/HDFS-611 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.20.1, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Zheng Shao Fix For: 0.20.2, 0.21.0, 0.22.0 Attachments: HDFS-611.branch-19.patch, HDFS-611.branch-19.v2.patch, HDFS-611.branch-20.patch, HDFS-611.branch-20.v2.patch, HDFS-611.trunk.patch, HDFS-611.trunk.v2.patch, HDFS-611.trunk.v3.patch, HDFS-611.trunk.v4.patch I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
[ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated HDFS-611: Status: Open (was: Patch Available) Heartbeats times from Datanodes increase when there are plenty of blocks to delete -- Key: HDFS-611 URL: https://issues.apache.org/jira/browse/HDFS-611 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.20.1, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Zheng Shao Fix For: 0.20.2, 0.21.0, 0.22.0 Attachments: HDFS-611.branch-19.patch, HDFS-611.branch-19.v2.patch, HDFS-611.branch-20.patch, HDFS-611.branch-20.v2.patch, HDFS-611.trunk.patch, HDFS-611.trunk.v2.patch, HDFS-611.trunk.v3.patch I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
[ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated HDFS-611: Attachment: HDFS-611.trunk.v3.patch Added a test case. Also changed the maximum number of block invalidation commands per heartbeat to 1000. That will make heartbeat response size ~ 100K, which maps to ~1ms on a 1Gbps NIC. This in turn maps to the capability of serving ~3000 nodes with 3 second heartbeat latency. Heartbeats times from Datanodes increase when there are plenty of blocks to delete -- Key: HDFS-611 URL: https://issues.apache.org/jira/browse/HDFS-611 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.20.1, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Zheng Shao Fix For: 0.20.2, 0.21.0, 0.22.0 Attachments: HDFS-611.branch-19.patch, HDFS-611.branch-19.v2.patch, HDFS-611.branch-20.patch, HDFS-611.branch-20.v2.patch, HDFS-611.trunk.patch, HDFS-611.trunk.v2.patch, HDFS-611.trunk.v3.patch I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
[ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated HDFS-611: Attachment: HDFS-611.branch-19.patch This patch adds a new class BlockFileDeleter, which delete block files asynchronously using one ThreadPoolExecutor for each volume. BlockFileDeleter is used internally in FSDataset. Heartbeats times from Datanodes increase when there are plenty of blocks to delete -- Key: HDFS-611 URL: https://issues.apache.org/jira/browse/HDFS-611 Project: Hadoop HDFS Issue Type: Bug Components: data-node Reporter: dhruba borthakur Assignee: Zheng Shao Attachments: HDFS-611.branch-19.patch I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
[ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated HDFS-611: Attachment: HDFS-611.trunk.patch Heartbeats times from Datanodes increase when there are plenty of blocks to delete -- Key: HDFS-611 URL: https://issues.apache.org/jira/browse/HDFS-611 Project: Hadoop HDFS Issue Type: Bug Components: data-node Reporter: dhruba borthakur Assignee: Zheng Shao Attachments: HDFS-611.branch-19.patch, HDFS-611.branch-20.patch, HDFS-611.trunk.patch I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
[ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated HDFS-611: Attachment: HDFS-611.branch-20.patch Heartbeats times from Datanodes increase when there are plenty of blocks to delete -- Key: HDFS-611 URL: https://issues.apache.org/jira/browse/HDFS-611 Project: Hadoop HDFS Issue Type: Bug Components: data-node Reporter: dhruba borthakur Assignee: Zheng Shao Attachments: HDFS-611.branch-19.patch, HDFS-611.branch-20.patch, HDFS-611.trunk.patch I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
[ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated HDFS-611: Fix Version/s: 0.22.0 0.21.0 0.20.2 Affects Version/s: 0.22.0 0.21.0 0.20.1 Status: Patch Available (was: Open) Heartbeats times from Datanodes increase when there are plenty of blocks to delete -- Key: HDFS-611 URL: https://issues.apache.org/jira/browse/HDFS-611 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.20.1, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Zheng Shao Fix For: 0.20.2, 0.21.0, 0.22.0 Attachments: HDFS-611.branch-19.patch, HDFS-611.branch-20.patch, HDFS-611.trunk.patch I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
[ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated HDFS-611: Attachment: HDFS-611.trunk.v2.patch Heartbeats times from Datanodes increase when there are plenty of blocks to delete -- Key: HDFS-611 URL: https://issues.apache.org/jira/browse/HDFS-611 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.20.1, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Zheng Shao Fix For: 0.20.2, 0.21.0, 0.22.0 Attachments: HDFS-611.branch-19.patch, HDFS-611.branch-19.v2.patch, HDFS-611.branch-20.patch, HDFS-611.branch-20.v2.patch, HDFS-611.trunk.patch, HDFS-611.trunk.v2.patch I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
[ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated HDFS-611: Status: Patch Available (was: Open) Heartbeats times from Datanodes increase when there are plenty of blocks to delete -- Key: HDFS-611 URL: https://issues.apache.org/jira/browse/HDFS-611 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.20.1, 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Zheng Shao Fix For: 0.20.2, 0.21.0, 0.22.0 Attachments: HDFS-611.branch-19.patch, HDFS-611.branch-19.v2.patch, HDFS-611.branch-20.patch, HDFS-611.branch-20.v2.patch, HDFS-611.trunk.patch, HDFS-611.trunk.v2.patch I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
[ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-611: -- Attachment: parallelDelete.txt This patch uses a pool of threads to delete block files on the datanodes. By default, a pool of 5 threads is used but this value canbe overridden by a config parameter hdfs.datanode.delete.threadpool.size. Heartbeats times from Datanodes increase when there are plenty of blocks to delete -- Key: HDFS-611 URL: https://issues.apache.org/jira/browse/HDFS-611 Project: Hadoop HDFS Issue Type: Bug Components: data-node Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: parallelDelete.txt I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
[ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-611: -- Attachment: (was: parallelDelete.txt) Heartbeats times from Datanodes increase when there are plenty of blocks to delete -- Key: HDFS-611 URL: https://issues.apache.org/jira/browse/HDFS-611 Project: Hadoop HDFS Issue Type: Bug Components: data-node Reporter: dhruba borthakur Assignee: dhruba borthakur I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
