[ https://issues.apache.org/jira/browse/HDFS-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138267#comment-13138267 ]
Hudson commented on HDFS-2500: ------------------------------ Integrated in Hadoop-Hdfs-0.23-Build #53 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/53/]) HDFS-2500. Avoid file system operations in BPOfferService thread while processing deletes. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190072 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetAsyncDiskService.java > Avoid file system operations in BPOfferService thread while processing deletes > ------------------------------------------------------------------------------ > > Key: HDFS-2500 > URL: https://issues.apache.org/jira/browse/HDFS-2500 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node > Affects Versions: 0.23.0 > Reporter: Todd Lipcon > Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2500-1.patch, hdfs-2500-branch-0.23.txt, > hdfs-2500.txt > > > While running a workload with concurrent writes and deletes, I saw a lot of > NotReplicatedYetExceptions being thrown due to late arrivals of blockReceived > reports from the DN. Looking at the DN logs, I found that the blockReceived > message was being delayed as much as 15 seconds because the OfferService > thread was blocked on file system operations processing deletes. We > previously moved the deletions to another thread, but it still accesses the > file system to determine the block length in the main thread. On a heavily > loaded system this can take a long time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira