[
https://issues.apache.org/jira/browse/HDFS-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137861#comment-13137861
]
Hudson commented on HDFS-2500:
------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #65 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/65/])
HDFS-2500. Avoid file system operations in BPOfferService thread while
processing deletes. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190072
Files :
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetAsyncDiskService.java
> Avoid file system operations in BPOfferService thread while processing deletes
> ------------------------------------------------------------------------------
>
> Key: HDFS-2500
> URL: https://issues.apache.org/jira/browse/HDFS-2500
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: hdfs-2500-1.patch, hdfs-2500-branch-0.23.txt,
> hdfs-2500.txt
>
>
> While running a workload with concurrent writes and deletes, I saw a lot of
> NotReplicatedYetExceptions being thrown due to late arrivals of blockReceived
> reports from the DN. Looking at the DN logs, I found that the blockReceived
> message was being delayed as much as 15 seconds because the OfferService
> thread was blocked on file system operations processing deletes. We
> previously moved the deletions to another thread, but it still accesses the
> file system to determine the block length in the main thread. On a heavily
> loaded system this can take a long time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
