[ https://issues.apache.org/jira/browse/HDFS-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2500: ------------------------------ Target Version/s: 0.23.0 Status: Patch Available (was: Open) > Avoid file system operations in BPOfferService thread while processing deletes > ------------------------------------------------------------------------------ > > Key: HDFS-2500 > URL: https://issues.apache.org/jira/browse/HDFS-2500 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node > Affects Versions: 0.23.0 > Reporter: Todd Lipcon > Assignee: Todd Lipcon > Attachments: hdfs-2500.txt > > > While running a workload with concurrent writes and deletes, I saw a lot of > NotReplicatedYetExceptions being thrown due to late arrivals of blockReceived > reports from the DN. Looking at the DN logs, I found that the blockReceived > message was being delayed as much as 15 seconds because the OfferService > thread was blocked on file system operations processing deletes. We > previously moved the deletions to another thread, but it still accesses the > file system to determine the block length in the main thread. On a heavily > loaded system this can take a long time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira