[
https://issues.apache.org/jira/browse/HADOOP-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White updated HADOOP-1187:
------------------------------
Resolution: Fixed
Fix Version/s: 0.13.0
Status: Resolved (was: Patch Available)
I've just committed this. Thanks Dhruba!
> DFS Scalability: avoid scanning entire list of datanodes in
> getAdditionalBlocks
> -------------------------------------------------------------------------------
>
> Key: HADOOP-1187
> URL: https://issues.apache.org/jira/browse/HADOOP-1187
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Fix For: 0.13.0
>
> Attachments: clientPendingCreate2.patch
>
>
> A new block allocations for a file scans the list of all known datanodes to
> find if the client that is a also a cluster node. If so, then it tries to
> allocate a replica locally. This check consumes plenty of CPU, especially if
> the number of datanodes in a cluster is large.
> An optimization: if the client is also a cluster node, then cache a reference
> to the corresponding DatanodeDescriptor from the entry in pendingCreate. The
> method getAdditionalBlock() uses the cached DatanodeDescriptor and thus
> avoids scanning the entire list of datanodes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
