[
https://issues.apache.org/jira/browse/HADOOP-659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465323
]
Konstantin Shvachko commented on HADOOP-659:
--------------------------------------------
Name-node needs some form of the global map of blocks in order to be able to
remove
blocks from neededReplications e.g. during block reports.
We should also use a parallel list view of blocks that lists them in the
priority order.
Since we plan to have only 2 priority levels, we can add first-priority items
to the
beginning of the list and low-priority to the end. That way when we iterate the
list
we will first choose higher priority blocks automatically.
Add and remove to/from the list are both O(1), so the performance remains the
same as we have now.
Iterating through the list is linear, which is better than what we have now O(n
log n).
I agree the main purpose of the issue is to replicate blocks that are about to
become extinct.
But if we can improve map/reduce performance by treating replication of config
and jar
files similarly, why not do that?
> Boost the priority of re-replicating blocks that are far from their
> replication target
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-659
> URL: https://issues.apache.org/jira/browse/HADOOP-659
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.7.2
> Reporter: Konstantin Shvachko
> Assigned To: Hairong Kuang
>
> I see two types of replications that should be accelerated compared to all
> others.
> 1. Blocks that have only one remaining copy (but are required to have higher
> replication).
> 2. Blocks that have less than 1/3 of their replicas in place.
> The latter occurs when map/reduce sets replication of certain files to 10,
> and we want
> it happen fast to achieve better performance on the tasks.
> So I think we should distinguish two major groups of under-replicated blocks:
> first-priority (having only 1 copy or less than 1/3 of required replicas),
> and the rest.
> The name-node places first-priority blocks into the beginning of the
> neededReplication
> list, and the rest are placed at the end. That way the first-priority blocks
> will be replicated
> first and then the others.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
