[
https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886979#action_12886979
]
Rodrigo Schmidt commented on HDFS-1094:
---------------------------------------
@Konstantin: We have a script that calculates the probabilities. The author is
on vacation, so I can't just upload it here without his consent (will do that
as soon as he agrees). I will post a table with values as soon as I get some
time. Right now I'm trying to finish up the first version of the code and test
it.
@Joydeep:
> a node group is a fixed (at any given time) of nodes
??? I think you missed the most important part of the sentence
I tried to get the meaning from the rest of your text but some things still
look weird. You seem to convey that node groups might be bigger than than the
number of replicas you want to create. If so, how do you choose the replicas
within the node group. In the algorithm I'm implementing it's not the case that
any choice of 3 nodes is valid within what I think you are calling a node group.
> instead of [i, (i+M-1)%racksize] - choose [ (i / (r/M))*r/M, (i / (r/M))*r/M
> + M-1] // fixed offset groups of M nodes each in a rack
And how do you cope with racks of different sizes in this case?
> Intelligent block placement policy to decrease probability of block loss
> ------------------------------------------------------------------------
>
> Key: HDFS-1094
> URL: https://issues.apache.org/jira/browse/HDFS-1094
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Attachments: prob.pdf, prob.pdf
>
>
> The current HDFS implementation specifies that the first replica is local and
> the other two replicas are on any two random nodes on a random remote rack.
> This means that if any three datanodes die together, then there is a
> non-trivial probability of losing at least one block in the cluster. This
> JIRA is to discuss if there is a better algorithm that can lower probability
> of losing a block.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
