[
https://issues.apache.org/jira/browse/HDFS-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886430#action_12886430
]
Konstantin Shvachko commented on HDFS-1094:
-------------------------------------------
The math looks good to me (in pdf file).
> Data loss probability P depends on time T.
Here the assumption is, correct me if it's wrong, that f nodes fail
simultaneously. Otherwise, we should take into account replication process,
which will be restoring some blocks while other nodes are still up, decreasing
the probability of data loss. Probability of losing f nodes simultaneously at a
particular moment does not depend on time. The probability of a simultaneous
failure of f nodes during a specific period of time depends on the length of
the period. So if you choose the parameter p in the document correctly
(depending on the time period), then you get the the probability of a data loss
during this period of time.
The assumption p = 0.01 or 0.001 seems arbitrary, but it probably does not
matter as you compare different strategies with the same value.
What is missing in the analysis is that the probability of loosing a whole rack
is much higher than the probability of loosing any 20 machines in the cluster.
It should be actually equivalent to the probability of loosing one machine,
because you loose one switch and the whole rack is out.
And that was one of the main reasons why we decided to replicate off rack.
Rodrigo, did I understand correctly that your idea is to experiment with
replication within the rack, that is, all replicas are placed on different
machines in the same rack?
> Intelligent block placement policy to decrease probability of block loss
> ------------------------------------------------------------------------
>
> Key: HDFS-1094
> URL: https://issues.apache.org/jira/browse/HDFS-1094
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Attachments: prob.pdf, prob.pdf
>
>
> The current HDFS implementation specifies that the first replica is local and
> the other two replicas are on any two random nodes on a random remote rack.
> This means that if any three datanodes die together, then there is a
> non-trivial probability of losing at least one block in the cluster. This
> JIRA is to discuss if there is a better algorithm that can lower probability
> of losing a block.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
