Three spindles failing on three different machines could potentially cause
data loss issues if they fail simultaneously-- which would be pretty
uncommon and affect only a small percentage of data. Otherwise if one
machine or individual spindle fails, the blocks that are stored on those
machines
Hi,
I use the default replication factor 3 here, the cluster has 10 nodes, each
of my datanode has 8 hard disks. If one of the nodes is down because of
hardware failure, i.e. the 8 hard disks will no longer be available
immediately during the down time of this machine, does it mean that I will
With rep of 3 you would have to lose 3 entire nodes to lose data. The rep
factor is 3 nodes, not 3 spindles.. The number of disks (sort of) determine
how hdfs spreads io across the spindles for the single copy of the data
(one of 3 nodes with copies) that the node owns. Note that things get
Hi,
If each of my datanode servers has 8 hard disks (a 10-node cluster) and I use
the default replication factor of 3, how will Hadoop handle it when a datanode
with total hardware failure suddenly?
Regards
Arthur
i think hadoop treats a node as a minimal unit, which means a internal failure
will be regards as a node failure.
datanode uses heartbeat to keep block alive, if any block failure was detected
by namenode, it will be recovered
in another datanode.
Yujian Zhang
Southeast University, Nanjing,
Hi Arthur,
In HDFS there will be block level replication, In case of total failure of
a datanode the lost blocks will get under replicated hence the namenode
will create copy of these under replicated blocks on some other datanode.
BR,
Harshit
On Wed, Dec 17, 2014 at 11:35 AM,
