Brian Bockelman wrote:
Hello all,
I'd like to take the datanode's capability to handle multiple
directories to a somewhat-extreme, and get feedback on how well this
might work.
We have a few large RAID servers (12 to 48 disks) which we'd like to
transition to Hadoop. I'd like to mount each of the disks individually
(i.e., /mnt/disk1, /mnt/disk2, ....) and take advantage of Hadoop's
replication - instead of pay the overhead to set up a RAID and still
have to pay the overhead of replication.
In my experience this is the right way to go.
However, we're a bit concerned about how well Hadoop might handle one of
the directories disappearing from underneath it. If a single volume,
say, /mnt/disk1 starts returning I/O errors, is Hadoop smart enough to
figure out that this whole volume is broken? Or will we have to restart
the datanode after any disk failure for it to search the directory
realize everything is broken? What happens if you start up the datanode
with a data directory that it can't write into?
In current implementation if at any point Datanode detects an unwritable or
unreadable drive it shuts itself down logging a message what went wrong and
reporting the problem to the name-node.
So yes if such thing happens you will have to restart the data-node.
But since the cluster takes care of data-node failures by re-replicating
lost blocks that should not be a problem.
Is anyone running in this fashion (i.e., multiple data directories
corresponding to different disk volumes ... even better if you're doing
it with more than a few disks)?
We have a large experience running 4 drives per data-node (no RAID).
So this is not something new or untested.
Thanks,
--Konstantin
