Brian Bockelman wrote:
On May 21, 2009, at 2:01 PM, Raghu Angadi wrote:
I think you should file a jira on this. Most likely this is what is
happening :
* two out of 3 dns can not take anymore blocks.
* While picking nodes for a new block, NN mostly skips the third dn as
well since '# active writes' on it is larger than '2 * avg'.
* Even if there is one other block is being written on the 3rd, it is
still greater than (2 * 1/3).
To test this, if you write just one block to an idle cluster it should
succeed.
Writing from the client on the 3rd dn succeeds since local node is
always favored.
This particular problem is not that severe on a large cluster but HDFS
should do the sensible thing.
Hey Raghu,
If this analysis is right, I would add it can happen even on large
clusters! I've seen this error at our cluster when we're very full
(>97%) and very few nodes have any empty space. This usually happens
because we have two very large nodes (10x bigger than the rest of the
cluster), and HDFS tends to distribute writes randomly -- meaning the
smaller nodes fill up quickly, until the balancer can catch up.
Yes. This would bite when ever a large portion of nodes can not accept
blocks. In general can happen whenever less than half the nodes have any
space left.
Raghu.
