Cagdas Gerede wrote:
Currently,
Block reports are computed by scanning all the files and folders in the
local disk.
This happens not only in startup, but also in periodic block reports.
What is the intention behind doing this way instead of creating the block
report from the data structure already located in the memory?
We should improve this. We could certainly rely on in-memory map. Block
reports is only half of the problem with scanning the local fs. The
other half is 'du', also performed in line. 'du' is even worse if you
have 'previous' directory on DataNode (i.e. non-finalized DFS).
See http://issues.apache.org/jira/browse/HADOOP-3232
We have to improve both and make sure these scans are either reduced or
removed and not block regular DataNode functions.
As Dhruba mentioned, only reason local FS scanning is done is to detect
if the files somehow disappeared. But this is strictly not required.
There are other parts that check this now (periodic block verifier, for
e.g.). As such we have not see such a problem till now. I vote for not
scanning the local fs.
This leaves us with the problem with 'du'.
Raghu.
One issue with the current approach is that it takes quite some time. For
instance, the system I am working with has 1 million blocks in each data
node and every time block report is computed, it takes about 4 minutes. You
might expect this may not any effect. But there is. The problem is the
computation and sending of block reports and sending the list of the new
received blocks are part of the same thread. As a result, when the block
report computation takes a long time, it delays the reporting of new
received blocks.
As you know when you are writing a multi block file and you request a new
block from the namenode, the namenode checks whether the very previous block
is replicated enough number of times. If at this point the data node is
computing the block report, it is very likely that it didn't inform the
namenode about the very previous block yet. As a result, the namenode
rejects this. Then client tries to repeat this 5 more times while doubling
the wait time in between up to about 6 seconds (starts with 200ms and
doubles it 5 times). Then, client raises an exception to the application.
Given that the block report computation takes minutes, this situation is
very likely to occur.
Do you have any suggestions on how to handle this situation?
Are there any plans to take block reporting code segment and reporting of
new received blocks to separate threads (ref: offerService method of
DataNode.java file)?
Thanks for your response,