Currently,

Block reports are computed by scanning all the files and folders in the
local disk.
This happens not only in startup, but also in periodic block reports.

What is the intention behind doing this way instead of creating the block
report from the data structure already located in the memory?
One issue with the current approach is that it takes quite some time. For
instance, the system I am working with has 1 million blocks in each data
node and every time block report is computed, it takes about 4 minutes. You
might expect this may not any effect. But there is. The problem is the
computation and sending of block reports and sending the list of the new
received blocks are part of the same thread. As a result, when the block
report computation takes a long time, it delays the reporting of new
received blocks.
As you know when you are writing a multi block file and you request a new
block from the namenode, the namenode checks whether the very previous block
is replicated enough number of times. If at this point the data node is
computing the block report, it is very likely that it didn't inform the
namenode about the very previous block yet. As a result, the namenode
rejects this. Then client tries to repeat this 5 more times while doubling
the wait time in between up to about 6 seconds (starts with 200ms and
doubles it 5 times). Then, client raises an exception to the application.
Given that the block report computation takes minutes, this situation is
very likely to occur.

Do you have any suggestions on how to handle this situation?
Are there any plans to take block reporting code segment and reporting of
new received blocks to separate threads (ref: offerService method of
DataNode.java file)?

Thanks for your response,


-- 
------------
Best Regards, Cagdas Evren Gerede
Home Page: http://cagdasgerede.info

Reply via email to