You bring up a good point. The creating and processing of block reports
do take a lot of resources. It affects DFS Scalability and performance
to some extent. Here are some more details:
http://issues.apache.org/jira/browse/HADOOP-1079

 

There is one thread in the Datanode that sends block confirmations to
the Namenode. The same thread computes the block report and sends it to
Namenode. This sequential nature is critical in ensuring that there is
no erroneous race condition in the Namenode. See
http://issues.apache.org/jira/browse/HADOOP-1135

 

The reason block reports are there is to detect inconsistencies if
blocks get deleted underneath the Datanode process. For example, an
administrator can erroneously remove blk_xxx files. Also, a disk
corruption can make a section of blk_xxx files unreadable. This is the
reason block reports exists.

 

My current thinking is that "block report processing" should compare the
blkxxx files on disk with the data structure in the Datanode memory. If
and only if there is some discrepancy between these two, then a block
report be sent to the Namenode. If we do this, then we will practically
get rid of 99% of block reports.

 

Thanks,

dhruba

 

________________________________

From: Cagdas Gerede [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, April 29, 2008 11:32 PM
To: [email protected]
Cc: dhruba Borthakur
Subject: Block reports: memory vs. file system, and Dividing
offerService into 2 threads

 

Currently,

 

Block reports are computed by scanning all the files and folders in the
local disk.

This happens not only in startup, but also in periodic block reports.

 

What is the intention behind doing this way instead of creating the
block report from the data structure already located in the memory?

One issue with the current approach is that it takes quite some time.
For instance, the system I am working with has 1 million blocks in each
data node and every time block report is computed, it takes about 4
minutes. You might expect this may not any effect. But there is. The
problem is the computation and sending of block reports and sending the
list of the new received blocks are part of the same thread. As a
result, when the block report computation takes a long time, it delays
the reporting of new received blocks. 

As you know when you are writing a multi block file and you request a
new block from the namenode, the namenode checks whether the very
previous block is replicated enough number of times. If at this point
the data node is computing the block report, it is very likely that it
didn't inform the namenode about the very previous block yet. As a
result, the namenode rejects this. Then client tries to repeat this 5
more times while doubling the wait time in between up to about 6 seconds
(starts with 200ms and doubles it 5 times). Then, client raises an
exception to the application. Given that the block report computation
takes minutes, this situation is very likely to occur.

 

Do you have any suggestions on how to handle this situation?

Are there any plans to take block reporting code segment and reporting
of new received blocks to separate threads (ref: offerService method of
DataNode.java file)?

 

Thanks for your response,

 


-- 
------------
Best Regards, Cagdas Evren Gerede
Home Page: http://cagdasgerede.info <http://cagdasgerede.info/>  

Reply via email to