Currently, Block reports are computed by scanning all the files and folders in the local disk. This happens not only in startup, but also in periodic block reports.
What is the intention behind doing this way instead of creating the block report from the data structure already located in the memory? One issue with the current approach is that it takes quite some time. For instance, the system I am working with has 1 million blocks in each data node and every time block report is computed, it takes about 4 minutes. You might expect this may not any effect. But there is. The problem is the computation and sending of block reports and sending the list of the new received blocks are part of the same thread. As a result, when the block report computation takes a long time, it delays the reporting of new received blocks. As you know when you are writing a multi block file and you request a new block from the namenode, the namenode checks whether the very previous block is replicated enough number of times. If at this point the data node is computing the block report, it is very likely that it didn't inform the namenode about the very previous block yet. As a result, the namenode rejects this. Then client tries to repeat this 5 more times while doubling the wait time in between up to about 6 seconds (starts with 200ms and doubles it 5 times). Then, client raises an exception to the application. Given that the block report computation takes minutes, this situation is very likely to occur. Do you have any suggestions on how to handle this situation? Are there any plans to take block reporting code segment and reporting of new received blocks to separate threads (ref: offerService method of DataNode.java file)? Thanks for your response, -- ------------ Best Regards, Cagdas Evren Gerede Home Page: http://cagdasgerede.info
