[
https://issues.apache.org/jira/browse/HADOOP-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533882
]
Raghu Angadi commented on HADOOP-2012:
--------------------------------------
Chatted with Rob and Dhruba. Here is the proposal :
- No data is stored persistently for now. If we need this we can add later.
- Datanode maintains last verification time for all the nodes in memory.
- When it first starts up, it keeps the blocks in random order. And then on
wards verifies them based on last verfication time. Initial random order is to
avoid not verifying some blocks across frequest restarts.
- Provides detailed stats on webui so that we can know if there any blocks that
are not verified in a log time. This also lets us know we should store any data
persistenly.
- Instead of just deleting a block when a corruption is detected, it could let
the namenode decide whether to remove the block immediately or not. This might
need a few changes on Namenode.
> Periodic verification at the Datanode
> -------------------------------------
>
> Key: HADOOP-2012
> URL: https://issues.apache.org/jira/browse/HADOOP-2012
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
>
> Currently on-disk data corruption on data blocks is detected only when it is
> read by the client or by another datanode. These errors are detected much
> earlier if datanode can periodically verify the data checksums for the local
> blocks.
> Some of the issues to consider :
> - How should we check the blocks ( no more often than once every couple of
> weeks ?)
> - How do we keep track of when a block was last verfied ( there is a .meta
> file associcated with each lock ).
> - What action to take once a corruption is detected
> - Scanning should be done as a very low priority with rest of the datanode
> disk traffic in mind.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
