[
https://issues.apache.org/jira/browse/HADOOP-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12500763
]
Doug Cutting commented on HADOOP-1134:
--------------------------------------
Calvin Yu noted on hadoop-user that join() seems to sometimes hang even if the
thread has been interrupted. In other places we use the idiom of a 'running'
flag that's checked in a thread's loop in conjunction with an interrupt, rather
than interrupt+join, and that seems to be reliable. So I think we should
switch to that here to.
Also, in the current patch, I don't see why the thread is held in a field. I
worry that someone might add code like 'if (sortProgressThread == null) ...',
and that we might somehow not always null this field. If it is kept in a local
variable around the call then this is much less of a risk.
So I think we should convert the createProgressThread method to a nested class
whose constructor starts the thread and which has a stop() method that sets a
flag. It would also be good if the 'try' block could be shared between
'collect()' and 'flush()'. I think this calls for a new method something like:
private void sortWithProgress() {
ProgressThread progress = new ProgressThread();
try {
sortAndSpillToDisk();
} finally {
progress.stop();
}
}
> Block level CRCs in HDFS
> ------------------------
>
> Key: HADOOP-1134
> URL: https://issues.apache.org/jira/browse/HADOOP-1134
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Attachments: bc-no-upgrade-05302007.patch,
> DfsBlockCrcDesign-05305007.htm
>
>
> Currently CRCs are handled at FileSystem level and are transparent to core
> HDFS. See recent improvement HADOOP-928 ( that can add checksums to a given
> filesystem ) regd more about it. Though this served us well there a few
> disadvantages :
> 1) This doubles namespace in HDFS ( or other filesystem implementations ). In
> many cases, it nearly doubles the number of blocks. Taking namenode out of
> CRCs would nearly double namespace performance both in terms of CPU and
> memory.
> 2) Since CRCs are transparent to HDFS, it can not actively detect corrupted
> blocks. With block level CRCs, Datanode can periodically verify the checksums
> and report corruptions to namnode such that name replicas can be created.
> We propose to have CRCs maintained for all HDFS data in much the same way as
> in GFS. I will update the jira with detailed requirements and design. This
> will include same guarantees provided by current implementation and will
> include a upgrade of current data.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
