[
https://issues.apache.org/jira/browse/HADOOP-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663042#action_12663042
]
dhruba borthakur commented on HADOOP-4663:
------------------------------------------
Now that the 0.18.3 issues are handled by HADOOP-4997, let's discuss what we
need to do for 0.19 and above. I propose that blocks that are created by
client-writes be created in the "tmp" directory whereas blocks created by
replcation requests be created in tmp_replication directory. On datanode
restarts, the blocks in the "tmp" directory are reclaimed whereas the blocks
in tmp_replication directory are discarded.
The reason I propose to start client-generated blocks in the "tmp" directory
(instead of the real block directory) is because these blocks are not yet
confirmed to the namenode. They are still being written and ideally should not
be included in any block report(s).
> Datanode should delete files under tmp when upgraded from 0.17
> --------------------------------------------------------------
>
> Key: HADOOP-4663
> URL: https://issues.apache.org/jira/browse/HADOOP-4663
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Raghu Angadi
> Assignee: dhruba borthakur
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: deleteTmp.patch, deleteTmp2.patch, deleteTmp_0.18.patch
>
>
> Before 0.18, when Datanode restarts, it deletes files under data-dir/tmp
> directory since these files are not valid anymore. But in 0.18 it moves these
> files to normal directory incorrectly making them valid blocks. One of the
> following would work :
> - remove the tmp files during upgrade, or
> - if the files under /tmp are in pre-18 format (i.e. no generation), delete
> them.
> Currently effect of this bug is that, these files end up failing block
> verification and eventually get deleted. But cause incorrect over-replication
> at the namenode before that.
> Also it looks like our policy regd treating files under tmp needs to be
> defined better. Right now there are probably one or two more bugs with it.
> Dhruba, please file them if you rememeber.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
