[
https://issues.apache.org/jira/browse/HADOOP-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668188#action_12668188
]
dhruba borthakur commented on HADOOP-4663:
------------------------------------------
Hairong, I understand your point of view. A bug in the generation-stamp
protocol could cause data corruption.
The alternative is to not promote blocks to the main storage when the datanode
restarts. In this case, the data that was written by the client to that
datanode is lost. This is data written by an application, and it would be nice
if HDFS can make every effort to recover this data instead of deleting it. This
is true with most other file systems. For example, if an application writes
some data to an ext3 file and then dies before closing/fsync it, the OS/FS does
not delete the data that was cached in the OS pages. It makes every effort to
persist it. If we can have similar semantics for HDFS, it will be a more
powerful system, isn't it? An HDFS application that does not issue a "fsync",
cannot rely on the fact that all the data it has written will be persisted, but
as long as HDFS makes a good effort to keep all the data, that will be nice,
isn't it?
So, this issue all boils down to the tradeoff of having a "file system that
never persists data unless the writer explicitly invoked a fsync" verses
"complexity of the namenode thereby introducing buggy code".
I can vote for one application that we run inhouse that would definitely like
the behaviour where HDFS makes every effort to persist data (rather than
invoking sync frequently). HBase can use this feature too (in the future) to
recover HBase transactions that lie beyond the sync point (good to have, not a
hard requirement).
> Datanode should delete files under tmp when upgraded from 0.17
> --------------------------------------------------------------
>
> Key: HADOOP-4663
> URL: https://issues.apache.org/jira/browse/HADOOP-4663
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Raghu Angadi
> Assignee: dhruba borthakur
> Priority: Blocker
> Fix For: 0.19.1
>
> Attachments: deleteTmp.patch, deleteTmp2.patch, deleteTmp_0.18.patch,
> handleTmp1.patch
>
>
> Before 0.18, when Datanode restarts, it deletes files under data-dir/tmp
> directory since these files are not valid anymore. But in 0.18 it moves these
> files to normal directory incorrectly making them valid blocks. One of the
> following would work :
> - remove the tmp files during upgrade, or
> - if the files under /tmp are in pre-18 format (i.e. no generation), delete
> them.
> Currently effect of this bug is that, these files end up failing block
> verification and eventually get deleted. But cause incorrect over-replication
> at the namenode before that.
> Also it looks like our policy regd treating files under tmp needs to be
> defined better. Right now there are probably one or two more bugs with it.
> Dhruba, please file them if you rememeber.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
