[
https://issues.apache.org/jira/browse/HDFS-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754052#action_12754052
]
dhruba borthakur commented on HDFS-200:
---------------------------------------
I think this is the scenario that you are facing:
1. The file is written for the first time and was not closed. The writer closed
the file but at this time only one of the three replicas have checked in with
the namenode.
2. The new writer invoked append() to write more data into the file. The new
writer found the one remaining replica of the block, stamped a new generation
stamp for this block, made it ready to receive new data for this file and lease
recovery is successful. The stamping of the new generation stamp essentially
invaidated the other two replicas of this block....this block now has only one
valid replica. The namenode won't start replicating this block till the block
is full. If this sole datanode now goes down, then the file will be "missing a
block". This is what you folks encountered.
One option is to set dfs.replication.min to 2. This will ensure that closing a
file (step 1) will be successful only when at least two replcias of the block
have checked in with the namenode. This should reduce the probability of this
problem occuring. Another option is to set the replication factor of the hbase
log file(s) to be greater than 3.
> In HDFS, sync() not yet guarantees data available to the new readers
> --------------------------------------------------------------------
>
> Key: HDFS-200
> URL: https://issues.apache.org/jira/browse/HDFS-200
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: dhruba borthakur
> Priority: Blocker
> Attachments: 4379_20081010TC3.java, fsyncConcurrentReaders.txt,
> fsyncConcurrentReaders11_20.txt, fsyncConcurrentReaders12_20.txt,
> fsyncConcurrentReaders13_20.txt, fsyncConcurrentReaders14_20.txt,
> fsyncConcurrentReaders3.patch, fsyncConcurrentReaders4.patch,
> fsyncConcurrentReaders5.txt, fsyncConcurrentReaders6.patch,
> fsyncConcurrentReaders9.patch,
> hadoop-stack-namenode-aa0-000-12.u.powerset.com.log.gz,
> hdfs-200-ryan-existing-file-fail.txt, hypertable-namenode.log.gz,
> namenode.log, namenode.log, Reader.java, Reader.java, reopen_test.sh,
> ReopenProblem.java, Writer.java, Writer.java
>
>
> In the append design doc
> (https://issues.apache.org/jira/secure/attachment/12370562/Appends.doc), it
> says
> * A reader is guaranteed to be able to read data that was 'flushed' before
> the reader opened the file
> However, this feature is not yet implemented. Note that the operation
> 'flushed' is now called "sync".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
