[
https://issues.apache.org/jira/browse/HADOOP-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665309#action_12665309
]
Jim Kellerman commented on HADOOP-4379:
---------------------------------------
I have attached two test programs that demonstrate the problem.
Writer.java writes 100 records of approx 1KB each per sync. Its command line
argument is the number of sync's to perform before aborting.
Reader.java is run after Writer and tries to read records from the file created
by Writer.
If you run Writer with 4096 syncs:
bin/hadoop Writer 4096
and then run Reader, it gets an EOFException without having read any records
(note that exception occurs in constructor:
{code}
Exception in thread "main" java.io.EOFException
at java.io.DataInputStream.readFully(Unknown Source)
at java.io.DataInputStream.readFully(Unknown Source)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1450)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1428)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
at Reader.main(Reader.java:34)
{code}
If you run Writer with 8192 syncs:
bin/hadoop Writer 8192
and then run Reader, it reads 684783 records before hitting an EOFException (in
next())
{code}
Read 684783 lines
java.io.EOFException
at java.io.DataInputStream.readFully(Unknown Source)
at
org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
at
org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1930)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1830)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
at Reader.main(Reader.java:41)
{code}
684783 records that are approximately 1KB in length is approximately 64MB (a
hadoop block).
In this test the partial block containing 134417 records is lost (even though a
considerable
number of syncs took place between filling the first block and the end of
Writer).
So you can read from an improperly closed file but only up to the last full
block, confirming
Luke's observation.
I will look into the patch shortly.
> In HDFS, sync() not yet guarantees data available to the new readers
> --------------------------------------------------------------------
>
> Key: HADOOP-4379
> URL: https://issues.apache.org/jira/browse/HADOOP-4379
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: dhruba borthakur
> Fix For: 0.19.1
>
> Attachments: 4379_20081010TC3.java, fsyncConcurrentReaders.txt,
> Reader.java, Writer.java
>
>
> In the append design doc
> (https://issues.apache.org/jira/secure/attachment/12370562/Appends.doc), it
> says
> * A reader is guaranteed to be able to read data that was 'flushed' before
> the reader opened the file
> However, this feature is not yet implemented. Note that the operation
> 'flushed' is now called "sync".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
