[
https://issues.apache.org/jira/browse/HDFS-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763179#action_12763179
]
Hairong Kuang commented on HDFS-679:
------------------------------------
It turns out there is also a bug at the DataNode side that handles appending to
partial checksum chunk incorrectly. Let me try to explain.
Suppose that a file has one byte 'a'. It is open for append. The client writes
one byte 'b' and calls hflush. A packet containing 'b" gets pushed to a
datanode. The datanode reads 'a' and its checksum from disk, validating its
on-disk data, then recaculates the checksum for "ab", writes 'b' to disk, and
overwrites the checksum for the chunk. So far so good. However, when the client
writes another byte "c" and calls hflush again, the client send a packet "bc"
not "c" to the datanode side. When the datanode reads "a" and checksums from
the disk, checksum validation fails because the checksum on disk is not the
checksum for "a" but for "ab". The write fails at the following lines in
BlockReceiver#computePartialChunkCrc:
{noformat}
if (partialCrc.getValue() != FSInputChecker.checksum2long(crcbuf)) {
String msg = "Partial CRC " + partialCrc.getValue() +
" does not match value computed the " +
" last time file was closed " +
FSInputChecker.checksum2long(crcbuf);
throw new IOException(msg);
}
{noformat}
> Block receiver unexpectedly throws an IOException complaining mismatched
> checksum
> ---------------------------------------------------------------------------------
>
> Key: HDFS-679
> URL: https://issues.apache.org/jira/browse/HDFS-679
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.21.0
>
>
> When I run TestClientProtocolForPipelineRecovery, I always see that the block
> receiver throws IOException complaining about mismatched checksum when
> receiving the last data packet. It turns out the checksum of last packet was
> unexpectedly set to be zero.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
