[
https://issues.apache.org/jira/browse/MAPREDUCE-3992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226815#comment-13226815
]
Steve Loughran commented on MAPREDUCE-3992:
-------------------------------------------
Todd
There's actually some good arguments for putting the checksum at the end
anyway; Stone at al. 1998 cover this in _[Performance of checksums and CRC's
over real data|ftp://cs.bell-labs.com/who/mbgreen/papers/ton98.pdf]_ -it looks
at the problem of packet splicing of TCP packets -and shows that if two packets
get spliced together, the TCP header may not detect this.
That's specific to TCP & IP headers, both of which are weak, and a packet-layer
network where splicing can occur; I can't see it happening at the HTTP level
without a very confused proxy getting in the way.
Header checksums wouldn't catch the corruption midway through a packet, nor
would length headers.
If the crcs where known in advance -i.e. could be picked up from the filesystem
.crc. files, a custom header could include them all as a list, which could then
be used as a substitute for interleaving. That requires them to be known before
streaming the data, and the CRC algorithm to be the same at both ends, which
may not work with the x86-specific CRC work in trunk.
> Reduce fetcher doesn't verify HTTP status code of response
> ----------------------------------------------------------
>
> Key: MAPREDUCE-3992
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3992
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1
> Affects Versions: 1.0.1
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
>
> Currently, the reduce fetch code doesn't check the HTTP status code of the
> response. This can lead to the following situation:
> - the map output servlet gets an IOException after setting the headers but
> before the first call to flush()
> - this causes it to send a response with a non-OK result code, including the
> exception text as the response body (response.sendError() does this if the
> response isn't committed)
> - it will still include the response headers indicating it's a valid response
> In the case of a merge-to-memory, the compression codec might then try to
> interpret the HTML response as compressed data, resulting in either a huge
> allocation (OOME) or some other nasty error. This bug seems to be present in
> MR1, but haven't checked trunk/MR2 yet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
