[
https://issues.apache.org/jira/browse/MAPREDUCE-5962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated MAPREDUCE-5962:
-----------------------------------
Attachment: mapreduce-5962.txt
The RawKVIteratorReader used in the shuffle wasn't properly passing the jobconf
into the IFileReader. This was causing an NPE when we tried to get the checksum
type out of the conf. I changed it to pass the jobConf, which may actually have
a slight performance advantage too due to avoiding the "new Configuration()"
call in IFileInputStream's ctor. Verified that the two unit tests that failed
before now pass on my machine
> Support CRC32C in IFile
> -----------------------
>
> Key: MAPREDUCE-5962
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5962
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance, task
> Affects Versions: 2.5.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: mapreduce-5962.txt, mapreduce-5962.txt
>
>
> Currently, the IFile format used by the MR shuffle checksums all data using
> the zlib CRC32 polynomial. If we allow use of CRC32C instead, we can get a
> large reduction in CPU usage by leveraging the native hardware CRC32C
> implementation (approx half a second of CPU time savings per GB checksummed).
--
This message was sent by Atlassian JIRA
(v6.2#6252)