[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated MAPREDUCE-5962:
-----------------------------------

    Attachment: mapreduce-5962.txt

The RawKVIteratorReader used in the shuffle wasn't properly passing the jobconf 
into the IFileReader. This was causing an NPE when we tried to get the checksum 
type out of the conf. I changed it to pass the jobConf, which may actually have 
a slight performance advantage too due to avoiding the "new Configuration()" 
call in IFileInputStream's ctor. Verified that the two unit tests that failed 
before now pass on my machine

> Support CRC32C in IFile
> -----------------------
>
>                 Key: MAPREDUCE-5962
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5962
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: performance, task
>    Affects Versions: 2.5.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: mapreduce-5962.txt, mapreduce-5962.txt
>
>
> Currently, the IFile format used by the MR shuffle checksums all data using 
> the zlib CRC32 polynomial. If we allow use of CRC32C instead, we can get a 
> large reduction in CPU usage by leveraging the native hardware CRC32C 
> implementation (approx half a second of CPU time savings per GB checksummed).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Reply via email to