[ https://issues.apache.org/jira/browse/MAPREDUCE-5962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon resolved MAPREDUCE-5962. ------------------------------------ Resolution: Won't Fix > Support CRC32C in IFile > ----------------------- > > Key: MAPREDUCE-5962 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5962 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: performance, task > Affects Versions: 2.5.0 > Reporter: Todd Lipcon > Assignee: Todd Lipcon > Priority: Major > Attachments: mapreduce-5962.txt, mapreduce-5962.txt > > > Currently, the IFile format used by the MR shuffle checksums all data using > the zlib CRC32 polynomial. If we allow use of CRC32C instead, we can get a > large reduction in CPU usage by leveraging the native hardware CRC32C > implementation (approx half a second of CPU time savings per GB checksummed). -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org