[
https://issues.apache.org/jira/browse/MAPREDUCE-1784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866645#action_12866645
]
Eli Collins commented on MAPREDUCE-1784:
----------------------------------------
I made it a warning since GzipCodec can function w/o native libraries (kind of,
it provides java input and output streams, eg GzipOutputStream, though it
returns null for createCompressor) and didn't want this to stop something that
should otherwise work (ie the user specifying
org.apache.hadoop.io.compress.GzipCodec doesn't imply native libraries must be
used). However it does seem like they should be notified that they requested a
codec that can't be created. How about CodecPool#getCompressor throws an
exception instead of returning null indicating it couldn't get a compressor for
the given codec (because codec.createCompressor() returned null)?
I could add a test that covers this method in TestCodec. I think adding a test
to TestReduceTask that uses a Gzip codec would also cover this.
> IFile should check for null compressor
> --------------------------------------
>
> Key: MAPREDUCE-1784
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1784
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: mapred-1784-1.patch
>
>
> IFile assumes that when it has a codec it can always get a compressor. This
> fails when mapred.compress.map.output is true but the native libraries are
> not installed, resulting in an NPE:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:102)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1198)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1091)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:359)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> {code}
> Let's make IFile handle this case by logging and using non-compressed
> streams.l
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
