[
https://issues.apache.org/jira/browse/MAPREDUCE-6612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15114729#comment-15114729
]
Devaraj K commented on MAPREDUCE-6612:
--------------------------------------
As per the stack trace, it looks like you are running the Job in local mode. Do
you see the issue when you run the same job in YARN mode(i.e.
mapreduce.framework.name=yarn) as well?
> Compressing map output error
> ----------------------------
>
> Key: MAPREDUCE-6612
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6612
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: wangjiayou
>
> I used gzip compressionor to compress map output,but there is error
> occurrence.my hadoop version 2.7.1
> Configuration conf = getConf();
> conf.setBoolean(Job.MAP_OUTPUT_COMPRESS, true);
> conf.setClass(Job.MAP_OUTPUT_COMPRESS_CODEC, GzipCodec.class,
> CompressionCodec.class);
> error info:
> 2016-01-21 15:02:23 [org.apache.hadoop.mapred.LocalJobRunner]-[INFO] reduce >
> copy
> 2016-01-21 15:02:23 [org.apache.hadoop.mapred.LocalJobRunner]-[WARN]
> job_local1329832043_0001
> java.lang.Exception:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in localfetcher#1
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
> Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError:
> error in shuffle in localfetcher#1
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: not a gzip file
> at
> org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.processBasicHeader(BuiltInGzipDecompressor.java:496)
> at
> org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.executeHeaderState(BuiltInGzipDecompressor.java:257)
> at
> org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.decompress(BuiltInGzipDecompressor.java:186)
> at
> org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:91)
> at
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199)
> at
> org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.shuffle(InMemoryMapOutput.java:97)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:157)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:102)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:85)
> 2016-01-21 15:02:23 [org.apache.hadoop.mapred.LocalJobRunner]-[INFO] reduce >
> copy
> 2016-01-21 15:02:23 [org.apache.hadoop.mapred.LocalJobRunner]-[INFO] reduce >
> copy
> 2016-01-21 15:02:24 [org.apache.hadoop.mapred.LocalJobRunner]-[INFO] reduce >
> copy
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
