Hi everyone,
(apologies if this gets posted on the list twice for some reason, my first
attempt was denied as "suspected spam")
I ran a job last night with Hadoop 0.18.0 on EC2, using the standard small
AMI. The job was producing gzipped output, otherwise I haven't changed the
configuration.
The final reduce steps failed with this error that I haven't seem before:
2008-10-01 05:02:39,810 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200809301822_0005_r_000001_0 Merging of the local FS files threw an
exception: java.io.IOException: java.io.IOException: Rec# 289050: Negative
value-length: -96
at org.apache.hadoop.mapred.IFile$Reader.next(IFile.java:331)
at org.apache.hadoop.mapred.Merger$Segment.next(Merger.java:134)
at
org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:225)
at org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:242)
at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:83)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(ReduceTask.java:2021)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(ReduceTask.java:2025)
2008-10-01 05:02:44,131 WARN org.apache.hadoop.mapred.TaskTracker: Error
running child
java.io.IOException: attempt_200809301822_0005_r_000001_0The reduce copier
failed
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2209)
When I try to download the data from HDFS I get a "Found checksum error"
warning message.
Any ideas what could be the cause? Would upgrading to 0.18.1 solve it?
Thanks,
/ Per
