On Jan 29, 2008, at 1:30 PM, Jason Venner wrote:
We have overridden the base class public class MapReduceBase
extends org.apache.hadoop.mapred.MapReduceBase
to have the configure method log the split name and split section
(or in the case of gzip'd files the file name).
We find it very helpful to make the job errors to the section of
the input file causing the problem.
Maybe we should just log it by default? Want to submit that patch?
Arun
Vadim Zaliva wrote:
I have a bunch of gzip files which I am trying to process with
Hadoop task. The task fails with exception:
java.io.EOFException: Unexpected end of ZLIB input stream at
java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:
223) at java.util.zip.InflaterInputStream.read
(InflaterInputStream.java:141) at
java.util.zip.GZIPInputStream.read(GZIPInputStream.java:92) at
org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.read
(GzipCodec.java:124) at java.io.BufferedInputStream.fill
(BufferedInputStream.java:218) at java.io.BufferedInputStream.read
(BufferedInputStream.java:237) at
org.apache.hadoop.mapred.LineRecordReader.readLine
(LineRecordReader.java:136) at
org.apache.hadoop.mapred.LineRecordReader.readLine
(LineRecordReader.java:128) at
org.apache.hadoop.mapred.LineRecordReader.next
(LineRecordReader.java:117) at
org.apache.hadoop.mapred.LineRecordReader.next
(LineRecordReader.java:39) at org.apache.hadoop.mapred.MapTask
$TrackedRecordReader.next(MapTask.java:147) at
org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:208) at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:
2016)
I guess some of files are invalid. However I could not find
anywhere in logs file name of the file causing this exception. Due
to the huge size of the dataset I would not want to extract files
from DFS and verify them with Gzip one by one. Any suggestions?
Thanks!
Sincerely,
Vadim
--
Jason Venner
Attributor - Publish with Confidence <http://www.attributor.com/>
Attributor is hiring Hadoop Wranglers, contact if interested
