Re: Reduce task failed: org.apache.hadoop.fs.FSError: java.io.IOException
Thanks Arun, Yeah I think it was a disk problem. Redoing the task went through fine. Since disk corruptions may be common, doesn't hadoop pick a replicated block? - Prasad. On Thursday 11 September 2008 11:38:53 pm Arun C Murthy wrote: > On Sep 11, 2008, at 9:10 AM, pvvpr wrote: > > Hello, > > Never came across this error before. Upgraded to 0.18.0 this > > morning and > > ran a nutch fetch job. Got this exception in both the reduce > > attempts of > > a task and they failed. All other reducers seemed to work fine, except > > for one task. > > > > Any ideas what could be the problem? > > Looks like data on disk got corrupted ( org.apache.hadoop.fs.FSError: > java.io.IOException: Input/output error); could you check the disks? > > Arun > > > - Prasad Pingali. > > IIIT, Hyderabad. > > > > 2008-09-11 06:31:19,837 INFO org.apache.hadoop.mapred.ReduceTask: > > attempt_200809101353_0021_r_04_0: Got 1 new map-outputs & number > > of > > known map outputs is 1 > > 2008-09-11 06:31:19,837 INFO org.apache.hadoop.mapred.ReduceTask: > > attempt_200809101353_0021_r_04_0 Scheduled 1 of 1 known outputs (0 > > slow hosts and 0 dup hosts) > > 2008-09-11 06:31:20,133 INFO org.apache.hadoop.mapred.ReduceTask: > > Shuffling 8230686 bytes (8230686 raw bytes) into RAM from > > attempt_200809101353_0021_m_95_0 > > 2008-09-11 06:31:22,332 INFO org.apache.hadoop.mapred.ReduceTask: Read > > 8230686 bytes from map-output for attempt_200809101353_0021_m_95_0 > > 2008-09-11 06:31:22,333 INFO org.apache.hadoop.mapred.ReduceTask: > > Rec #1 > > from attempt_200809101353_0021_m_95_0 -> (33, 134) from machine10 > > 2008-09-11 06:31:28,837 INFO org.apache.hadoop.mapred.ReduceTask: > > attempt_200809101353_0021_r_04_0: Got 1 new map-outputs & number > > of > > known map outputs is 1 > > 2008-09-11 06:31:28,838 INFO org.apache.hadoop.mapred.ReduceTask: > > attempt_200809101353_0021_r_04_0 Scheduled 1 of 1 known outputs (0 > > slow hosts and 0 dup hosts) > > 2008-09-11 06:31:29,585 INFO org.apache.hadoop.mapred.ReduceTask: > > Shuffling 21454877 bytes (21454877 raw bytes) into Local-FS from > > attempt_200809101353_0021_m_74_0 > > 2008-09-11 06:31:37,831 INFO org.apache.hadoop.mapred.ReduceTask: Read > > 21454877 bytes from map-output for > > attempt_200809101353_0021_m_74_0 > > 2008-09-11 06:31:37,832 INFO org.apache.hadoop.mapred.ReduceTask: > > attempt_200809101353_0021_r_04_0We have 19 map outputs on disk. > > Triggering merge of 10 files > > 2008-09-11 06:31:38,033 INFO org.apache.hadoop.mapred.Merger: > > Merging 10 > > sorted segments > > 2008-09-11 06:31:43,838 INFO org.apache.hadoop.mapred.ReduceTask: > > attempt_200809101353_0021_r_04_0 Need another 9 map output(s) > > where 0 > > is already in progress > > 2008-09-11 06:31:43,838 INFO org.apache.hadoop.mapred.ReduceTask: > > attempt_200809101353_0021_r_04_0: Got 0 new map-outputs & number > > of > > known map outputs is 0 > > 2008-09-11 06:31:43,838 INFO org.apache.hadoop.mapred.ReduceTask: > > attempt_200809101353_0021_r_04_0 Scheduled 0 of 0 known outputs (0 > > slow hosts and 0 dup hosts) > > 2008-09-11 06:32:03,095 INFO org.apache.hadoop.mapred.Merger: Down > > to the > > last merge-pass, with 10 segments left of total size: 262561859 bytes > > 2008-09-11 06:32:51,044 WARN org.apache.hadoop.mapred.ReduceTask: > > attempt_200809101353_0021_r_04_0 Merging of the local FS files > > threw > > an exception: org.apache.hadoop.fs.FSError: java.io.IOException: > > Input/output error > >at > > org.apache.hadoop.fs.RawLocalFileSystem > > $LocalFSFileInputStream.read(RawLocalFileSystem.java:149) > >at java.io.BufferedInputStream.read1(BufferedInputStream.java: > > 256) > >at java.io.BufferedInputStream.read(BufferedInputStream.java: > > 317) > >at java.io.DataInputStream.read(DataInputStream.java:132) > >at > > org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:380) > >at > > org.apache.hadoop.fs.ChecksumFileSystem > > $ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:208) > >at > > org > > .apache > > .hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:236) > >at > > org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:191) > >at > > org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:159) > >at java.io.DataInputStream.read(DataInputStream.java:132) > >at org.apache.hadoop.mapred.IFile$Reader.readData(IFile.java: > > 263) > >at org.apache.hadoop.mapred.IFile$Reader.rejigData(IFile.java: > > 293) > >at > > org.apache.hadoop.mapred.IFile$Reader.readNextBlock(IFile.java:277) > >at org.apache.hadoop.mapred.IFile$Reader.next(IFile.java:339) > >at org.apache.hadoop.mapred.Merger$Segment.next(Merger.java: > > 134) > >at > > org.apache.hadoop.mapred.Merger > > $MergeQueue.adjustPriorityQueue(Merger.java:225) > >at org.apache.hadoop.mapred.Merger >
Re: Reduce task failed: org.apache.hadoop.fs.FSError: java.io.IOException
On Sep 11, 2008, at 9:10 AM, pvvpr wrote: Hello, Never came across this error before. Upgraded to 0.18.0 this morning and ran a nutch fetch job. Got this exception in both the reduce attempts of a task and they failed. All other reducers seemed to work fine, except for one task. Any ideas what could be the problem? Looks like data on disk got corrupted ( org.apache.hadoop.fs.FSError: java.io.IOException: Input/output error); could you check the disks? Arun - Prasad Pingali. IIIT, Hyderabad. 2008-09-11 06:31:19,837 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200809101353_0021_r_04_0: Got 1 new map-outputs & number of known map outputs is 1 2008-09-11 06:31:19,837 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200809101353_0021_r_04_0 Scheduled 1 of 1 known outputs (0 slow hosts and 0 dup hosts) 2008-09-11 06:31:20,133 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling 8230686 bytes (8230686 raw bytes) into RAM from attempt_200809101353_0021_m_95_0 2008-09-11 06:31:22,332 INFO org.apache.hadoop.mapred.ReduceTask: Read 8230686 bytes from map-output for attempt_200809101353_0021_m_95_0 2008-09-11 06:31:22,333 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 from attempt_200809101353_0021_m_95_0 -> (33, 134) from machine10 2008-09-11 06:31:28,837 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200809101353_0021_r_04_0: Got 1 new map-outputs & number of known map outputs is 1 2008-09-11 06:31:28,838 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200809101353_0021_r_04_0 Scheduled 1 of 1 known outputs (0 slow hosts and 0 dup hosts) 2008-09-11 06:31:29,585 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling 21454877 bytes (21454877 raw bytes) into Local-FS from attempt_200809101353_0021_m_74_0 2008-09-11 06:31:37,831 INFO org.apache.hadoop.mapred.ReduceTask: Read 21454877 bytes from map-output for attempt_200809101353_0021_m_74_0 2008-09-11 06:31:37,832 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200809101353_0021_r_04_0We have 19 map outputs on disk. Triggering merge of 10 files 2008-09-11 06:31:38,033 INFO org.apache.hadoop.mapred.Merger: Merging 10 sorted segments 2008-09-11 06:31:43,838 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200809101353_0021_r_04_0 Need another 9 map output(s) where 0 is already in progress 2008-09-11 06:31:43,838 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200809101353_0021_r_04_0: Got 0 new map-outputs & number of known map outputs is 0 2008-09-11 06:31:43,838 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200809101353_0021_r_04_0 Scheduled 0 of 0 known outputs (0 slow hosts and 0 dup hosts) 2008-09-11 06:32:03,095 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 262561859 bytes 2008-09-11 06:32:51,044 WARN org.apache.hadoop.mapred.ReduceTask: attempt_200809101353_0021_r_04_0 Merging of the local FS files threw an exception: org.apache.hadoop.fs.FSError: java.io.IOException: Input/output error at org.apache.hadoop.fs.RawLocalFileSystem $LocalFSFileInputStream.read(RawLocalFileSystem.java:149) at java.io.BufferedInputStream.read1(BufferedInputStream.java: 256) at java.io.BufferedInputStream.read(BufferedInputStream.java: 317) at java.io.DataInputStream.read(DataInputStream.java:132) at org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:380) at org.apache.hadoop.fs.ChecksumFileSystem $ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:208) at org .apache .hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:236) at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:191) at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:159) at java.io.DataInputStream.read(DataInputStream.java:132) at org.apache.hadoop.mapred.IFile$Reader.readData(IFile.java: 263) at org.apache.hadoop.mapred.IFile$Reader.rejigData(IFile.java: 293) at org.apache.hadoop.mapred.IFile$Reader.readNextBlock(IFile.java:277) at org.apache.hadoop.mapred.IFile$Reader.next(IFile.java:339) at org.apache.hadoop.mapred.Merger$Segment.next(Merger.java: 134) at org.apache.hadoop.mapred.Merger $MergeQueue.adjustPriorityQueue(Merger.java:225) at org.apache.hadoop.mapred.Merger $MergeQueue.next(Merger.java:242) at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:83) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier $LocalFSMerger.run(ReduceTask.java:2021) Caused by: java.io.IOException: Input/output error at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:199) at org.apache.hadoop.fs.RawLocalFileSystem $TrackingFileInputStream.read(RawLocalFileSystem.java:90) at org.apache.hadoop.fs.RawLocalFileSystem $LocalFSFileInputStream.read(RawLocalFileSystem.j
Reduce task failed: org.apache.hadoop.fs.FSError: java.io.IOException
Hello, Never came across this error before. Upgraded to 0.18.0 this morning and ran a nutch fetch job. Got this exception in both the reduce attempts of a task and they failed. All other reducers seemed to work fine, except for one task. Any ideas what could be the problem? - Prasad Pingali. IIIT, Hyderabad. 2008-09-11 06:31:19,837 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200809101353_0021_r_04_0: Got 1 new map-outputs & number of known map outputs is 1 2008-09-11 06:31:19,837 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200809101353_0021_r_04_0 Scheduled 1 of 1 known outputs (0 slow hosts and 0 dup hosts) 2008-09-11 06:31:20,133 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling 8230686 bytes (8230686 raw bytes) into RAM from attempt_200809101353_0021_m_95_0 2008-09-11 06:31:22,332 INFO org.apache.hadoop.mapred.ReduceTask: Read 8230686 bytes from map-output for attempt_200809101353_0021_m_95_0 2008-09-11 06:31:22,333 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 from attempt_200809101353_0021_m_95_0 -> (33, 134) from machine10 2008-09-11 06:31:28,837 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200809101353_0021_r_04_0: Got 1 new map-outputs & number of known map outputs is 1 2008-09-11 06:31:28,838 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200809101353_0021_r_04_0 Scheduled 1 of 1 known outputs (0 slow hosts and 0 dup hosts) 2008-09-11 06:31:29,585 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling 21454877 bytes (21454877 raw bytes) into Local-FS from attempt_200809101353_0021_m_74_0 2008-09-11 06:31:37,831 INFO org.apache.hadoop.mapred.ReduceTask: Read 21454877 bytes from map-output for attempt_200809101353_0021_m_74_0 2008-09-11 06:31:37,832 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200809101353_0021_r_04_0We have 19 map outputs on disk. Triggering merge of 10 files 2008-09-11 06:31:38,033 INFO org.apache.hadoop.mapred.Merger: Merging 10 sorted segments 2008-09-11 06:31:43,838 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200809101353_0021_r_04_0 Need another 9 map output(s) where 0 is already in progress 2008-09-11 06:31:43,838 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200809101353_0021_r_04_0: Got 0 new map-outputs & number of known map outputs is 0 2008-09-11 06:31:43,838 INFO org.apache.hadoop.mapred.ReduceTask: attempt_200809101353_0021_r_04_0 Scheduled 0 of 0 known outputs (0 slow hosts and 0 dup hosts) 2008-09-11 06:32:03,095 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 262561859 bytes 2008-09-11 06:32:51,044 WARN org.apache.hadoop.mapred.ReduceTask: attempt_200809101353_0021_r_04_0 Merging of the local FS files threw an exception: org.apache.hadoop.fs.FSError: java.io.IOException: Input/output error at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:149) at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) at java.io.DataInputStream.read(DataInputStream.java:132) at org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:380) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:208) at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:236) at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:191) at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:159) at java.io.DataInputStream.read(DataInputStream.java:132) at org.apache.hadoop.mapred.IFile$Reader.readData(IFile.java:263) at org.apache.hadoop.mapred.IFile$Reader.rejigData(IFile.java:293) at org.apache.hadoop.mapred.IFile$Reader.readNextBlock(IFile.java:277) at org.apache.hadoop.mapred.IFile$Reader.next(IFile.java:339) at org.apache.hadoop.mapred.Merger$Segment.next(Merger.java:134) at org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:225) at org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:242) at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:83) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(ReduceTask.java:2021) Caused by: java.io.IOException: Input/output error at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:199) at org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream.read(RawLocalFileSystem.java:90) at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:143)
