We have started to see this class of error under hadoop 0.16.1 on a medium sized hdfs cluster under moderate load
wangxu wrote: > Hi,all > I am using hadoop-0.18.0-core.jar and nutch-2008-08-18_04-01-55.jar, > and running hadoop on one namenode and 4 slaves. > attached is my hadoop-site.xml, and I didn't change the file > hadoop-default.xml > > when data in segments are large,this kind of errors occure: > > java.io.IOException: Could not obtain block: blk_-2634319951074439134_1129 > file=/user/root/crawl_debug/segments/20080825053518/content/part-00002/data > at > org.apache.hadoop.dfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1462) > at > org.apache.hadoop.dfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1312) > at > org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1417) > at java.io.DataInputStream.readFully(DataInputStream.java:178) > at > org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:64) > at > org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:102) > at > org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1646) > at > org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1712) > at > org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1787) > at > org.apache.hadoop.mapred.SequenceFileRecordReader.getCurrentValue(SequenceFileRecordReader.java:104) > at > org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:79) > at > org.apache.hadoop.mapred.join.WrappedRecordReader.next(WrappedRecordReader.java:112) > at > org.apache.hadoop.mapred.join.WrappedRecordReader.accept(WrappedRecordReader.java:130) > at > org.apache.hadoop.mapred.join.CompositeRecordReader.fillJoinCollector(CompositeRecordReader.java:398) > at > org.apache.hadoop.mapred.join.JoinRecordReader.next(JoinRecordReader.java:56) > at > org.apache.hadoop.mapred.join.JoinRecordReader.next(JoinRecordReader.java:33) > at > org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:165) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:45) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227) > at > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2209) > > > how can I correct this? > thanks. > Xu > > -- Jason Venner Attributor - Program the Web <http://www.attributor.com/> Attributor is hiring Hadoop Wranglers and coding wizards, contact if interested
