Can you include the complete stack trace of the IOException you are seeing?
--Bobby Evans On 9/23/11 2:15 AM, "Sofia Georgiakaki" <geosofie_...@yahoo.com> wrote: Good morning! I would be grateful if anyone could help me about a serious problem that I'm facing. I try to run a hadoop job on a 12-node luster (has 48 task capacity), and I have problems when dealing with big input data (10-20GB) which gets worse when I increase the number of reducers. Many tasks get killed (for example 25 out of the 148 map tasks, and 15 out of 40 reducers) and the job struggles to finish. The job is heavy in general, as it builds an Rtree on hdfs. During the reduce phase, I also create and write some binary files on HDFS using FSDataOutputStream. and I noticed that sometimes some tasks fail to write correctly to their particular binary file, throwing an IOexception when they try to execute dataFileOut.write(m_buffer); . I'm using 0.20.203 version and I had also tested the code on 0.20.2 before (facing the same problems with killed tasks!) I would appreciate any advice/idea, as I have to finish my diploma thesis (it has taken me a year, I hope not to take longer). Thank you very much in advance Sofia
