Lecture: Large-scale data processing with Hadoop MapReduce

evert.lamme...@sara.nl. More information on: * Dr. Jimmy Lin: http://ischool.umd.edu/content/jimmy-lin-0 * Hadoop: http://hadoop.apache.org/ * Hadoop @ BiG Grid: http://www.sara.nl/project/hadoop _______ Evert Lammerts SARA Phone: +31 6 10558303, +31 20 5923017

Lecture: Large-scale data processing with Hadoop MapReduce

evert.lamme...@sara.nl. More information on: * Dr. Jimmy Lin: http://ischool.umd.edu/content/jimmy-lin-0 * Hadoop: http://hadoop.apache.org/ * Hadoop @ BiG Grid: http://www.sara.nl/project/hadoop _______ Evert Lammerts SARA Phone: +31 6 10558303, +31 20 5923017

getMapOutput, EOFException, IllegalStateException

When Reducers start running during a certain job (mapred.reduce.slowstart.completed.maps = 0.8) it takes about 20 minutes before the DN stopd reacting. This seems to be due to a number of Exceptions in the TT - at least, it's the only place I'm seeing errors. The three recurring ones are getMap

Map 100%, Reduce 99.99%?

Hi all, Why do we every now and then see a job remaining in Running state with no more Mappers or Reducers running, while the reduce progress tells us it's 99.99% done? Might this be due to a stranded process? Cheers, Evert

RE: Could not obtain block

09, 2011 4:31 PM To: mapreduce-user@hadoop.apache.org Cc: Evert Lammerts; 'hdfs-u...@hadoop.apache.org'; cdh-u...@cloudera.org Subject: Re: Could not obtain block El 3/9/2011 6:27 AM, Evert Lammerts escribió: > We see a lot of IOExceptions coming from HDFS during a job that does nothing

Could not obtain block

org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:534) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122) Cheers, Evert Lammerts Consultant eScience & C