Ryan, If you're using getOutputPath, try replacing it with getWorkOutputPath.
http://hadoop.apache.org/core/docs/r0.18.3/api/org/apache/hadoop/mapred/ FileOutputFormat.html#getWorkOutputPath(org.apache.hadoop.mapred.JobConf ) Koji -----Original Message----- From: Ryan Shih [mailto:ryan.s...@gmail.com] Sent: Monday, March 02, 2009 11:01 AM To: core-user@hadoop.apache.org Subject: Potential race condition (Hadoop 18.3) Hi - I'm not sure yet, but I think I might be hitting a race condition in Hadoop 18.3. What seems to happen is that in the reduce phase, some of my tasks perform speculative execution but when the initial task completes successfully, it sends a kill to the new task started. After all is said and done, perhaps one in every five or ten which kill their second task ends up with zero or truncated output. When I code it to turn off speculative execution, the problem goes away. Are there known race conditions that I should be aware of around this area? Thanks in advance, Ryan
