Hello, I am using FileSystem.startLocalOutput() and FileSystem.completeLocalOutput() in my reduce tasks (more than 1) to produce some output. I have two questions:
1. if everything runs correctly, after the call to completeLocalOutput(), the output is copied to HDFS and the local file is deleted. However if the reduce tasks are killed in the middle for whatever reason, the local files are not deleted. How can I delete it when the reduce task failed. 2. if speculative execution is on, how can I force the two speculative tasks that are working on the same output to write to different local path, in case they happens to run on the same machine? Also, will there be any problem if they both succeed and copy the output back to HDFS in the same path in HDFS? appreciate any help. Cheers, Cedric
