On Dec 15, 2008, at 8:08 AM, Sandhya E wrote:
I have a scenario where, while a map/reduce is working on a file, the input file may get deleted and copied with a new version of the file. All my files are compressed and hence each file is worked on by a single node. I tried simulating the scenario of deleting the file before mapreduce was over, and map/reduce went ahead without complaining. Can I assume this will be the case always.
No, the results will be completely non-deterministic. Don't do this. That said, the thing that will save you in micro-tests of this is that if the file is missing at some point, the task will fail and retry.
-- Owen
