That's got to be it

On Mon, Aug 11, 2008 at 9:55 PM, lohit <[EMAIL PROTECTED]> wrote:
>>redoing each other's work and stomping on each others output files.
> I am assuming your tasks (reducers) are generating these files and these are 
> not the output file like part-00000
>
> Looks like you have speculative execution turned on.
> hadoop tries to execute parallel attempts of map/reduce tasks if it finds out 
> one of them is falling behind. All those task attempts are appended with a 
> number as you can see _0 and _1.
> If you have tasks which generate files to common files, then you hit this 
> problem.
> There are two ways out of this
> 1. turn off speculative execution by setting mapred.speculative.execution to 
> false
> 2. if you are generating outputs, try to use taskID for unique attempt.
>
>>I've attached the JSP output that indicates this; let me know if you
>>need any other details.
> No attachement.

I guess the listserv must have eaten it, as the one in my sent folder
has it.  It looks like this:

Task Attempts   Machine Status  Progress        Start Time      Shuffle
Finished        Sort Finished   Finish Time
task_200808062237_0031_r_000000_0       snark-0002.liveoffice.com       RUNNING 
88.01%  11-Aug-2008
14:11:13        11-Aug-2008 16:21:00 (2hrs, 9mins, 47sec)       11-Aug-2008
16:21:00 (0sec)
task_200808062237_0031_r_000000_1       snark-0005.liveoffice.com       RUNNING 
88.01%11-Aug-2008
16:21:03        11-Aug-2008 16:21:04 (0sec)     11-Aug-2008 16:21:04 (0sec)



        Last 4KB
Last 8KB
All
        0

Reply via email to