Speculative execution must be on for hadoop in your setup. With speculative execution 'ON' framework starts running tasks on different nodes in parallel. The one that finishes first contributes, the other one is killed.
-Shrijeet On Mon, Jan 24, 2011 at 11:56 AM, Vijay <tec...@gmail.com> wrote: > Hi, > > When I run hive queries, everything finishes successfully but in the > jobtracker UI, I see a lot of failed/killed task attempts. For example > I have a simple query that finishes in 4 minutes. On the jobtracker UI > here are the map and reduce stats: > > MAP - Num Tasks: 138, Complete: 138, Failed/Killed Task Attempts: 0/41 > REDUCE - Num Tasks: 20, Complete: 20, Failed/Killed Task Attempts: 0/3 > > I don't see any error messages for the tasks that are in the KILLED > category. It looks as if it ran just fine. I guess I don't understand > what the above stats mean. > > The Failed/Killed numbers always seem proportional to the total number > of tasks (and to the length of the entire job to an extent). Does > anybody have any insight into this? > > Thanks, > Vijay >
