[ http://issues.apache.org/jira/browse/HADOOP-491?page=all ]
Yoram Arnon updated HADOOP-491:
-------------------------------
Summary: streaming jobs should allow programs that don't do any IO for
a long time (was: there should be a protocol for a map or reduce task to tell
the jobtracker its alive)
Description:
The jobtracker relies on task to send heartbeats to know the tasks are still
alive.
There is a 600 seconds timeout preset.
hadoop streaming also uses input to or output from the program it spawns to
indicate progress, sending appropriate heartbeats.
Some spawned programs spend longer that 600 seconds without any output while
being perfectly healthy.
It would be good to enhance the interface between hadoop streaming and the
programs it spawns to track a healthy program in the absense of output.
There are certain dangers with this protocol: e.g. a task can run a separate
thread that does nothing but send "i'm alive" message. This would be a user
bug to abuse the API in such way.
was:
The jobtracker relies on task to send heartbeats to know the tasks are still
alive.
There is a 600 seconds timeout preset.
hadoop streaming also uses output from the program it spawns to indicate
progress, sending appropriate heartbeats.
Some spawned programs spend longer that 600 seconds without any output while
being perfectly healthy.
It would be good to enhance the interface between hadoop streaming and the
programs it spawns to track a healthy program in the absense of output.
There are certain dangers with this protocol: e.g. a task can run a separate
thread that does nothing but send "i'm alive" message. This would be a user
bug to abuse the API in such way.
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: http://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
>
> The jobtracker relies on task to send heartbeats to know the tasks are still
> alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to
> indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while
> being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the
> programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate
> thread that does nothing but send "i'm alive" message. This would be a user
> bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
